Abstract
We use Group Relative Policy Optimization (GRPO) to train a large language model (LLM) for MiniGrid level generation with a fun-aligned reward, chosen for its stability under the sparse, episodic feedback inherent to gameplay-based evaluation. The reward encourages skill-discriminative challenges, meaningful object interactions, and diverse level layouts, producing levels that are not only playable but also fun and behaviorally rich.
Problem — Game Level Generation
- Level design is labor-intensive.
- There is no generative method to generate "fun" levels.
- ⇒ How to generate fun game?
Motivation — LLMs: Good but Not Enough
LLMs have rich game design knowledge prior.
LLMs don't know how to generate legal, fun levels.
Methodology
How We Define Fun?
We propose three key dimensions to evaluate whether a generated game level is "fun":
Skill-Discriminative
A fun level should differentiate players of different skill levels — better players should achieve better outcomes.
Diverse
Generated levels should exhibit variety in structure and solution paths, avoiding repetitive patterns.
Interaction
Levels should encourage meaningful object interactions, creating engaging gameplay experiences.
SFT + Multi-Reward GRPO
Data Bootstrapping with SFT
We use 11 maze-generation algorithms (random DFS, Prim's, recursive division, etc.) to construct a structurally diverse Supervised Fine-Tuning (SFT) dataset for Qwen, exposing the model to a broad range of spatial layouts and difficulty profiles.
Multi-Reward GRPO
To capture "fun," we define three proxy rewards—skill discrimination, diversity, and interaction richness—and optimize them via Group Relative Policy Optimization (GRPO), which handles the inherent sparsity of these signals by comparing relative performance across sampled outputs.
Fun-aligned Reward Design
Format
LLM output is parseable as a valid MiniGrid environment.
Solvability
Generated level is confirmed solvable by BFS.
Regret
Performance gap between a strong and a weak agent; rewards skill-discriminative levels.
Interaction
Number of object interactions the agent performs in an episode.
Diversity
Mean pairwise Hamming distance across a group of generated levels.
Results
Main Comparison
| Method | Format Correctness ↑ | Solvability ↑ | Held-out Regret ↑ | Held-out Interactions ↑ | Diversity (path-Jaccard) ↑ |
|---|---|---|---|---|---|
| Vanilla LLM | 1.8% | 1.8% | 0.004 | 0.004 | 0.010 |
| SFT LLM | 94.4% | 78.2% | 0.101 | 0.232 | 0.287 |
| GRPO LLM (Ours) | 95.0% | 79.4% | 0.173 | 0.864 | 0.324 |
↑ indicates higher is better. Bold denotes best result. "—" = data to be filled in.
Visualization
Regret Trend during Training
Held-out Generalization Performance
Generated Level Samples Comparison
Vanilla LLM
SFT LLM
GRPO LLM (Ours)
Ablation Study
We conduct ablation experiments to understand the contribution of each component. We examine: (1) removing SFT pre-training, and (2) removing individual reward terms one at a time.
| Configuration | Format Correctness ↑ | Solvability ↑ | Held-out Regret ↑ | Held-out Interactions ↑ | Diversity (path-Jaccard) ↑ |
|---|---|---|---|---|---|
| Ours | 95.0% | 79.4% | 0.173 | 0.864 | 0.324 |
| w/o regret reward | 96.6% | 88.2% | 0.152 | 1.106 | 0.291 |
| w/o interaction reward | 97.6% | 81.2% | 0.152 | 0.252 | 0.360 |
| w/o diversity reward | 95.4% | 87.8% | 0.156 | 1.024 | 0.309 |
"—" = data to be filled in. Bold denotes best result.
Conclusion
- Framework Proposal: We propose a two-stage Supervised Fine-Tuning (SFT) + Group Relative Policy Optimization (GRPO) framework that trains a large language model (LLM) to generate fun game levels, leveraging the complementary strengths of imitation learning and reward-driven optimization.
- Decomposing "Fun": We decompose "fun" into three complementary proxy rewards — skill discrimination (regret), interaction richness, and layout diversity — each grounded in actual agent gameplay and structural analysis, providing a principled and measurable approximation of an otherwise subjective concept.
- Experimental Validation: Our model substantially outperforms baselines on all gameplay-oriented metrics, and ablation studies confirm that each reward captures a distinct, non-redundant aspect of fun, validating the design of our multi-reward objective.
Limitations
- No human evaluation was conducted; proxy rewards may not fully capture human-perceived fun.
- Experiments are limited to MiniGrid and may not generalize to more complex game environments.
- Regret reward quality depends on pre-trained agents, which may bias generation toward certain level styles.
Future Work
- Incorporate human playtesting or preference feedback to better align with human notions of fun.
- Extend the framework to more complex environments such as MiniHack or NetHack.
- Explore curriculum-based reward shaping for controllable difficulty generation across skill levels.