Constrained Code Generation

Summary Statistics

1. Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation tasks. However, unconstrained generation often produces code that, while syntactically correct, may not adhere to project-specific patterns, naming conventions, or structural requirements. This limitation becomes critical in enterprise environments where code must conform to established style guides, API contracts, and security policies.

Ananke addresses this challenge through constrained decoding, a technique that modifies the token probability distribution during generation to ensure outputs satisfy specified constraints. Unlike post-hoc validation or prompt engineering approaches, constrained decoding guarantees that generated code structurally conforms to requirements from the first token.

1.1 Research Questions

1. RQ1 (Correctness): Does constrained generation maintain or improve functional correctness compared to unconstrained generation?
2. RQ2 (Quality): How does constraint enforcement affect code quality metrics including readability, complexity, and security?
3. RQ3 (Efficiency): What is the computational overhead of constraint enforcement during generation?

2. Methodology

2.1 Benchmark Design

Our benchmark consists of 15 programming tasks designed to evaluate code generation across diverse domains. Following best practices from HumanEval and MBPP, each task includes:

• A natural language description of the required functionality
• Explicit requirements specifying function signatures, behavior, and constraints
• A comprehensive unit test suite for functional correctness verification
• Reference implementation for quality comparison
• Constraint specification in Ananke format (regex patterns, type constraints, structural requirements)

2.2 Task Categories

2.3 Evaluation Metrics

3. Experimental Setup

3.1 Configuration

3.2 Ananke Pipeline

The constrained generation pipeline consists of three stages:

1. Clew (Extraction): Parse source code to extract structural patterns and constraints
2. Braid (Compilation): Compile constraints into llguidance-compatible format
3. Maze (Generation): Generate code with constraints enforced via vLLM structured outputs

3.3 Constraint Enforcement vs. Scoring

4. Results

4.1 pass@k Metrics (HumanEval-style)

Following the methodology from Chen et al. (2021), we report pass@k which measures the probability that at least one of k samples passes all tests:

4.2 Per-Task Results

4.3 Timing Analysis

Constraint Compilation Overhead: The Ananke constraint compilation (Braid → llguidance) adds approximately 4ms per task. This one-time cost is amortized across multiple generations using the same constraints and can be cached for production use.

5. Analysis

5.1 Key Findings

RQ1 (Correctness): Constrained generation achieves comparable or better functional correctness. This demonstrates that constraint enforcement does not negatively impact the model's ability to generate functionally correct code.

RQ2 (Quality): Constrained generation shows a consistent improvement in quality metrics, with an average delta of +14.5 points (σ=3.4). The improvement is primarily driven by constraint adherence (100% vs 50%), indicating that explicit constraint enforcement effectively guides the model toward desired code patterns.

RQ3 (Efficiency): When the model is warm, constrained generation shows 81% speedup compared to baseline. This is consistent with findings from JSONSchemaBench that constrained decoding can actually improve throughput through reduced token sampling space.

5.2 Constraint Adherence Analysis

The most significant improvement from constrained generation is in constraint adherence (100% vs 50%). This includes:

• Export statements: Constrained generation always produces properly exported functions
• Type annotations: TypeScript type signatures are consistently correct
• Naming conventions: Function and variable names match requirements

5.3 Limitations

• Sample size: Evaluation covers 15 tasks; larger benchmarks would increase statistical power
• Single model: Results are specific to Qwen2.5-Coder-7B; larger models may show different patterns
• pass@1 only: We measure single-sample correctness; pass@k with k>1 would provide additional insight
• TypeScript only: Evaluation focuses on TypeScript; multi-language evaluation is future work

6. Conclusion

This evaluation demonstrates that Ananke's constrained code generation approach provides measurable benefits over unconstrained generation. Constrained generation wins on 15/15 tasks with an average quality improvement of +14.5 points.

The primary benefit of constraint enforcement is ensuring structural compliance with project requirements, as evidenced by the 100% constraint adherence rate. This is particularly valuable in enterprise environments where code must conform to established patterns and conventions.

Future work will expand the benchmark to include additional languages, implement pass@k evaluation with multiple samples per task, and evaluate against larger language models.

References

• [1] Chen et al. "Evaluating Large Language Models Trained on Code." arXiv:2107.03374 (HumanEval)
• [2] Austin et al. "Program Synthesis with Large Language Models." arXiv:2108.07732 (MBPP)
• [3] Geng et al. "JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for LLMs." arXiv:2501.10868
• [4] Zheng et al. "Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review." arXiv:2406.12655