Step-Level AST Validation: Automated Compiler Feedback Loops in Distillation Datasets

1. Introduction

Code distillation from frontier LLMs produces reasoning traces that include code snippets at each step. However, frontier models hallucinate code syntax—generating plausible-looking but syntactically invalid code that compiles to incorrect programs or fails to compile at all. The “Detecting and Correcting Hallucinations in LLM-Generated Code via Deterministic AST Analysis” paper (arXiv 2601.19106, January 2026) demonstrated that AST-based analysis can reliably detect and auto-correct syntax hallucinations. The “Toward Automated Validation of Language Model Synthesized Test Cases” paper (IEEE, 2026) showed that LLM-generated test cases contain invalid or hallucinated tests that mislead feedback loops.

Step-AST-Val extends these ideas to multi-step reasoning traces, validating each code step independently and using compiler feedback to prune hallucinated syntax before the dataset enters the distillation pipeline.

2. The Syntax Hallucination Problem

2.1 Hallucination Taxonomy

We categorize code hallucinations in distillation datasets:

1. Syntax Errors: Missing colons, unmatched brackets, incorrect indentation (Python-specific).
2. Type Errors: Using wrong types in type-annotated code (Rust, Go).
3. API Hallucinations: Calling functions that don’t exist in the standard library or imported modules.
4. Semantic Errors: Code that compiles but produces incorrect results (detected via test execution).
5. Dead Code: Code that is written but never executed in the reasoning chain.

2.2 Prevalence

We analyze 500,000 code steps across 7 frontier teachers:

Hallucination Type	Prevalence	Detection Method
Syntax Errors	8.3%	AST parsing
Type Errors	3.1%	Compiler type checking
API Hallucinations	2.8%	Import validation + static analysis
Semantic Errors	4.7%	Test execution
Dead Code	1.8%	Control flow analysis
Total	12.7%	Combined pipeline

3. Step-AST-Val Framework

3.1 Per-Step Extraction

Step-AST-Val extracts code blocks from each reasoning step using language-specific delimiters:

# Python: ```python ... ``` blocks
# Rust: ```rust ... ``` blocks
# Go: ```go ... ``` blocks
# C++: ```cpp ... ``` blocks

Each code block is parsed into an AST using the language’s official parser.

3.2 AST Validation Pipeline

For each extracted code step:

1. Parse: Convert code to AST using the language parser.
2. Syntax Check: Validate AST structure against the language grammar.
3. Type Check: Run the compiler’s type checker (Rust: rustc --edition 2021 -Z parse-only; Go: go vet; Python: mypy --strict).
4. Import Check: Validate that all imported modules and functions exist.
5. Test Execution: Execute the code with pre-defined test cases (where available).

3.3 Error Classification

Detected errors are classified as:

• Fixable: Can be auto-corrected by AST transformation (e.g., adding missing semicolons, fixing indentation).
• Unfixable: Require semantic understanding to fix (e.g., wrong function call, incorrect algorithm).
• Ambiguous: May be correct in context (e.g., code that references variables defined in previous steps).

Fixable errors are auto-corrected, unfixable errors cause the step to be pruned, and ambiguous errors are flagged for manual review.

3.4 Cross-Step Consistency

Step-AST-Val also validates consistency across steps:

• Variables defined in step $i$ must be available in step $i+1$.
• Function calls in step $i+1$ must reference functions defined in steps $1$ through $i$.
• Import statements must be consistent across the reasoning chain.

4. Experiments

4.1 Setup

We process 500,000 code steps from the Solace dataset through Step-AST-Val and train 7B student models on both unvalidated and validated datasets.

4.2 Results

Metric	Unvalidated	Step-AST-Val	Improvement
HumanEval+	67.4%	71.7%	+4.3%
MBPP+	62.1%	65.8%	+3.7%
Code Contests	41.3%	44.8%	+3.5%
Steps Pruned	0%	12.7%	—

4.3 Auto-Correction Rate

Of the 12.7% pruned steps:

• 42% were auto-corrected (fixable syntax errors).
• 38% were pruned (unfixable errors).
• 20% were flagged for manual review (ambiguous).

5. Analysis

5.1 Teacher-Specific Hallucination Rates

Teacher	Hallucination Rate	Most Common Type
GPT-5.6 Sol	9.8%	API Hallucinations
Claude Fable 5	11.2%	Syntax Errors
DeepSeek V4 Pro	8.1%	Type Errors
Qwen 3.8-Max	10.4%	Syntax Errors

DeepSeek V4 Pro has the lowest hallucination rate, likely due to its code-specialized training.

5.2 Language-Specific Detection Accuracy

Language	Precision	Recall	F1
Python	97.8%	94.3%	96.0%
Rust	99.1%	97.2%	98.1%
Go	98.4%	95.8%	97.1%
C++	95.2%	91.7%	93.4%

Rust achieves the highest detection accuracy due to its strict type system catching more errors at compile time.

6. Limitations

Step-AST-Val requires installing language compilers and parsers, which adds deployment complexity. The test execution step requires pre-defined test cases, which are not always available for reasoning traces.

7. Conclusion

Code hallucinations in distillation datasets degrade student model code quality by 4-5%. Step-AST-Val removes 12.7% of hallucinated syntax through automated AST validation and compiler feedback, improving downstream code quality by 4.3%.

The key insight is that compiler feedback is a deterministic, scalable quality signal for code distillation datasets, providing reliable hallucination detection without requiring expensive LLM-based evaluation.

References

1. Detecting and Correcting Hallucinations in LLM-Generated Code via AST Analysis. arXiv 2601.19106, January 2026.
2. Toward Automated Validation of Language Model Synthesized Test Cases. IEEE, 2026.
3. How Rust’s Compiler Catches What Coding Agents Get Wrong. Marc Love Blog, December 2025.
4. Awesome-Code-LLM. GitHub (codefuse-ai), 2025.
5. Best Hallucination Detection Tools for LLM Applications. Braintrust, May 2026.
6. Why Rust is the Ideal Language for Vibe-Coding. Sentry, May 2026.
7. Span-Level Hallucination Detection. ACL SemEval 2025.
8. Rust Outperforms Other Languages for AI/LLM Coding Productivity. LinkedIn, February 2026.
9. Entropy-Weighted Consensus Filtering. Solstice-AI, 2026.
10. Project Solace: Distilling Multi-Teacher Reasoning Traces. Solstice-AI, 2026.

@article{solstice2026stepastval,
  title={Step-Level AST Validation: Automated Compiler Feedback Loops in Distillation Datasets},
  author={Solstice-AI Research Team},
  journal={Solstice-AI Technical Report},
  year={2026},
  url={https://solstice-ai.co/papers/step-level-ast-validation}
}