ARC-Tune — ARC-AGI LLM Fine-Tuning

LoRA fine-tuning an open-weight LLM on an augmented ARC-AGI dataset, evaluated against ConceptARC

What it does#

Fine-tunes Starling-LM-7B-alpha with LoRA to solve tasks from the Abstraction and Reasoning Corpus (ARC-AGI) and ConceptARC — grid puzzles that require inducing a shared transformation rule from a handful of input/output examples — and evaluates the result against the base model.

Data augmentation#

The 400 ARC training tasks are programmatically augmented into roughly 23k examples, then converted into text prompts (grids rendered as literal list-of-lists) via a shared instruction template, giving the fine-tuning run far more coverage than the raw dataset provides.

Fine-tuning#

The base model is loaded in 4-bit (bitsandbytes), fitted with a LoRA adapter (peft), and trained on the augmented prompt/response pairs with trl's SFTTrainer.

Evaluation#

Base and fine-tuned models are run over the ARC evaluation, training, and ConceptARC sets, with completions parsed and scored by exact grid match. ConceptARC's grouping of tasks by reasoning concept (Center, Copy, Count, ...) pinpoints which specific concepts the fine-tune improved, rather than just an aggregate accuracy number.