GeekCoding101 make your way to geek

OpenAI o3 on ARC-AGI: Scores, Cost, Limits

OpenAI's o3 scored 75.7% on ARC-AGI's Semi-Private set and 87.5% at 172x compute. What ARC-AGI tests, what it cost, and why it isn't AGI.

5 min read
Daily Ai Insight Issue 17 20241221
On this pageThe announcement in brief

Published Dec 21, 2024 · Daily AI Insight

In December 2024, OpenAI previewed its o3 reasoning model, and the ARC Prize team reported that o3 scored 75.7% on the ARC-AGI Semi-Private Evaluation Set at roughly $20 per task. In a high-compute configuration using 172x more compute, it reached 87.5%. That was a large jump over earlier generative models on a benchmark built to reward adapting to new problems rather than recalling training data. Even so, o3 still failed some easy tasks and is not AGI.

Note: This post covers the December 2024 preview results. Models, benchmarks and scores have moved on since then. For current figures, check the ARC Prize leaderboard at arcprize.org.

The announcement in brief

On December 20, 2024, OpenAI closed out its 12-day holiday announcement series by previewing two models: o3 and o3-mini. On the same day, the ARC Prize organization published its evaluation of o3 on the ARC-AGI benchmark.

o3’s ARC-AGI results at a glance

SettingSemi-Private Eval scoreComputeCost per task
Efficient mode75.7%Baseline~$20
High-compute mode87.5%172x efficient modeMuch higher (scales with compute)
oai-o3-arc-agi-score

Source: https://arcprize.org/blog/oai-o3-pub-breakthrough

What is ARC-AGI?

ARC stands for the Abstraction and Reasoning Corpus. François Chollet introduced it in 2019 as a test of how well a system handles problems it has not seen before.

Each task is a small visual puzzle on a grid:

  • You get a few example pairs: an input grid and the output grid it should become.
  • You work out the rule that turns inputs into outputs.
  • You apply that rule to a new input.

The tasks are designed to be:

  • Simple for humans: most people can spot the pattern with basic logical reasoning.
  • Hard for AI: especially for models that have not been trained on similar examples, because memorized patterns do not transfer.

The Semi-Private Evaluation Set is a held-out set of tasks that is not published. Scores on it are harder to inflate by training on the test.

How o3 differs from traditional LLMs

Traditional GPT-style models lean heavily on what they absorbed during training. In effect, they retrieve and run “programs” they have already learned. That works well for familiar tasks but struggles when a problem needs a new combination of ideas.

ARC Prize’s analysis of o3 suggested a different approach:

  1. Program search and execution: o3 appears to generate natural-language “programs”, meaning chains of thought (CoT), for a task and run them internally.
  2. Evaluation and refinement: it appears to evaluate several candidate reasoning paths and pick the best one. ARC Prize compared this to search methods such as Monte-Carlo Tree Search (MCTS).

OpenAI did not publish o3’s internals, so treat this as informed interpretation, not a confirmed architecture.

Why it costs so much

This kind of search is compute-hungry. Solving a single task can take millions of tokens. That explains both the high score and the price tag, especially in high-compute mode.

Efficiency vs. cost

The two modes show a clear trade-off:

  • Efficient mode: about $20 per task for 75.7%.
  • High-compute mode: 172x the compute for 87.5%. That is 11.8 more points at a far higher cost.

At the time of the announcement, cost-performance was the main open question. A model that needs that much compute per puzzle is impressive in a research setting but expensive to run at scale.

What made o3 a big step

  1. Task adaptability: o3 builds task-specific reasoning on the fly instead of relying only on patterns it memorized.
  2. Generalization: its ARC-AGI scores were far above earlier GPT-series models on a benchmark that targets exactly this skill.
  3. Architecture over scale: ARC Prize argued that simply scaling up GPT-4-style models would not have produced comparable results. How a model reasons at inference time mattered as much as its size.

Is o3 AGI?

No. ARC Prize said so directly, for two reasons.

  • It fails some simple tasks. Even in high-compute mode, o3 missed some puzzles that people find easy. This shows gaps in basic reasoning.
  • ARC-AGI-2 looked much harder. Early testing suggested o3 might score below 30% on the planned ARC-AGI-2 benchmark, while humans were expected to score over 95%.

o3 was a milestone on ARC-AGI, not the finish line.

What came next (as announced in December 2024)

  1. ARC-AGI-2: ARC Prize planned to launch a harder benchmark in 2025 and invited open-source research building on o3’s results.
  2. Deeper analysis of o3: researchers wanted to understand how o3 works and where it gets stuck.
  3. Third-generation benchmarks: the ARC Prize Foundation said it was already working on a further generation of tests for adaptability and generalization.

Conclusion

The o3 results showed that a model built to search over its own reasoning at inference time can do far better on novel tasks than models that mostly recall training data. They also showed what that costs: a lot of compute per problem, and still no reliable handling of easy puzzles. For anyone tracking reasoning models, December 2024 is a useful reference point. Compare any newer model against these numbers and against ARC-AGI-2.

FAQ

What did OpenAI o3 score on ARC-AGI? 75.7% on the Semi-Private Evaluation Set in efficient mode (about $20 per task), and 87.5% in high-compute mode using 172x the compute. These are the December 2024 preview results reported by ARC Prize.

What does ARC stand for in ARC-AGI? Abstraction and Reasoning Corpus. It is a set of grid puzzles where you infer a rule from a few examples and apply it to a new input.

What is “high-compute mode”? The same o3 system given far more compute per task (172x efficient mode). This lets it explore more reasoning paths, which raised the score from 75.7% to 87.5% at a much higher cost.

Is o3 AGI? No. It still failed some easy tasks, and early tests suggested it would score below 30% on ARC-AGI-2, where humans were expected to score over 95%.

Was o3-mini tested on ARC-AGI? The ARC Prize results covered in this post are for o3. OpenAI previewed o3-mini in the same announcement.

Disclaimer: The content above includes contributions generated with the assistance of AI tools.

Thanks for reading my blog post! Feel free to check out my other posts or contact me via the social links in the footer.

Keep reading

All posts