Reasoning models: what changes when the model thinks first

Until recently, getting a language model to reason meant asking it to: a "think step by step" line, a few worked examples, a scratchpad tag. Since September 2024 a different kind of model has become an ordinary API call. It writes its own working before it answers, and you pay for every token of that working whether or not you can read it. That changes how you prompt, what a request costs and how long it takes. This is a practical guide to those changes as they stand in March 2025.

What "thinking first" means

Asking a model to show its working is an old trick. In January 2022, Jason Wei and colleagues showed that few-shot examples with the intermediate steps written out, which they called chain-of-thought prompting, improved large models on arithmetic, commonsense and symbolic reasoning. A few months later, Takeshi Kojima and colleagues found that adding "Let's think step by step" before the answer helped even with no examples.

Both are prompting techniques. The model is unchanged; you're asking it to spend more tokens before it commits.

A reasoning model is trained to do that by itself. When OpenAI introduced o1 in September 2024, it said performance improved with more reinforcement learning during training and "with more time spent thinking (test-time compute)". Test-time compute is compute spent while answering a request rather than while training. Today it's mostly serial: a long stretch of reasoning, then the answer. (Anthropic also describes a parallel kind: sample several attempts and pick one by majority vote or a scoring model.)

The reasoning itself arrives as thinking tokens. Whether you can read them depends on the provider:

• OpenAI keeps o1's raw chain of thought hidden. The API reports how many reasoning tokens were used, not what they said.
• DeepSeek returns R1's reasoning as a separate field, and the open weights produce it as plain text.
• Anthropic returns Claude 3.7 Sonnet's thinking as thinking blocks, with the occasional redacted_thinking block encrypted by its safety systems.

Visible thinking helps with debugging, but treat it as a draft, not a log. Anthropic says plainly it can't be sure the visible thinking reflects what actually drives the answer.

What you can call in March 2025

Six months in, there's a real choice, and the models differ in what they show you and what you can turn.

A few things the table hides:

• Claude 3.7 Sonnet is one model in two modes. You switch thinking on per request, so the same model can answer instantly or think for a while. Anthropic lets you set the budget anywhere up to its 128K output limit.
• DeepSeek-R1 came with six distilled models from 1.5B to 70B parameters, built on Qwen2.5 and Llama 3, so you can run a reasoning model on your own hardware. Alibaba's Qwen team added QwQ-32B, under Apache 2.0, on 6 March.
• DeepSeek's API docs said in late February that a reasoning_effort setting was on its way. Until it arrives, you can't dial R1's thinking down through the API, and its chain of thought can run to 32K tokens.
• Gemini 2.0 Flash Thinking is still an experimental model. Google updated it on 21 January as gemini-2.0-flash-thinking-exp-01-21. The API returns only the answer, with no JSON mode. I'd try it, but not build on it yet.

Prompting: give it the goal, not the steps

The advice from all three providers points the same way. Here's how I'd sum up the difference:

What each provider says:

• OpenAI's guide for o1 and o3-mini: reasoning models often don't need few-shot examples, so try zero-shot first, and be very specific about the end goal. Separate the parts of the input with Markdown, XML tags or section titles, and keep retrieved context to what's relevant.
• Anthropic's extended thinking tips: Claude often does better with a high-level instruction to think carefully than with prescriptive steps. Start general, read the thinking, then add specifics where it goes wrong. Examples still work, but try without them first.
• The DeepSeek-R1 paper: few-shot prompting consistently made R1 worse, so describe the problem directly and specify the output format, zero-shot. The README adds: no system prompt, every instruction in the user turn.

One nuance: the same README suggests a "reason step by step" line for maths problems, paired with a fixed place for the final answer. So the phrase isn't harmful, just mostly redundant; the useful half is the bit about where the answer goes.

A good prompt for a reasoning model reads like a brief to a capable colleague, not a script:

Goal: a staff rota for next week, Monday to Saturday, two shifts a day.
 
Constraints:
- 6 staff; nobody works more than 5 shifts.
- Every shift needs at least one key-holder (Priya or Sam).
- Ravi can't work Saturday evening.
 
A correct answer:
- satisfies every constraint above
- is JSON matching the schema below, with nothing before or after it
 
Schema: {"shifts": [{"day": "...", "shift": "morning|evening", "staff": ["..."]}]}

Structured outputs still matter

The thinking is free-form. The answer your code reads shouldn't be. Support varies:

• o1 and o3-mini support function calling and Structured Outputs, and take instructions as developer messages rather than system messages (from o1-2024-12-17 onwards). They avoid Markdown by default; OpenAI's guide says to put Formatting re-enabled on the first line of the developer message to get it back.
• Claude 3.7 Sonnet works with tools while thinking, but not with forced tool use, a prefilled response, or changes to temperature, top_p or top_k.
• DeepSeek's deepseek-reasoner didn't support function calling or JSON output in late February, and ignores temperature. Validate its text yourself.

Cost: you pay for the thinking

All three APIs bill thinking as output:

• OpenAI's SDK documents reasoning tokens as part of the completion tokens, for billing, for output limits and for the context window. You see the count in usage.completion_tokens_details.reasoning_tokens.
• Anthropic bills thinking tokens, including redacted ones, as output tokens, and they count toward your rate limits.
• DeepSeek prices deepseek-reasoner's chain of thought and final answer at the same output rate.

Here are the list prices in early March 2025, per million tokens:

DeepSeek also takes 75% off R1 between 16:30 and 00:30 UTC.

The rate per token matters less than the multiplier. Say a task needs a 200-token answer and the model thinks for 3,000 tokens first. Those numbers are illustrative, but the arithmetic isn't: you're billed for 3,200 output tokens, sixteen times the answer. On o3-mini that's about $0.014 a request instead of $0.0009. On o1 it's about $0.19 instead of $0.012.

Budgets and effort are the knobs

You can't set the exact number of thinking tokens, but you can bound them:

# Not run: needs an OpenAI key. Parameter names from the SDK as of March 2025.
from openai import OpenAI
 
client = OpenAI()
resp = client.chat.completions.create(
    model="o3-mini",
    reasoning_effort="low",        # "low" | "medium" | "high"
    max_completion_tokens=8000,    # caps reasoning AND the visible answer together
    messages=[
        {"role": "developer", "content": "Return only JSON matching the schema."},
        {"role": "user", "content": "..."},
    ],
)
print(resp.choices[0].message.content)
print(resp.usage.completion_tokens_details.reasoning_tokens)

# Not run: needs an Anthropic key.
import anthropic
 
client = anthropic.Anthropic()
resp = client.messages.create(
    model="claude-3-7-sonnet-20250219",
    max_tokens=8000,                                       # must be above the budget
    thinking={"type": "enabled", "budget_tokens": 2000},  # minimum is 1,024
    messages=[{"role": "user", "content": "..."}],
)
for block in resp.content:
    if block.type == "text":
        print(block.text)
print(resp.usage.output_tokens)  # thinking is billed as output

# Not run: needs a DeepSeek key. DeepSeek's API is OpenAI-compatible.
from openai import OpenAI
 
client = OpenAI(api_key="...", base_url="https://api.deepseek.com")
resp = client.chat.completions.create(
    model="deepseek-reasoner",
    messages=[{"role": "user", "content": "..."}],
)
print(resp.choices[0].message.reasoning_content)  # the chain of thought
print(resp.choices[0].message.content)            # the answer

Three things catch people out:

• On OpenAI, a tight cap can buy you nothing. max_completion_tokens covers reasoning and answer together (the older max_tokens doesn't work with o1-series models). Set it too low and reasoning can use all of it: a forum report from November 2024 shows an o1-mini response with 4,000 completion tokens, all reasoning, and an empty answer, all billed. At o1's launch, OpenAI suggested leaving room for around 25,000 tokens while experimenting.
• On Claude, the budget is a target, not a hard limit. Anthropic's guide suggests starting at the 1,024 minimum and raising it step by step, though the same page also suggests 16,000 or more for complex tasks. Changing the budget invalidates prompt-cache entries in the message history; a cached system prompt and tool definitions survive.
• Old thinking doesn't pile up. Anthropic strips thinking from earlier turns and doesn't bill it as input. DeepSeek doesn't carry earlier reasoning forward either, and returns a 400 error if you send reasoning_content back, so drop that field before you append a turn.

Latency: the answer waits for the thinking

The answer can't start until the thinking is done. With hidden reasoning, a user watching a streamed response sees nothing during that time. With Claude you can stream the thinking as it's generated and show some sign of progress, which helps, but doesn't make the wait shorter.

Anthropic's docs are direct about the operational side. Expect longer response times. Streaming is required once max_tokens is above 21,333. For thinking budgets above 32K, they recommend the Batch API, because requests that long can run into timeouts and connection limits.

The budget is therefore a latency setting as much as a cost setting. Once you've measured where quality stops improving, setting the budget there rather than higher saves time as well as money.

When not to use one, and a simple router

The providers say this themselves. OpenAI's API price list recommends GPT-4o or GPT-4o mini for everyday tasks and o1 for complex, multi-step and STEM problems. Anthropic recommends extended thinking for particularly complex work such as maths, coding and analysis.

I'd keep reasoning models out of:

• Classification. Sentiment, intent, topic. One hop, nothing to plan.
• Extraction. Fields from an invoice or an email into a schema. A chat model with Structured Outputs is cheaper and faster.
• Short rewrites. A change of tone, a paragraph summary, a sentence translated.
• Anything a person is waiting on. Autocomplete, the first reply in a chat, voice.
• Arithmetic and dates. If code can compute it, have the model call code.

That still leaves plenty: multi-constraint planning, debugging, maths, careful analysis of a long document, and anything where checking the answer is easier than describing the method.

Most products have both kinds of request, so the pattern I'd recommend is a cheap model first and escalation only on failure:

The check in the middle is what makes this work, and it should be code, not another model call. Escalate when:

• The output fails validation: the schema, or a rule you can test, such as totals that don't add up or a required field left empty.
• A deterministic test fails. For code, run the tests. For a plan, run a constraint checker.
• The cheap model flags doubt in an explicit needs_review field. That's a weak signal, so measure how well it predicts real failures before trusting it.
• The input is in a category you've measured as hard. Send those straight to the reasoning tier.

Inside the reasoning tier, do the same again: start at low effort or a small budget and raise it only for what still fails. Claude 3.7 Sonnet makes this easy, because thinking on and off is one parameter on the same model. Log each request's route, tokens and result; that log tells you whether the router earns its keep.

How to tell whether thinking helped

Launch benchmarks tell you about the benchmark. To know whether thinking helps on your task, test side by side on your own inputs:

1. Collect 50 to 200 real inputs with answers you can check in code: exact match, schema plus rules, unit tests.
2. Run several arms on the same set: the cheap model with a direct prompt, the cheap model with "think step by step", and the reasoning model at its lowest and at a higher effort or budget.
3. Record four numbers per arm: accuracy, output tokens including thinking (from the usage fields), median and slow-tail latency, and cost per correct answer. Compare on the last one; it folds in both the price and the misses.
4. Run each arm more than once. DeepSeek's README recommends multiple runs and averaging, and that applies to any model sampled at a non-zero temperature.
5. Count format failures separately from wrong answers. A right answer in the wrong place still fails in production, but the fix is a better output spec, not more thinking.
6. Read the misses, and the visible thinking, as hints about where the model went wrong, not as proof.

Repeat it when a new model or price lands. Over the last six months, that would have been most months.

What I'd take away

Reasoning models are the right tool when the hard part of a task is working out what to do, and the wrong one when the task is doing a known thing quickly. What I'd carry into the next project:

• Write prompts as briefs. Goal, constraints, what a correct answer looks like, the exact output format. Cut the step-by-step scripts.
• Treat thinking as a line item. It's billed as output, it can be many times the answer, and effort or budget sets it per request. Choose the setting per task, from measurements.
• Route. Put a cheap model, a strict schema and a code check in front of the reasoning model, so you only pay for thinking where it's needed.
• Measure on your own data. Fifty examples with automatic checks, a few arms, cost per correct answer. A small, honest test beats a leaderboard.