Small models, local machines: when 3B parameters is enough

A lot of useful AI work is small. Pull five fields out of a message, put a ticket in one of five buckets, tidy a sentence. You don't need the biggest model for that, and sometimes the text isn't allowed to leave the machine anyway. The real question is where "small" stops being enough, so I ran a 3-billion-parameter model on an M2 Mac with 8 GB of memory and measured it: how fast it is, how much memory it takes, what it gets right and where it falls over.

What "open-weight" means

A model is two things: the code that runs it, and the weights, the billions of numbers learned in training. An open-weight model is one whose weights you can download and run on your own hardware.

It isn't the same as open source. The Open Source Initiative published version 1.0 of its Open Source AI Definition in October 2024. It asks for three things: the parameters, the complete code used to train and run the model, and enough information about the training data that a skilled person could build a substantially equivalent system. Most models people call "open" ship weights and inference code but not their training data or full training pipeline. By that definition they're open-weight, not open source.

When you're building something, the practical question is narrower: what does the licence let me do? Some small models come under standard open-source licences such as Apache 2.0 or MIT. Others come under the maker's own terms, with use policies, user-count thresholds or a non-commercial clause. You have to read each one, and sometimes each size. Qwen2.5 is Apache 2.0 at most sizes, but not at 3B.

The small models worth knowing in October 2025

The families I'd look at for local work, with sizes and licences as their makers published them:

Three things stand out.

The 1B to 4B tier filled up this year. Gemma 3 1B and 4B, Qwen3 1.7B and 4B, Phi-4-mini at 3.8B: all of them fit in a few gigabytes once quantised. Google pitched the 270M Gemma specifically as a base to fine-tune for narrow jobs such as text classification and data extraction. One thing to know about Qwen3: it reasons step by step before answering by default, which can be switched off. On a laptop producing under 20 tokens a second, those extra tokens are seconds you wait.

Mixture-of-experts models need memory for all their weights, not just the active ones. gpt-oss-20b, released with the 120b as OpenAI's first open-weight language models since GPT-2, uses 3.6B parameters per token, which helps speed. But OpenAI says it runs within 16 GB of memory, twice what this laptop has. Active parameters set the speed; total parameters set the memory bill.

The model I tested is the awkward case. qwen2.5:3b was the model already on this machine, and its licence is research-only. For anything you'd ship, I'd repeat these tests on an Apache 2.0 or MIT model of similar size, such as Qwen3 4B or Phi-4-mini. The method below carries over. The numbers won't, exactly.

Quantisation in plain terms

Weights are usually released as 16-bit numbers, two bytes each. Qwen2.5 3B has about 3.1 billion of them, so the full-precision weights come to roughly 6 GB, too much to run comfortably beside everything else on an 8 GB machine.

Quantisation stores each weight in fewer bits. The file Ollama downloaded is 1.93 GB, which works out at about 5 bits per weight on average. You give up a little accuracy for a model a third of the size, and it also generates faster: producing each token means reading roughly all the weights from memory, so fewer bytes per weight means more tokens per second. Apple quotes 100 GB/s of memory bandwidth for the M2, which puts a ceiling of about 50 tokens a second on a 1.9 GB model. I measured about a third of that.

Two names you'll see everywhere:

GGUF is the file format from the llama.cpp project, introduced in August 2023. One file holds the quantised weights and the metadata needed to run them, such as the architecture and tokenizer. Ollama reports its copy of qwen2.5:3b as a GGUF file.

Q4_K_M is the quantisation type, and it reads in three parts:

• Q4: weights are mostly stored in 4 bits.
• K: the "k-quant" scheme added to llama.cpp in June 2023. Weights are grouped in blocks of 32 inside super-blocks of 256, and each block gets its own scale and minimum, so 4-bit numbers can follow the real values closely. Counting those scales, Q4_K costs 4.5 bits per weight.
• M: the medium mix. As llama.cpp first described it, half of the attention.wv and feed_forward.w2 tensors are kept at 6 bits (Q6_K) and the rest use Q4_K.

Q4_K_M is what Ollama serves for this model, and it sits in the middle: go lower and you shrink further at more cost to quality; go higher and you pay in memory you may not have.

Running it with Ollama

Ollama downloads a model, keeps it in memory and serves it on localhost:11434. ollama run qwen2.5:3b gives you a chat in the terminal; for code there's a REST API. Two of its features matter most for small models.

Structured outputs. Since Ollama 0.5.0, released in December 2024, the format field accepts a JSON schema and the reply is constrained to match it. Every schema-constrained reply in my tests parsed. Ollama's own tips for it: say "return as JSON" in the prompt as well, and set the temperature to 0.

Timings. Every reply reports how long Ollama spent loading the model, reading the prompt and generating, in nanoseconds, with token counts for each. The API docs give the formula: tokens per second is eval_count / eval_duration * 10^9.

Every quality test below goes through this helper. It uses only the standard library:

import json
import urllib.request
 
OLLAMA = "http://localhost:11434/api/chat"
 
 
def ask(prompt, schema=None, system=None, model="qwen2.5:3b", temperature=0, max_tokens=512):
    messages = [{"role": "system", "content": system}] if system else []
    messages.append({"role": "user", "content": prompt})
    body = {
        "model": model,
        "messages": messages,
        "stream": False,
        "options": {"temperature": temperature, "num_ctx": 4096, "num_predict": max_tokens},
    }
    if schema:
        body["format"] = schema  # Ollama constrains the output to this JSON schema
    req = urllib.request.Request(
        OLLAMA, data=json.dumps(body).encode(), headers={"Content-Type": "application/json"}
    )
    with urllib.request.urlopen(req, timeout=300) as resp:
        reply = json.load(resp)
    text = reply["message"]["content"]
    return json.loads(text) if schema else text

num_ctx is 4,096 tokens, which is also Ollama's default context window for most models. Ollama keeps a model loaded for five minutes after its last request (keep_alive defaults to 5m), and that's the difference between the warm and cold numbers below.

Speed and memory on an 8 GB M2

The benchmark: unload the model, send one request cold, send the same 48-token prompt three more times warm, then send one prompt of 1,110 tokens. Each request streams, so I can time the first token myself and also read Ollama's counters from the last chunk.

def chat_stream(prompt, max_tokens=256):
    """Stream one reply; return wall-clock time to first token plus Ollama's own stats."""
    body = {
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "stream": True,
        "options": {**OPTIONS, "num_predict": max_tokens},
    }
    t0 = time.perf_counter()
    ttft = None
    with post("/api/chat", body) as resp:
        for line in resp:
            chunk = json.loads(line)
            if ttft is None and chunk.get("message", {}).get("content"):
                ttft = time.perf_counter() - t0
            if chunk.get("done"):
                final = chunk
    wall = time.perf_counter() - t0
    ns = 1e9
    return {
        "ttft_s": round(ttft, 3),
        "wall_s": round(wall, 3),
        "load_s": round(final["load_duration"] / ns, 3),
        "prompt_tokens": final["prompt_eval_count"],
        "prompt_s": round(final["prompt_eval_duration"] / ns, 3),
        "prompt_tok_per_s": round(final["prompt_eval_count"] / (final["prompt_eval_duration"] / ns), 1),
        "output_tokens": final["eval_count"],
        "gen_tok_per_s": round(final["eval_count"] / (final["eval_duration"] / ns), 1),
    }

cold : {'ttft_s': 9.396, 'wall_s': 17.731, 'load_s': 8.804, 'prompt_tokens': 48, 'prompt_s': 0.388, 'prompt_tok_per_s': 123.8, 'output_tokens': 156, 'gen_tok_per_s': 19.8}
warm1: {'ttft_s': 4.195, 'wall_s': 14.07, 'load_s': 0.026, 'prompt_tokens': 48, 'prompt_s': 0.204, 'prompt_tok_per_s': 234.9, 'output_tokens': 156, 'gen_tok_per_s': 17.0}
warm2: {'ttft_s': 4.458, 'wall_s': 17.005, 'load_s': 1.146, 'prompt_tokens': 48, 'prompt_s': 0.275, 'prompt_tok_per_s': 174.3, 'output_tokens': 156, 'gen_tok_per_s': 12.4}
warm3: {'ttft_s': 3.269, 'wall_s': 13.86, 'load_s': 0.028, 'prompt_tokens': 48, 'prompt_s': 0.228, 'prompt_tok_per_s': 210.8, 'output_tokens': 156, 'gen_tok_per_s': 15.8}
long : {'ttft_s': 10.646, 'wall_s': 13.824, 'load_s': 0.331, 'prompt_tokens': 1110, 'prompt_s': 7.28, 'prompt_tok_per_s': 152.5, 'output_tokens': 43, 'gen_tok_per_s': 17.7}

generation, median of 5 runs

17 tok/s

generation, median of 5 runs

to first token, model loaded

0.25 s

to first token, model loaded

to first token from cold

9.2 s

to first token from cold

memory with a 4,096-token context

2.1 GB

memory with a 4,096-token context

The warm figure is 0.23 s and 0.26 s from two runs; the third spent 1.1 s reloading and took 1.4 s. The cold start is almost all loading: 8.8 of the 9.2 seconds. Memory, from ollama ps after the run:

NAME          ID              SIZE      PROCESSOR    CONTEXT    UNTIL
qwen2.5:3b    357c53fb659c    2.1 GB    100% GPU     4096       4 minutes from now

That's the 1.93 GB of weights plus the cache for a 4,096-token context, all on the M2's GPU, and about a quarter of the machine's memory.

In practice, at 17 tokens a second a label of a few tokens takes well under a second to generate, while the 156-token answer above took about 9 seconds to generate. Long inputs are the other slow part: the model needed 7.3 seconds just to read the 1,110-token prompt before it could start answering.

What it gets right

Classification

Twelve short support messages for an imaginary online shop, five labels, each label defined in the system prompt. The schema's enum means the model can't answer anything else.

LABELS = ["billing", "delivery", "returns", "account", "other"]
SCHEMA = {
    "type": "object",
    "properties": {"label": {"type": "string", "enum": LABELS}},
    "required": ["label"],
}
SYSTEM = """Classify the customer message for an online store. Labels:
billing: payments, charges, invoices, payment methods
delivery: shipping status, delays, lost parcels
returns: sending items back, exchanges, refunds for products
account: login, password, profile details
other: anything else"""

ok  billing   (expected billing ) I was charged twice for order 5521, please fix.
ok  delivery  (expected delivery) My parcel says delivered but I never got it.
ok  returns   (expected returns ) The shoes are a size too small, how do I send them back?
ok  account   (expected account ) I can't log in, the reset email never arrives.
ok  other     (expected other   ) Do you have a physical shop in Pune?
ok  billing   (expected billing ) Why is there a 49 rupee convenience fee on my invoice?
ok  delivery  (expected delivery) Courier keeps rescheduling, it's been five days.
ok  returns   (expected returns ) Item arrived broken. I want my money back.
ok  account   (expected account ) Please change the email address on my profile.
ok  billing   (expected billing ) Can I pay with UPI?
ok  returns   (expected returns ) You sent me the blue one instead of black.
ok  other     (expected other   ) Love the new collection, any discounts for students?
 
12/12 correct

Twelve messages I wrote myself is a smoke test, not a benchmark. But this is the shape of task a 3B model handles: short input, a small fixed set of answers, and a schema that makes any other answer impossible.

Extraction

Five made-up receipts in the styles you see in India, extracted into merchant, date, line items and total. The schema has merchant, date, an items array of name, qty and unit_price, and total; the system prompt is one line: "Extract the receipt into JSON. Write dates as YYYY-MM-DD. Prices are numbers without currency symbols." Three came back perfect. These are the other two:

Sharma Hardware, 07/10/2025: 3 tins of paint at 650 each, 2 brushes at 85 each. Grand total 2120.
{"merchant": "Sharma Hardware", "date": "2025-07-10", "items": [{"name": "tins of paint", "qty": 3, "unit_price": 650}, {"name": "brushes", "qty": 2, "unit_price": 85}], "total": 2120}
 
Lunch at Udupi Corner on 15 Oct 2025: 2 masala dosas at ₹90 each and a filter coffee for ₹30.
{"merchant": "Udupi Corner", "date": "2025-10-15", "items": [{"name": "Masala Dosa", "qty": 2, "unit_price": 90}, {"name": "Filter Coffee", "qty": 1, "unit_price": 30}], "total": 240}

The first reads 07/10/2025 month-first, as 10 July; written in India, it's 7 October. The second receipt has no total at all, so the model supplied one: 240. Two dosas at 90 and a coffee at 30 is 210.

Both failures look exactly like correct output: valid JSON, right shape, wrong values. A schema guarantees the shape. It says nothing about whether the values are true.

Where it falls over

The sibling question comes from the "Alice in Wonderland" paper by Nezhurina and colleagues (June 2024), which used it to show state-of-the-art models breaking down on a short common-sense problem: "Alice has 4 brothers and she also has 1 sister. How many sisters does Alice's brother have?" The answer is 2, Alice and her sister. At temperature 0:

Alice has 1 sister, which is Alice herself. Therefore, her brother does not have any sisters.

Then five more tries at temperature 0.7, final answers only: 2, 1, 1, 1 and 4. One right out of six. At this size, getting a question like this right depends on which sample you happen to get.

Arithmetic went better than I expected. A three-step word problem (14 notebooks at ₹45, 12% off above 10, paid with ₹1,000) came back correct at ₹445.60, with every step written out. Multiplying 4,787 by 356 was stranger. The model worked through the partial products and got 1,704,172, then wrote:

So, 4,787 × 356 = 170,4172. However, upon rechecking, it seems there was a miscalculation in the step-by-step breakdown. Let's correct it:

With a 400-token cap, the first run stopped partway through that second pass. With a higher cap, it repeated the same steps and boxed the same, correct, answer. A right answer you can't rely on costs nearly as much as a wrong one, because you have to check it either way. Arithmetic belongs in code.

Checks in code, and escalation for the rest

My first fix for the date was the obvious one: add "Dates in the text are written day first" to the prompt. On the rerun, the model still returned 2025-07-10. Telling a small model a rule doesn't mean it will apply it.

What worked was moving the judgement out of the model. Ask it to copy the date exactly as written, parse that in code with a day-first rule, and check the sum of the items against the total. Anything that fails a check goes to a bigger model.

def parse_date(text):
    text = re.sub(r"(\d)(st|nd|rd|th)\b", r"\1", text.strip())
    for fmt in ("%d %b %Y", "%d %B %Y", "%d/%m/%Y", "%d-%b-%Y"):  # day first, always
        try:
            return datetime.strptime(text, fmt).date().isoformat()
        except ValueError:
            pass
    return None
 
 
def problems(receipt, text):
    """Cheap checks in code. Anything returned is a reason not to trust the small model."""
    found = []
    if receipt["merchant"].lower() not in text.lower():
        found.append("merchant not in the text")
    if receipt["date"] is None:
        found.append(f"can't read date {receipt['date_as_written']!r}")
    added_up = sum(i["qty"] * i["unit_price"] for i in receipt["items"])
    if not receipt["items"] or abs(added_up - receipt["total"]) > 0.5:
        found.append(f"items add up to {added_up:g}, total says {receipt['total']:g}")
    return found
 
 
def escalate(text, reasons):
    # In a real system, this is where the large hosted model gets called.
    return {"route": "hosted", "why": reasons}
 
 
def handle(text):
    receipt = ask(text, schema=RECEIPT, system=SYSTEM)
    receipt["date"] = parse_date(receipt["date_as_written"])
    reasons = problems(receipt, text)
    if reasons:
        return escalate(text, reasons)
    return {"route": "local", "receipt": receipt}

The schema is the same as before with date swapped for date_as_written, and the system prompt now says "Copy the date exactly as it is written." The same five receipts:

local   Green Leaf Cafe, 12 Oct    '12 Oct 2025' -> 2025-10-12, total 200
local   Paid ₹1,499 to City Book   '3rd October 2025' -> 2025-10-03, total 1499
local   Sharma Hardware, 07/10/2   '07/10/2025' -> 2025-10-07, total 2120
local   Thanks for shopping at F   '19-Oct-2025' -> 2025-10-19, total 169
hosted  Lunch at Udupi Corner on   items add up to 210, total says 240

Four stay local and are now right, including the date. One goes up, with the reason attached. The escalate function is a stub on purpose: which hosted model you call matters less than deciding, in code, what gets sent there.

One lesson from getting here: checks only catch what they check. My first version compared totals but trusted the model's date, and the wrong date went through as a clean local answer. Every field you rely on needs either a check or a design where the model only copies and code interprets.

Local small model or hosted large model?

How I'd weigh it:

Privacy and offline. This is the strongest reason. The text never leaves the laptop, and it works with no network at all. If that's a hard requirement, the question isn't whether to go local but how to work within a small model's limits.

Cost. There's no per-token bill, but you pay in hardware and in your own time spent on checks and fallbacks. For low volumes, a hosted model may well cost less than the engineering.

Latency. Warm, a quarter of a second to the first token is quick. Cold, nine seconds is not, and long prompts are slow to read. Keep the model loaded and keep the prompts short.

Volume versus ceiling. One laptop gives you one request at a time at about 17 tokens a second. That's plenty for a person's own tools and too little for a busy service. And the ceiling is fixed: the prompt fix for the date didn't take, and I wouldn't bet on any prompt making the sibling puzzle reliable. When the task needs more than the model has, route it elsewhere rather than prompting harder.

What I'd take away

Try the small model on your own task before choosing. Twenty made-up examples told me more about qwen2.5:3b than a leaderboard would: it's reliable at picking from a list and at copying values into a schema, and unreliable the moment it has to work something out.

Let the model copy and let code compute. Dates, totals and anything else with a rule should be parsed or recalculated in code, where the rule actually holds.

Check before you escalate, and escalate with a reason. A small local model with a few honest checks in front of a large hosted one keeps most of the traffic local, and the hard cases still get a strong model. If privacy is why you went local, decide what escalation is allowed to send before you build it.

And read the licence of the exact model and size you'll deploy. The one I measured here is fine for experiments like this, and not for a product.