How to run an 8B LLM on a laptop: hardware, quantization and tools

OOpen-weights models in the 7 to 9 billion parameter range are now good enough for a surprising amount of real work: summarising documents, drafting emails, extracting fields, answering questions over your notes and helping with code. And they run on an ordinary laptop. A local model costs nothing per token, works offline, and keeps your data on your machine.

This guide explains what hardware you need to run an 8B model locally, how to choose a quantization level, which tools to use, how to estimate speed before you download anything, and which tasks local models handle well.

How much memory does an 8B model need?

A model's weights must fit in memory, plus some room for the context (the key-value cache) and the operating system. The weight size is simple arithmetic: parameters × bytes per parameter.

Figures are approximate: quantized formats store some extra scaling data, so real files are slightly larger than the naive calculation. The context also needs memory; long contexts of tens of thousands of tokens can add several gigabytes.

The practical takeaway: a 4-bit or 5-bit 8B model runs comfortably on a 16 GB laptop, and squeezes onto 8 GB if you close other applications and keep context short.

Apple Silicon, NVIDIA or CPU only?

• Apple Silicon Macs use unified memory shared by the CPU and GPU, so the whole model can sit in "GPU memory" without a separate graphics card. This makes MacBooks unusually good for local inference.
• Windows and Linux laptops with an NVIDIA GPU run fastest when the model fits entirely in the GPU's VRAM. An 8 GB card fits a 4-bit 8B model with modest context; layers that do not fit can be offloaded to system RAM at a speed cost.
• CPU-only works, especially with recent processors, but expect noticeably slower generation.

Estimate tokens per second before you start

Generating text is usually limited by memory bandwidth, not raw compute. To produce each new token, the hardware must read essentially all of the model's weights from memory once. That gives a useful upper bound:

max tokens/sec ≈ memory bandwidth (GB/s) ÷ model size in memory (GB)

Example: 100 GB/s laptop, 4.9 GB 4-bit model
100 ÷ 4.9 ≈ 20 tokens/sec (theoretical ceiling; real numbers land below it)

Check your machine's memory bandwidth in its specifications. Two consequences follow. Smaller quantizations are faster, not only smaller. And a laptop with higher memory bandwidth generates faster at the same model size, even with a similar CPU.

Prompt processing, the time before the first token appears, depends more on compute and grows with input length. Long documents take longer to start answering. For a proper method to measure both, see designing a latency benchmark that matches what users feel.

Choosing a quantization level

Quantization stores weights with fewer bits. For 8B-class models, 4-bit and 5-bit variants are the sweet spot for laptops. Fluent writing survives quantization well; exact recall, arithmetic and code degrade first. Our full guide, quantization without tears, explains what you lose at each level.

A simple rule: start with a 4-bit "medium" variant (for example a Q4_K_M file in the GGUF format), test it on your own tasks, and step up to 5-bit or 6-bit only if you see errors that matter and have the memory.

Tools to run local models

Most tools expose a local HTTP endpoint compatible with common client libraries, so you can point existing code at your laptop by changing a base URL.

# Example with Ollama: download and chat with a model
ollama pull <model-name>
ollama run <model-name> "Summarise this paragraph in one sentence: ..."

Getting the most out of a laptop

A few settings make a large difference to the experience:

• Keep the context window modest. Many tools default to a context length far larger than you need. A smaller context uses less memory and starts answering faster.
• Close memory-hungry applications. Browsers with many tabs can easily take several gigabytes, which pushes the model into swap and slows generation dramatically.
• Plug in. Laptops often throttle on battery. Expect lower speeds unplugged and noticeable battery drain during long generations.
• Watch the temperature. Fanless laptops slow down under sustained load. Short bursts of work are fine; long batch jobs are better on a desktop or a server.
• Use the GPU path. On Macs, confirm your tool is using Metal; on NVIDIA machines, confirm layers are offloaded to the GPU. Running entirely on the CPU by accident is the most common cause of disappointing speed.
• Keep a model per job. A small, fast model for autocomplete and classification and a larger one for drafting often beats one compromise model.

What local 8B models are good at

Small models are not miniature frontier models. They have less knowledge and weaker multi-step reasoning. They are, however, very capable at narrow, well-specified tasks:

• classification and routing (is this email a complaint, a question or spam?);
• extraction of fields from documents with a clear schema;
• summaries and rewrites of text you provide;
• retrieval-augmented question answering over your own notes, where the facts are in the context;
• private drafting of sensitive material you do not want to send to a cloud API.

They struggle with open-ended factual questions (they hallucinate more), long multi-step plans and hard maths. Our article on where small models quietly win explains how to combine a local model with a larger hosted one, and distillation explained for builders covers how small models get trained to be good at narrow jobs.

A simple test plan for your laptop

1. Pick 20 to 30 real tasks from your work.
2. Run them with a 4-bit 8B model and note quality, time to first token and tokens per second.
3. Run the same tasks with a hosted model you trust as a reference.
4. Mark which tasks the local model handles acceptably. Those are your local workload.
5. Re-test when new open-weights models ship; the frontier of what fits on a laptop keeps moving.

Frequently asked questions

Can I run an LLM on a laptop with 8 GB of RAM?

Yes, a 4-bit quantized 7B or 8B model can run in 8 GB of RAM, but it will be tight. Close other applications, keep the context short, or choose a smaller 3B to 4B model for a smoother experience. 16 GB is comfortable.

How fast is a local LLM on a laptop?

Speed depends mostly on memory bandwidth and model size. Divide your memory bandwidth in GB/s by the model's size in GB for a rough ceiling on tokens per second; real speeds are somewhat lower.

Are local LLMs private?

Inference runs entirely on your machine, so prompts and outputs do not leave it unless a tool sends telemetry or you connect it to online services. Check each tool's settings and documentation.

Which quantization should I choose for an 8B model?

Start with a 4-bit medium-quality variant such as Q4_K_M. If you see errors on tasks that need precision, and you have spare memory, try 5-bit or 6-bit.