Can you run a local LLM without a graphics card?

Yes. Every mainstream local runtime can execute a model entirely on the CPU using system RAM. It is several times slower than a GPU, but a small model on CPU is fast enough for file questions, summarising and drafting. A graphics card is an upgrade path, not an entry requirement.

What happens if a model does not fit in VRAM?

The runtime splits the model, keeping some layers on the GPU and running the rest on the CPU. This degrades gracefully until VRAM headroom runs out, at which point throughput falls sharply. On our RTX 3080 test system, moving from 13 GPU layers to 14 dropped generation from 52.0 to 37.2 tokens per second, and 17 layers collapsed to 5.0 tokens per second.

Does quantisation reduce VRAM requirements?

Yes. A 16-bit model needs about 2 GB per billion parameters, an 8-bit model about 1 GB, and a 4-bit model about 0.6 GB. Four-bit quantisation is the usual default for consumer hardware because it roughly triples the model size that fits with a modest quality cost.

How much VRAM do you need to run a local LLM?

Q: How much VRAM do you need to run a local LLM?

At 4-bit quantisation a model needs roughly 0.6 GB of VRAM per billion parameters, plus the KV cache and about 1 GB of headroom for the operating system. In practice: 8 GB of VRAM comfortably runs a 7–9B model, 12 GB runs a 12–14B model, 16 GB runs a 20B-class model, and 24 GB runs a 30B-class model. Models can also run on the CPU with no graphics card at all, more slowly.

About 0.6 GB per billion parameters at 4-bit, plus the KV cache, plus roughly a gigabyte you do not get to use. The interesting part is what happens when a model nearly fits.

3 August 2026· Lemonade· 9 min read

The short answer

8 GB of VRAM runs a 7–9B model comfortably. 12 GB runs a 12–14B model. 16 GB runs a 20B-class model. 24 GB runs a 30B-class model, or a smaller one alongside a vision model. No graphics card at all still works — the model runs on your CPU and system RAM, several times slower but perfectly usable for file questions and drafting.

The arithmetic

A model's weights dominate its memory footprint, and the footprint is just parameter count multiplied by bytes per parameter. Quantisation is the lever that moves it:

Weight memory per billion parameters
Precision	Per 1B params	Typical use
FP16 / BF16	≈ 2.0 GB	Training and datacenter serving
8-bit (Q8)	≈ 1.0 GB	Quality-sensitive local work with VRAM to spare
4-bit (Q4_K_M)	≈ 0.6 GB	The consumer default
3-bit and below	≈ 0.45 GB	Desperation; quality degrades noticeably

So an 8B model at 4-bit is around 4.8 GB of weights. A 14B is around 8.4 GB. A 30B is around 18 GB, and a 70B is around 42 GB — which is why 70B-class models are not a consumer single-GPU proposition without offloading most of the model to system RAM.

Four-bit is the default for a reason: it roughly triples the model that fits on a given card, and for chat, summarising and document work the quality cost is small compared with dropping to a model a third of the size. Below about 4 bits per weight the trade stops being favourable.

What is actually in VRAM

Weights are the biggest line but not the only one. Three others matter:

The KV cache. Every token in the context window stores keys and values for every layer. It is small at 4k context and can reach several gigabytes at long contexts on larger models. Modern architectures use grouped-query attention, which cuts it substantially, and most runtimes can quantise the cache to 8 bits — the single most effective knob when you are a few hundred megabytes short.
Compute buffers. Activations, scratch space and the batch being processed. Usually a few hundred megabytes, larger when you process a long prompt in one go.
Everything else on your desktop. Windows and most Linux desktops both composite on the GPU. A browser with hardware acceleration, a second monitor and a video call can easily hold a gigabyte. Plan to leave about 1 GB free, and more if you keep a browser open — which you do.

The practical formula is therefore:

VRAM_needed ≈ params_B × 0.6 GB + KV cache + 0.5 GB buffers + 1 GB desktop A planning estimate at 4-bit quantisation, not a specification. Real footprints vary by architecture, attention scheme, context length and runtime. Measure on your own machine before trusting any figure, including this one.

Size table by card

What a card runs well, assuming 4-bit weights and a working desktop underneath:

What fits, by VRAM
VRAM	Comfortable model size	What that gets you
None (CPU)	3–4B	Chat, file search and summarising, slowly but reliably
6 GB	7–8B, short context	One capable general model, nothing else resident
8 GB	7–9B	Chat, images and tool use in a single model
10–12 GB	12–14B	A general model plus a dedicated vision model
16 GB	20B-class	Long contexts without cache anxiety
24 GB	30B-class	A large model, vision and code together

These are the same bands behind the requirements table on our home page, where they are expressed as disk totals for a working Lemonade install.

When a model nearly fits

This is the part most guides skip, and it is where the real performance lives.

When a model is too large for VRAM, runtimes do not give up. They split it: some transformer layers execute on the GPU, the rest on the CPU. The intuition is that this degrades smoothly — put 60% of the layers on the GPU and get roughly 60% of the benefit.

It does not degrade smoothly. While building the tiered runtime described in our research report, we measured layer placement for a 35B sparse model on an RTX 3080 with 10 GB of VRAM and 32 GB of system RAM. The model has 40 routed layers. We varied how many went to the GPU:

Generation throughput by layer split, RTX 3080
GPU layers	CPU layers	Throughput
13	27	52.03 tok/s
14	26	37.24 tok/s
17	24	5.01 tok/s

One extra layer cost 28% of throughput. Four more cost 90%. The card was not out of memory in the sense of refusing to allocate — it was out of reserve, and once allocation pressure starts, the runtime spends its time on memory management rather than arithmetic.

What to take from this

The best local setup is rarely the one that pushes the most onto the GPU. If generation feels slow, try moving fewer layers to the card, not more — and leave roughly one layer's worth of VRAM unallocated. On our test machine that single change was worth more than every caching scheme we built afterwards.

Sparse models change the sum

Mixture-of-experts models break the link between how big a model is and how much computation it does per token. The 35B model in our tests activates about 3B parameters for any given token — the rest of the weights sit idle until the router picks them.

That is good news for capability per token generated and bad news for memory planning, because idle weights still have to live somewhere. A sparse 35B at 4-bit is still roughly 23 GB on disk. What sparsity buys you is that this 23 GB can be spread across VRAM, system RAM and even NVMe, with only the active fraction needing to be reachable quickly.

This is why our own requirements table lists “10 GB VRAM + 32 GB system RAM” as a route to a 35B-class model. It is not that the model fits in 10 GB. It is that a 10 GB card plus enough RAM, with the right layer placement and prefetching, can run it at a usable speed. Our measured gains there came from unglamorous things: correct placement, multi-token prediction (26.17 → 35.60 tok/s), and prefetching the exact weights the next step will need into RAM (12.75 → 14.66 tok/s).

Running with no GPU

A graphics card is an accelerator, not an entry ticket. Every mainstream local runtime executes models on the CPU using system RAM, and a 3–4B model on a recent CPU generates fast enough to read along with.

Two things matter more than core count. The first is memory bandwidth: generation is bandwidth-bound, so dual-channel RAM at a decent speed beats extra cores. The second is not using all your cores — in our multi-token prediction tests, eight threads reached 40.23 tok/s where twelve reached 38.45 and sixteen reached 34.29. Past a point, threads contend for the same memory bus and add synchronisation instead of throughput.

How Lemonade decides

We built Lemonade so that none of the above is the user's problem. On first run it checks your GPU, VRAM and system RAM, then downloads models sized to what you actually have — and sets the layer split with reserve left over rather than maximising GPU residency. It unloads one model to make room for another when a request needs a different one.

The figures it shows you are estimates derived from active parameters, context, RAM fit, CPU resources and GPU placement. Once you benchmark a model on your own machine, your measured numbers should replace ours. Nobody's table beats your hardware.

Corrections and disagreements: support@aionx.aionapp.org. Benchmark figures come from the paired runs described in Nanite for Experts.