Benchmarking Local LLMs: Where VRAM Limits, MoE, and PLE Architecture Collide

Aug 8, 2026 · Article

Running large language models locally has evolved rapidly with new architectural patterns like Mixture of Experts (MoE) and Parameter-Level Effective (PLE) architectures. To understand real-world inference characteristics across consumer hardware, I benchmarked a series of local models using Ollama.

The test suite covers two distinct hardware environments:

  1. GPU Setup: NVIDIA GeForce RTX 4060 Ti (16GB VRAM), Intel Core i7-12700F, 32GB RAM testing gemma4:26b (MoE), gemma4:12b (dense), and mistral-small3.2:24b (dense).
  2. CPU-only Setup: Intel Core i5-13420H (16GB RAM, no discrete GPU) testing the lightweight gemma4:e4b and gemma4:e2b models.

The two setups sit side by side as reference points for different bottleneck classes (high-bandwidth VRAM vs. system RAM bandwidth, and dense vs. sparse/PLE architectures).

Benchmark Results

1. GPU Results (NVIDIA GeForce RTX 4060 Ti 16GB VRAM, Intel Core i7-12700F, 32GB RAM)

All models tested on GPU were quantized using standard Q4_K_M for an apples-to-apples comparison on quantization level.

ModelPlacementVRAM UsedPrompt Eval RateGeneration RateLoad Time (Cold)
gemma4:26b (MoE, 26B A4B)56% CPU / 44% GPU15 ± 1 GB53 ± 13 tok/s32 ± 1 tok/s22s ± 1s
gemma4:12b (dense)100% GPU13 ± 1 GB113 ± 34 tok/s32 ± 1 tok/s1s ± 0.5s
mistral-small3.2:24b (dense)63% CPU / 37% GPU13 ± 1 GB571 ± 240 tok/s4 ± 1 tok/s1s ± 0.5s

2. CPU-only Results (Intel Core i5-13420H, 16GB RAM)

Tested on battery-efficient mobile silicon with integrated graphics, running fully on CPU:

ModelPlacementPrompt Eval RateGeneration RateLoad Time (Cold)
gemma4:e4b100% CPU26 ± 4 tok/s10 ± 4 tok/s2s ± 0.5s
gemma4:e2b100% CPU72 ± 5 tok/s21 ± 5 tok/s2s ± 0.5s

Model Specifications (ollama show)

Here is the structural breakdown of each model retrieved via ollama show:

Fieldgemma4:e2bgemma4:e4bgemma4:26bgemma4:12bmistral-small3.2:24b
Architecturegemma4gemma4gemma4 (MoE, 26B A4B)gemma4 (dense)mistral3 (dense)
Total Parameters5.1B8.0B25.8B11.9B24.0B
Active Params / Token~2.3B effective (PLE)~4.5B effective (PLE)3.8B11.9B (all active)24.0B (all active)
Context Length131,072131,072262,144262,144131,072
Embedding Length1,5362,5602,8163,8405,120
QuantizationQ4_K_MQ4_K_MQ4_K_MQ4_K_MQ4_K_M
VisionYesYesYesYesYes
AudioYesYesNoYesNo
ToolsYesYesYesYesYes
ThinkingYesYesYesYesNo
Default Temperature1.01.01.01.00.15

Key Findings & Architecture Insights

1. Cold-Start Overhead on Large Footprints

The cold start for gemma4:26b required 22s ± 1s while streaming 15 ± 1 GB of model weights across PCIe into VRAM and system memory. On the Intel Core i5-13420H (16GB RAM, no discrete GPU), e2b and e4b had much faster cold starts (2s ± 0.5s) due to their smaller footprint.

2. Full GPU Offload Doubles Prompt Evaluation

When a model fits entirely into VRAM (gemma4:12b at 100% GPU offload), prompt evaluation hits 113 ± 34 tok/s, roughly doubling the split-memory gemma4:26b (53 ± 13 tok/s).

Interestingly, on CPU-only, gemma4:e2b reached 72 ± 5 tok/s prompt eval. The PLE architecture minimizes the active computation path during prompt ingest, making it feel remarkably snappy even on the Intel Core i5-13420H processor.

3. Generation Speed is Memory-Bandwidth Bound

A common misconception is that generation rate correlates linearly with active parameter count. However:

  • gemma4:12b (11.9B active params, 100% GPU) produced 32 ± 1 tok/s.
  • gemma4:26b (3.8B active params, 56% CPU / 44% GPU) produced 32 ± 1 tok/s.

Why did the MoE model match the fully offloaded 12B model?

  • In gemma4:26b, only 3.8B parameters are activated per token step, dramatically reducing per-token memory read demands over the PCIe/system RAM bus.
  • In gemma4:12b, all 11.9B parameters must be read from VRAM on every single token step.

The two differing bottlenecks (slower PCIe/RAM fetching 3.8B params vs. faster VRAM fetching 11.9B params) converged at the exact same output generation rate of 32 ± 1 tok/s.

4. The 262K Context Tax on VRAM Allocation

Both gemma4:26b and gemma4:12b support a massive 262,144 token context window. When Ollama calculates layer offloading, it pre-allocates KV cache headroom for this context window. This large default KV allocation is why 26b split 56/44 across CPU and GPU on the 16GB NVIDIA GeForce RTX 4060 Ti rather than fitting more layers into VRAM.

5. Why Mistral Small 3.2 24B Dropped to 4 ± 1 tok/s

Despite having fewer total parameters (24.0B) than Gemma 4 26B (25.8B), mistral-small3.2:24b produced a sluggish 4 ± 1 tok/s.

Checking ollama ps revealed the culprit:

  • Mistral Small 3.2 uses a wider embedding dimension (5120) and a dense transformer architecture.
  • Its uncompressed on-disk footprint at Q4_K_M is ~36 GB.
  • On the 16GB NVIDIA GeForce RTX 4060 Ti, Ollama was forced to place 63% on CPU RAM and only 37% on GPU.
  • Because every token generation step must sweep all 24B dense parameters across the slow system RAM bus, generation throughput ground to a halt. In contrast, its prompt eval remained high (571 ± 240 tok/s) because batch matrix multiplication is compute-bound and saturates parallel execution units.

Actionable Recommendations

1. Right-Sizing CPU-only Deployments

If running on an Intel Core i5-13420H (16GB RAM, no discrete GPU):

  • Use gemma4:e2b as the default interactive driver: at 21 ± 5 tok/s, it feels fluid for conversational chat and agentic workflows.
  • Reserve gemma4:e4b (10 ± 4 tok/s) for tasks requiring deeper reasoning or higher fidelity tool execution.
  • Avoid running 12b or 26b CPU-only on 16GB machines; memory starvation and single-digit token rates will result.

2. Clamp num_ctx Unless Long-Context is Essential

The default 131k–262k context sizes consume gigabytes of VRAM/RAM for KV caches. If working on standard tasks, restrict the context window in your Modelfile or API call:

FROM gemma4:26b
PARAMETER num_ctx 16384

Clamping num_ctx to 16k or 32k frees several gigabytes of VRAM, allowing Ollama to offload more layers to the GPU.

3. Leverage Multi-Token Prediction (MTP) Drafters

Google released dedicated speculative-decoding drafter weights alongside Gemma 4. MTP drafters predict multiple future tokens per step and verify them in a single forward pass, providing up to ~3x generation throughput gains on memory-bandwidth-bound setups.

Ollama introduced native Gemma 4 MTP support in v0.23.1.

4. Switch to Quantization-Aware Training (QAT) Checkpoints

For memory-constrained setups, standard post-training quantization (PTQ) can degrade output quality at low bitrates. DeepMind’s official QAT checkpoints retain higher fidelity per byte, making 3-bit or 4-bit runs much more capable on limited RAM.

References & Further Reading