Skip to main content

Managing Model Context Length

By default, GoingMerry assigns a model context size of 2048 tokens for local execution to preserve VRAM limits. However, modern model architectures (like Llama 4, Gemma 4, or DeepSeek) support context windows of 8K, 32K, or even 128K tokens.

This guide explains how to increase or decrease context lengths.

1. Modelfile Configuration​

To set a persistent context length for a customized model, use the num_ctx parameter inside a Modelfile:

FROM gemma4:latest

# Set a 8192 token context length size
PARAMETER num_ctx 8192

Compile the new tag:

merry create gemma4-8k -f ./Modelfile

2. API Execution Overrides​

If you want to set the context length dynamically per API request, pass num_ctx inside the options request body:

curl http://localhost:11434/api/generate -d '{
  "model": "gemma4",
  "prompt": "Summarize this long document...",
  "options": {
    "num_ctx": 16384
  }
}'

3. Memory Footprint and KV-Cache​

Increasing the context length significantly increases the size of the model's key-value (KV) cache.

VRAM Cost Estimation​

As context length scales, VRAM consumption increases quadratically or linearly depending on the attention mechanism:

  • 2K Context: ~200MB KV Cache.
  • 8K Context: ~1.2GB KV Cache.
  • 32K Context: ~6GB+ KV Cache.

If VRAM is exceeded, GoingMerry will offload layers to standard CPU memory, which may reduce inference speed. Ensure you have adequate memory headroom before executing large context queries.