Skip to main content

GPU Acceleration & Memory Management

GoingMerry is engineered to maximize throughput by dynamically mapping model calculations onto graphics hardware.

1. Dynamic Layer Sharding​

When a model is loaded:

  1. GoingMerry checks the total available VRAM on your system.
  2. It calculates the memory footprint of the target model's weight matrices and kv-cache.
  3. If the model fits entirely in VRAM, 100% of the layers are offloaded to the GPU.
  4. If VRAM is constrained, GoingMerry shards the model, offloading a portion of the layers to standard CPU/RAM system memory.

You can inspect this sharding layout using:

merry ps

2. Multi-GPU Layouts​

If your host machine features multiple graphics cards, GoingMerry will automatically distribute layer computations across them.

Restricting Execution​

If you wish to restrict execution to a single target GPU, set device environment variables:

# restrict to the first GPU card
export CUDA_VISIBLE_DEVICES=0
merry serve

3. GPU VRAM Allocation Limits​

In multi-tenant or heavy development environments, you may wish to limit the amount of GPU memory GoingMerry utilizes to leave headroom for other graphics software (such as game engines, viewport rendering, or training scripts):

  • Modelfile Parameter: Set the parameter num_gpu to limit target layers. 
    PARAMETER num_gpu 24
  • Execution Threshold: Set the environment variable MERRY_GPU_OVERHEAD to preserve a strict VRAM buffer (e.g. 1.5GB): 
    export MERRY_GPU_OVERHEAD=1500000000
    merry serve