Skip to main content

Module infer

Module infer 

Source
Expand description

Lag-Llama inference engine with KV caching.

Two-phase inference:

  1. Prefill: full forward pass over context tokens, collects K/V cache per layer.
  2. Decode: per-step single-token forward pass in pure Rust (zero Candle overhead).

Structsยง

LagLlamaModel