Field SOP
Field SOP

Running 27B on 4GB VRAM: Disk, Not VRAM, Stops You First

A five-step SOP for layer-wise inference with airllm (October 12, 2026; figures are vendor self-reported): (1) respect the triple version lock - torch must be a CUDA 12 build (no prebuilt flash-attn wheel for CUDA 13), plus flash-attn, plus a transformers branch that depends on the model (4.56.x for Kimi K3, 5.8+ for Qwen3.8-27B, an in-tree qwen4_exp build for Flash-Next), verified with both nvidia-smi and torch.cuda.is_available(); (2) pick a tier from the README measurements - the most realistic entry point is Qwen3.8-27B at 3.33GB (RTX 3090); (3) the first run pays for the download (Flash-Next checkpoint around 360GB, K3 in the TB class), finish it once and keep the cache directory, and reclaim pre-split originals with delete_original=True; (4) watch VRAM peaks with nvidia-smi and manage expectations - layer-wise streaming moves far more bytes than it computes, the vendor publishes no tokens-per-second figure so none is invented here, and slow is the design, not a bug; (5) wire it into agent or API workflows: fine for one-shot task-shaped work, wrong for token-by-token interactivity; training takes a different path (frozen weights streamed layer by layer, adapters kept on GPU). Five dedicated trap sections: the version matrix, the hundred-GB-to-TB first download that stalls, the hidden disk and host-RAM gates (64GB-class RAM to start), structurally slow output, and K3's hard flash-attn plus CUDA 12 constraints. Repo: github.com/lyogavin/airllm (Apache-2.0).

Published October 12, 202610 min read
<!-- airllm-layered-inference-sop | sop | Running 27B on 4GB VRAM: Disk, Not VRAM, Stops You First -->

What This Route Actually Solves

Let us be clear about one thing first: AirLLM is not a quantization tool, not a distillation tool, and not a pruning tool. It does exactly one thing: it treats every layer of the model as a separately movable file, loading the layer it needs onto the GPU from disk or host memory, computing that layer, and swapping it back out. In the README's own words this is layer-wise streaming, and it is what lets a 70B-class model fit on a single 4GB card - and even puts a Kimi K3 (2.8T) class model under the 4GB mark.

Why is this worth a full SOP? Because it is the only option on the "not enough VRAM" list that does not degrade model quality. GGUF quantization costs quality, distillation trades away capability, CPU offload hybrids remain hostage to memory bandwidth. AirLLM concentrates the entire cost into a single axis: speed. Slow is acceptable. Slow and wrong answers is not.

The repository is github.com/lyogavin/airllm, under the Apache-2.0 license. Verified against the GitHub API on 2026-10-12: 36,747 stars, 3,829 forks, 160 open issues, last pushed 2026-10-12, with training support added in a 2026-09 update.

Worth separating from the other pieces on this site. Our local deployment tool comparison benchmarks five deployment tools; the Ollama local deployment SOP covers quantization tiers and VRAM pitfalls; the Qwen3.8-Flash-Next deployment SOP maps that model from hosted API to on-device. This article is a completely different branch: no quantization, no distillation, brute-force VRAM savings through layered streaming.

Boundary Statement

Every official AirLLM number is a VRAM footprint, not a speed figure. No hands-on benchmarking was done for this article, so no tokens-per-second number and no time-to-first-token number appears here. The speed discussion is limited to the structural reason for the slowness, because that is design, not a bug.

Step 1: Install the Environment and Lock the Versions

AirLLM's setup is more involved than most people expect, because three interlocking components have to agree. Get this wrong and every later step fails, usually with error messages that do not name the real cause.

The first lock is that torch must be a CUDA 12 build. This is a hard constraint: flash-attn has no prebuilt wheel for CUDA 13, and torch's build tracks CUDA, so a CUDA 13 torch means flash-attn will not compile.

The second lock is flash-attn. Kimi K3's model code mandates flash attention, and the install command is verbatim from the README:

bash
pip install compressed-tensors flash-attn

The third lock is that the transformers branch depends on the model, and this is the one people miss. Qwen3.8-27B needs transformers 5.8+. Kimi K3 needs transformers 4.56.x. Qwen3.8-Flash-Next, a Qwen4ExpForConditionalGeneration model, needs the in-tree qwen4_exp from transformers main, installed straight from source: pip install git+https://github.com/huggingface/transformers.git.

These are not three versions of one requirement. They are a matrix, one cell per model, so running 27B and then K3 in one virtual environment means switching transformers versions in between.

Verify twice, because one check misses half the failures. First, the driver and hardware layer:

bash
nvidia-smi

That confirms the card is present, the driver version, and current VRAM occupancy. Second, whether torch can actually see the card:

bash
python -c "import torch; print(torch.version.cuda, torch.cuda.is_available())"

The torch.version.cuda value should read 12.x. If it prints 13.x, the first lock already slipped - reinstall now while it is still cheap. A False from torch.cuda.is_available() points at the driver or the environment, not at AirLLM.

Expected result: nvidia-smi lists the GPU, and the second command prints something like 12.4 True. Failure signal: torch.cuda.is_available() returns False while nvidia-smi looks healthy - a PyTorch-driver mismatch, not an AirLLM bug.

Step 2: Pick a Target From the Measured Table

Count your card's VRAM, then read the README's measured table.

TargetVRAM footprint (README)End-to-end test hardwareExtra hard dependencies
Qwen3.8-27B (dense VL, Gated DeltaNet + Gated Attention)3.33GBRTX 3090transformers 5.8+
Qwen3.8-Flash-Next (125B, with roughly 51B n-gram embeddings)5.95GBRTX 4090in-tree qwen4_exp; checkpoint about 360GB
Kimi K3 (2.8T total, 104B active)3.72GBRTX 6000 Adacompressed-tensors, flash-attn, transformers 4.56.x
DeepSeek-V3 (671B)about 12GBREADME figure-
70B-class models (headline claim)4GB or lessREADME figure-

Qwen3.8-27B at 3.33GB is the most realistic starting tier. Three reasons: the footprint fits what a 4GB card can absorb, the checkpoint is tens of gigabytes rather than hundreds, and it needs one dependency lock instead of three.

Kimi K3's 3.72GB is seductive, but the bill comes due elsewhere: 2.8T parameters, 896 experts with 16 activated per token, a 1M token context, and a TB-scale disk footprint for a single node. Its weights also carry a modified MIT license, not the standard MIT, open-sourced in full on 2026-07-27 after a 2026-07-17 release. Read the LICENSE text yourself rather than assuming the terms you remember.

The Qwen3.8-Flash-Next route has its own quirk: the n-gram embedding tables are mapped to host memory, with the README stating a 64GB host is sufficient, and the decoder layers streamed. Its demand on system RAM is therefore stricter than its demand on VRAM.

Step 3: The First Inference, Where Download Time Dominates

Layered inference presupposes that every layer already exists locally as a file. The first run downloads the full checkpoint first: roughly 360GB for the Qwen3.8-Flash-Next checkpoint, TB-scale for Kimi K3. Interruption partway through is close to inevitable, so two disciplines:

  • Do not touch the cache directory once the download completes. Losing an HF cache to a cleanup means re-fetching hundreds of gigabytes. Treat the cache path as a fixed asset, not a scratch directory.
  • Reclaim the original with delete_original=True during the split phase. The parameter deletes the source checkpoint after AirLLM slices it into per-layer files, which for a 360GB-scale model is the single biggest disk reclaim available. It is also irreversible - confirm the split completed and loading works before letting it delete, or you end up with no original and no complete split.

Expected result: the first inference is preceded by a download measured in hours, or days at TB scale. Failure signal: the download keeps breaking at the same point. Before blaming the network, check whether your disk has room for the full checkpoint plus temporary files.

Step 4: Watch VRAM and Manage Speed Expectations

Watch VRAM with nvidia-smi in a second terminal. The peak is the only reliable evidence of whether your card is sufficient; the README figures are the maintainers' baseline on their own hardware, and your peak will not necessarily match.

On speed, this article gives you no number, because the official source gives none either. Inventing a tokens-per-second figure is the easiest way to fool yourself. What is real is the structural slowness: every token walks a cycle of load layer, compute, evict layer, and the bytes moved dwarf the arithmetic performed. Time to first token is therefore noticeably worse than a conventional model, and it is wrong for token-by-token interactive use. Batch tasks that produce a complete answer at once are the shape it is built for.

The test is simple: if you can accept hitting send, making coffee, and coming back to read the result, AirLLM fits. If you need a live conversation, it is the wrong tool.

Step 5: Wiring Up Agents and APIs, and the Training Fork

On the inference side there are two typical usages. The first is one-shot task workflows: feed in a batch of material, get back a report or an analysis, no dialogue in the middle. This is where AirLLM's value is clearest, because it puts a model tier on a 4GB card that was previously out of reach.

The second is wrapping it as an endpoint for an agent, with one caveat: the agent's interaction model must tolerate long latency. If an agent assumes every reply lands in a few hundred milliseconds, attaching AirLLM degrades its behavior - not from wrong answers, but from not being able to wait.

Training is a different path entirely. The README's 2026-09 update describes training as streaming frozen weights layer by layer while keeping the adapter resident in VRAM: only the small adapter parameters (LoRA and friends) stay on the card while the large weights keep flowing from disk. Two README figures: Qwen3.8-Flash-Next (125B) training under 6GB on an RTX 3060 Ti, and Qwen3.8-27B training around 2GB at sequence length 512. This logic differs entirely from inference and needs its own evaluation.

If you end up torn between quantizing and layered streaming, the AirLLM resource page maps the VRAM and deployment routes; the local coding model comparison picks a model by task type, and the Mellum 2.1 local deployment SOP is a far lighter parameter count.

Five Pitfalls

Pitfall one: treating the version matrix as a single lock. CUDA 12 build, flash-attn, and the matching transformers branch are three independent constraints with three different failure points. A CUDA 13 torch build breaks the flash-attn compile. transformers 5.8+ cannot find Kimi K3's model code. A stable release cannot supply qwen4_exp at all, because that module only exists in-tree on main. The three errors look nothing alike but share one root cause, so check the matrix first.

Pitfall two: treating the first download like a normal model download. 360GB is the Qwen3.8-Flash-Next scale; Kimi K3 is TB-scale. Bring the patience of a 7B download and you will interrupt it, clear the cache, and lose hundreds of gigabytes for nothing. There is exactly one correct posture: ample disk, ample time, an untouched cache.

Pitfall three: assuming slow means a broken install. This is the most common misdiagnosis. When a single token takes seconds, the first reaction is that something is misconfigured - yet layered streaming simply is that speed, and the maintainers never promised interactive latency. If you start doubting your environment, reread Step 4.

Pitfall four: treating training as an accessory of inference. Training is frozen-weight streaming with the adapter resident; inference is per-layer streaming of the full weights. Conflating them leads to false assumptions about VRAM budget, load timing, and optimizer state.

Pitfall five: reading K3 as a free lunch for "2.8T on 4GB." The VRAM figure is real, but the cost lives elsewhere: TB-scale disk, hard flash-attn dependencies, and a modified MIT license, which can differ from the standard text on redistribution terms.

Setup Checklist

  1. nvidia-smi lists the target GPU
  2. torch.version.cuda prints 12.x and torch.cuda.is_available() is True
  3. flash-attn installs cleanly (K3 also needs compressed-tensors)
  4. transformers version checked against the matrix: 5.8+ for 27B, 4.56.x for K3, in-tree qwen4_exp for Flash-Next
  5. Target model selected, with disk space for the full checkpoint plus temporary files
  6. Cache directory protected; delete_original=True enabled only after the split is verified
  7. nvidia-smi running alongside to watch peak VRAM
  8. Speed expectations calibrated to "structurally slow," validated with a one-shot task
  9. If running K3, the modified MIT license text has been read
  10. Training needs evaluated separately, without reusing inference assumptions

One-line close: AirLLM turns "not enough VRAM" from a dead end into a trade - no quantization, no distillation, layer by layer from disk, with no quality loss, and the entire bill charged to speed. So the first question is not whether your card can hold 3.33GB. It is whether your disk can hold 360GB first, and whether your workflow can tolerate hitting send and walking away.

FAQ

Q1: How fast is AirLLM? Is there an official tokens-per-second figure? A1: No. The official README publishes VRAM footprint numbers only - no tokens per second, no time to first token. The structural reason for the slowness is certain though: every token runs a load, compute, evict cycle whose bytes moved vastly exceed the arithmetic performed. It suits one-shot workflows, not token-by-token interaction.

Q2: Why must torch be a CUDA 12 build? Can another version work? A2: Because flash-attn has no prebuilt wheel for CUDA 13, and torch's CUDA build version is bound to it. Build torch against CUDA 13 and flash-attn will not compile. Verify by printing torch.version.cuda in the environment - it should read 12.x. Do this before installing flash-attn rather than debugging a failed compile afterwards.

Q3: What license does K3 carry? Can it be used commercially? A3: Kimi K3's weights carry a modified MIT license, not the standard MIT; the binding terms are the LICENSE text in the repository and on the model page, so do not assume the standard MIT wording applies. Read and archive the original before any commercial use. Also price in the operational cost: 2.8T total parameters, 104B active, 896 experts with 16 activated per token, a 1M context, and a TB-scale single-node disk footprint.

Q4: Can one machine run both 27B and K3? A4: Not from a single virtual environment. The transformers branches differ - 27B needs 5.8+, K3 needs 4.56.x. Switch versions between runs, or build two environments. Qwen3.8-Flash-Next is the most demanding case: it requires the in-tree qwen4_exp from transformers main.

Q5: Can a 4GB card really run a 70B or a 2.8T model? A5: The README's measured table gives specific figures: Qwen3.8-27B at 3.33GB end-to-end on an RTX 3090, Qwen3.8-Flash-Next 125B at 5.95GB on an RTX 4090, Kimi K3 2.8T at 3.72GB on an RTX 6000 Ada, and DeepSeek-V3 671B at roughly 12GB. The 70B-under-4GB claim is the README headline. All of these are VRAM footprints, and each holds only if your disk, your time budget, and your patience for speed are large enough.


References

All commands and figures here are a 2026-10-12 snapshot of the official README, not benchmarked on real hardware; no speed data is published officially, so no estimate is offered. Not investment advice.

This article is AI-assisted and human-edited. Last updated: 2026-10-12

FAQ

How fast is AirLLM? Is there an official tokens-per-second figure?
No. The official README publishes VRAM footprint numbers only - no tokens per second, no time to first token. The structural reason for the slowness is certain though: every token runs a load, compute, evict cycle whose bytes moved vastly exceed the arithmetic performed. It suits one-shot workflows, not token-by-token interaction.
Why must torch be a CUDA 12 build? Can another version work?
Because flash-attn has no prebuilt wheel for CUDA 13, and torch's CUDA build version is bound to it. Build torch against CUDA 13 and flash-attn will not compile. Verify by printing `torch.version.cuda` in the environment - it should read 12.x. Do this before installing flash-attn rather than debugging a failed compile afterwards.
What license does K3 carry? Can it be used commercially?
Kimi K3's weights carry a **modified MIT license**, not the standard MIT; the binding terms are the LICENSE text in the repository and on the model page, so do not assume the standard MIT wording applies. Read and archive the original before any commercial use. Also price in the operational cost: 2.8T total parameters, 104B active, 896 experts with 16 activated per token, a 1M context, and a TB-scale single-node disk footprint.
Can one machine run both 27B and K3?
Not from a single virtual environment. The transformers branches differ - 27B needs 5.8+, K3 needs 4.56.x. Switch versions between runs, or build two environments. Qwen3.8-Flash-Next is the most demanding case: it requires the in-tree qwen4_exp from transformers main.
Can a 4GB card really run a 70B or a 2.8T model?
The README's measured table gives specific figures: Qwen3.8-27B at 3.33GB end-to-end on an RTX 3090, Qwen3.8-Flash-Next 125B at 5.95GB on an RTX 4090, Kimi K3 2.8T at 3.72GB on an RTX 6000 Ada, and DeepSeek-V3 671B at roughly 12GB. The 70B-under-4GB claim is the README headline. All of these are VRAM footprints, and each holds only if your disk, your time budget, and your patience for speed are large enough.

Related

Field SOP

Transcribe 13-Minute Audio in 59 Seconds, Fully Local

A five-step local speech-to-text SOP (faster-whisper as the main engine, repo github.com/SYSTRAN/faster-whisper): pick by VRAM (24GB card large-v3+float16+beam 5; 8GB card int8_float16, step down to medium if tight; CPU-only small/base+int8), install (GPU needs CUDA 12 cuBLAS plus cuDNN 9, older stacks pin ctranslate2 3.24.0/4.4.0), a minimal transcription script, three speed-and-VRAM levers (fp16 to int8 saves about 1.6GB, batch 8 costs about 1.5GB for roughly 3.7x speedup, beam comes third), and advanced steps (Silero VAD, word timestamps, hand-rolled SRT, WhisperX diarization). One benchmark framing is stated everywhere: large-v2, beam_size=5, 13 minutes of audio on an RTX 3070 Ti 8GB — the original implementation takes 2m23s/4708MB, faster-whisper int8 59s/2926MB, batched 16s/4500MB; on CPU the piece honestly notes whisper.cpp fp32 (2m05s/1049MB) beats faster-whisper fp32, with no sweeping claims. The pitfall list includes never using pure int8 on GPU (CTranslate2 docs: compute capability 7.0/6.1+ required, older cards silently fall back to float32). A single closing paragraph points to NVIDIA's Nemotron 3.5 Transcribe as the new-engine option, leaving the benchmarks to the open-source column piece.

Oct 29, 202610 min read
Field SOP

A 3B Model That Chews 200k Pages a Day: DeepSeek-OCR Deploy SOP

A hands-on DeepSeek-OCR deployment SOP: the vision side squeezes a whole page into a few dozen to a few hundred vision tokens while the 3B MoE decoder keeps only about 570M active parameters — the official paper cites 200k+ pages of training data per day on one A100-40G, and a 24GB consumer card gets you in. The two official repos take different routes: environment and Transformers inference on generation 2 (Apache-2.0), upstream vLLM and API serving on generation 1's mature ecosystem (MIT). Verbatim commands for the cuda11.8+torch2.6.0+flash-attn 2.7.3 stack, a five-mode resolution token ledger (64 to 400+), the NGram anti-repetition processor with the 128821/128822 table whitelist, community-only VRAM guidance (nothing official), and a ten-item troubleshooting checklist.

Oct 25, 202610 min read
Field SOP

Can One GPU Run Matrix-Game 4.0? A Local Deploy SOP

A nine-step SOP for running Matrix-Game 4.0 locally on a single GPU, with every command and flag taken verbatim from the v4 subdirectory README (repository: github.com/SkyworkAI/Matrix-Game, file Matrix-Game-4/README.md). Step 1, align on requirements: model inference needs Linux, an NVIDIA GPU and Python 3.12 together, the reference A100 80 GB deployment uses PyTorch 2.12.0 / CUDA 13.2, and the repo ships a dedicated RTX 5090 runtime that does not auto-detect hardware, so the matching preset must be selected explicitly; the browser demo needs a Linux server that also builds the game and GStreamer pipeline while the Safari/Chrome client needs no GPU; standalone play and recording need macOS or Linux plus CMake, C++20, SDL2 and FFmpeg, with the legacy Windows entry retained but not validated for feature parity; and weights and assets are not shipped in the repo, so budget extra space for the Hugging Face Hub cache. Step 2, install inside the Matrix-Game-4/ subdirectory or everything after is wrong: git clone, cd Matrix-Game/Matrix-Game-4, python3.12 -m venv .venv and source .venv/bin/activate, then pip install torch==2.12.0 torchvision==0.27.0 --index-url https://download.pytorch.org/whl/cu132 as a matched pair, then pip install -e '.[inference,server,download,test]' with all four extras; FlashAttention is optional and only rtx5090-sage needs SageAttention while plain rtx5090 uses PyTorch SDPA. Step 3, downloads come as two manifests, game assets and model weights, pinned to revision 4419accdb7bb7881234c5914de8cd915266e0325 with --scene switchable across castle, racing and all, and one server process supporting only one scene model; the downloader verifies size and SHA256 against the manifests so interrupted runs can simply restart, shared T5/tokenizer and Wan VAE pull automatically from the official base, and matrix-game verify rechecks an installed scene. Step 4, offline rendering proves the model produces output: matrix-game render --scene ancient-egypt --input lowpoly.mp4 --resources resources --preset realtime --prompt-mode fixed --output outputs/egypt, where the input is a low-poly proxy stream rather than real footage and the output is rendered.mp4 plus a configuration report with FPS and frame count preserved; the three presets matter - realtime is for A100, RTX 5090 switches to rtx5090, and quality is slower yet is exactly the default when --preset is omitted. Step 5, play and record locally without loading a model: bash scripts/build_game.sh then bash scripts/play.sh --scene ancient-egypt, giving the two shooter scenes a 48 FPS, 2x simulation-time demo mode with F9 recording and Escape menu, and MATRIX_GAME_DATA_DIR overriding the default data directory; racing uses build_racing.sh with 1x simulation time, and the browser demo does not enable the shooters' 2x mode either, advancing 1/--fps seconds per captured frame. Step 6, the browser demo splits server and client: mediamtx configs/mediamtx.yml in one terminal and CUDA_VISIBLE_DEVICES=0 bash scripts/serve.sh --scene ancient-egypt --resources resources --preset realtime --fps 40 in another, then open http://127.0.0.1:8025/, with --reference-only testing controls, HUD, audio and transport without loading a model; --fps 40 is a capture cadence target, not a throughput or latency guarantee. Step 7, the two RTX 5090 presets: rtx5090 needs no extra installation while rtx5090-sage requires building SageAttention 2.2.0 with the supplied CUDA-graph patch and CUDA 13.2 at git checkout eb615cf6cf4d221338033340ee2de1c37fbdba4a; both reuse the same scene checkpoints and shared resources, converting DiT block linear weights to FP8 at load with activations quantized during inference while embeddings and the output head stay BF16, and startup warms kernels and captures CUDA graphs so the first launch is slower than steady state. Step 8, acceptance in two forms: a keypress producing a picture in the browser demo (officially about under 150 ms, not independently reproduced) or an offline render turning a low-poly video into a finer output, with the three performance accounts written side by side (official 480p 60 FPS, 83 FPS and 18.5 GiB; the project page panel still at 0; and the README's about 38-39 FPS on an A100 at 864 x 480), remembering that model throughput, pipeline throughput, presentation frame rate and end-to-end latency are four different things. Step 9, six pitfalls: one process supports one session and one scene model; it binds to loopback by default and must not be exposed to the internet, since the official statement calls it a trusted single-user demo rather than a public cloud-gaming service; --fps 40 is not a throughput guarantee; Windows is unvalidated; disk must accommodate the Hub cache; and VRAM splits into an A100 80 GB reference tier and an RTX 5090 consumer tier with the 5080 and 5070 still pending.

Oct 16, 202610 min read