Open Source
Open Source

On-Device AI: 3GB RAM Runs a 35B MoE Model

ModelBest and the Tsinghua THUNLP-backed OpenBMB team released MiniCPM-V 4.7 (official repo changelog October 12, 2026; press coverage October 13), billed officially as the first open MoE multimodal model to run losslessly and smoothly on-device, in 35B-A3B (35B total / 3B active per token) and 1B variants. Speed per the official documentation only: 13+ tokens/s at about 3 GB of memory on an iPhone 17 Pro Max (third-party 15+/17+ figures conflict with the docs and are not adopted); the 3 GB is the model's runtime footprint, not the device floor - expert weights stay on flash and stream in on demand (MoE-SSD), and the official app hardware table recommends 12 GB+ RAM for 35B-A3B and 4 GB+ for 1B. The core trio: LES (Lookahead Expert Scheduling - cross-layer routing predictions affect prefetching only, never the compute routing, so misses cost nothing in correctness; officially a 5x real-device speedup, claimed as the industry's first open-source implementation), fine-grained layered quantization, and DSpark speculative sampling; vision keeps hybrid 4x/16x token compression and adds intra-ViT early compression based on LLaVA-UHD v4. The ecosystem: FP8/int4/AWQ/GPTQ/GGUF weights on Hugging Face and ModelScope; inference via SGLang, vLLM, llama.cpp and Ollama; fine-tuning via SWIFT and LLaMA-Factory plus the co-open-sourced SRFT self-replay method; the official inference path runs on OpenBMB/llama.cpp-omni (MIT, a fresh llama.cpp-based repo created October 13); official apps for iOS, Android and HarmonyOS live in the sibling repo OpenBMB/MiniCPM-V-Apps; the Day-0 chip list per the docs is Intel, AMD, Arm, Rockchip and Houmo (FlagOS adds Ascend, Moore Threads, MetaX - claimed). License: weights and code are unified under Apache-2.0 (docs/project_en.md verified: "weights and code are open-sourced under the Apache-2.0 license"); the registration questionnaire is optional and not a commercial precondition. GitHub API snapshot October 15, 2026: 26,753 stars / 2,096 forks, pushed October 14. Repo: github.com/OpenBMB/MiniCPM-V.

Published October 15, 202610 min read
<!-- minicpm-v-4-7-opensource-resource | open-source | On-Device AI: 3GB RAM Runs a 35B MoE Model -->

Three gigabytes of memory, running a model with 35 billion total parameters. That is not a cloud inference bill — it is something happening on a phone. Around October 13, 2026, ModelBest and the OpenBMB team, which grew out of Tsinghua University's THUNLP lab, released MiniCPM-V 4.7; the official repository changelog is dated October 12, and Chinese media coverage followed in a burst. The headline claim can be compressed into one sentence: according to the official description, this is the first open-source MoE multimodal model to run on-device with lossless quality and smooth speed, shipped in two variants — 35B-A3B and 1B. The first one is the 35-billion-parameter heavyweight that fits in 3GB of RAM.

The project lives at OpenBMB/MiniCPM-V, repository address: github.com/OpenBMB/MiniCPM-V. As of a GitHub API snapshot taken on 2026-10-15, this repository, created in January 2024, holds 26,753 stars and 2,096 forks, with the latest push on 2026-10-14.

Zoom out and the trajectory matters: "put large models on phones" has been the MiniCPM series' home turf since 2024 — the first generation's technical report was titled "MiniCPM-V: A GPT-4V Level MLLM on Your Phone." What changed with 4.7 is the architecture class: earlier generations proved small dense models could be useful; this release takes on MoE, a design widely assumed to be unrunnable on a phone.

Why Running MoE on a Phone Is Hard

On paper, MoE (Mixture of Experts) looks like a free lunch: total parameters set the capability ceiling, while each token activates only a small slice of them. The 35B-A3B variant of MiniCPM-V 4.7 combines 35 billion total parameters with 3 billion activated per token. By activation count it does the work of a 3B model; by capability it draws on a 35B ceiling.

The catch is storage. The official documentation is candid about this: an MoE model's storage requirement is not its active-parameter count. The router can pick different experts at every layer for every token, and the full expert weights far exceed phone RAM, so they sit on flash storage and get loaded on demand. The documentation acknowledges this makes inference "possible" — but repeated loading stalls slow decoding until real-time conversation is off the table.

The industry's other approach is subtraction: fix a subset of experts for an entire prompt to reduce loading frequency. The cost, per the documentation: this pre-decides routing outcomes, experts outside the subset get excluded, and capability degrades. Faster answers, dumber answers.

So the real problem was never "can it run" but the team's repeated bar: lossless and smooth — no capability discount, sufficient throughput. It is a three-way optimization under memory capacity, storage bandwidth, and model capability constraints.

LES: Letting Expert Weights Start Early

ModelBest's answer is called LES — Lookahead Expert Scheduling. The official analogy is a relay runner who starts accelerating before the baton arrives.

LES does three things. First, cross-layer routing prediction: a predictor guesses which experts the next layer will need and preloads them into memory. Second, priority scheduling: when the real router makes its selection, computation goes first to the selected experts that are already prefetched, eliminating the wait. Third, overlapping compute and loading: experts that have arrived are being computed while the ones still in flight keep loading — two pipelines running in parallel.

Here is the design detail that decides success or failure, and that is easy to miss: prediction affects prefetching and scheduling only; it never changes the compute routing. The model picks whichever experts it would have picked anyway, and any expert the predictor missed is loaded and evaluated as usual. That is what "lossless" means in engineering terms — LES optimizes the timing of weight arrival, not the model's decision path. You gamble a little memory bandwidth on prediction hit rate, and losing the bet costs nothing but a slower load.

By the official account, LES delivers a 5x speedup on real devices, and the team describes it as the industry's only open-source solution of its kind. Two companions round it out: fine-grained layer-wise quantization shrinks expert weights' storage footprint and memory traffic so loading fits on-device constraints, and DSpark speculative sampling attacks decoding speed from the other direction — guessing several tokens at once. The three together produce that 3GB runtime figure for 35B-A3B.

One more piece deserves note: the vehicle for LES is OpenBMB/llama.cpp-omni (github.com/OpenBMB/llama.cpp-omni), described as "Omni inference in C/C++," a llama.cpp-based fork under the MIT license. The GitHub API snapshot of 2026-10-15 shows it was created on 2026-10-13 and sits at 16 stars and 2 forks — brand new, but this is the official main path for running 4.7 on devices.

Two Sizes and the Scoreboard

MiniCPM-V 4.7 ships in two sizes. The 35B-A3B is the series' first MoE model and the flagship on-device experience; the 1B is extreme compression for the long tail of existing devices.

The speed figure should be quoted exactly as the official documentation states it: 35B-A3B reaches a decoding speed of 13+ tokens/s with just 3GB of memory on an iPhone 17 Pro Max — the original wording, consistent across the README and the model document minicpm_v4dot7_en.md. Third-party write-ups have cited 15+ and 17+ tokens/s; neither matches the official document, so neither is used here. The documentation's examples include three unedited screen recordings captured directly on devices — 35B-A3B on an iPhone 17 Pro Max and an iPad (M4), and the 1B on an iPhone 17 Pro Max.

The scoreboard needs a rule up front: what follows is the official claim, not independently verified. Officially, 35B-A3B leads its tier across 34 authoritative benchmarks spanning multimodal understanding, STEM reasoning, OCR, and long-video understanding; the 1B sits in the first tier for its size. The official cross-model comparison: in Instruct mode, 35B-A3B scores higher overall than Qwen3.6 35B-A3B while spending less inference time, and the 1B substantially outperforms Qwen3.5 0.8B on both overall score and inference efficiency, while improving on MiniCPM-V 4.6's core capabilities. The team calls this advancing the performance-efficiency Pareto frontier — less per-token computation, undiminished capability.

The vision side has its own economy. Version 4.7 carries forward the series' mixed 4x/16x visual token compression and adds intra-ViT early compression, a technique from LLaVA-UHD v4 that compresses visual tokens inside the vision encoder itself, cutting computation in every downstream layer. Developers can switch between 4x (finer detail) and 16x (higher efficiency) per task via the downsample_mode parameter — full detail for document OCR, saved tokens for casual chat.

The Ecosystem: From Weights to Fine-Tuning

Half of any open-source project's value is ecosystem thickness, and 4.7 ships with a lot of it:

The full weight family. Hugging Face and ModelScope both carry the release, with FP8, int4, AWQ (W4A16), GPTQ (W4A16), and GGUF variants for each size, plus DSpark speculative-decoding draft heads — GPU users get FP8/AWQ, CPU-only users take GGUF, on-device users get the official layered quantization.

Inference and fine-tuning frameworks. The official documentation lists SGLang, vLLM, llama.cpp, and Ollama for inference, and SWIFT and LLaMA-Factory for fine-tuning. OpenVINO provides day-0 support on Intel platforms with notebook examples. A companion Cookbook covers six task guides: single-image, multi-image, video, OCR, PDF parsing, and visual grounding.

Self-Replay Fine-Tuning (SRFT). An officially open-sourced customization method designed to extend model capabilities at low cost while reducing forgetting of existing ones — a saner bet for vertical customization than brute-force full fine-tuning.

Platforms and chips. Official support covers iOS, Android, and HarmonyOS, with apps and source code in the companion repository OpenBMB/MiniCPM-V-Apps. The official day-0 chip list, per the documentation, covers Intel, AMD, Arm, Rockchip, and Houmo. Team announcements add that adaptation through FlagOS extends coverage to domestic chips including Huawei Ascend, Moore Threads, and MetaX (official claim).

Series-level credibility. Officially, the MiniCPM series has surpassed 60 million cumulative downloads across text, vision, and speech models — unverifiable independently, but the 26,753-star repository and two-plus years of steady generational updates suggest this is not a one-off release.

The Weight License, Straight from the Source

Licensing questions should not be guessed; they should be read. The main repository's LICENSE file is Apache-2.0 (verified against the original text). The README's "License & usage" section presents a three-part structure: Apache-2.0 for code, a Model license entry pointing to the Model License section of docs/project_en.md, and a Usage statement.

The final authority on weights is the Model License section of docs/project_en.md, whose key sentence reads: "The MiniCPM-o/V model weights and code are open-sourced under the Apache-2.0 license" — weights and code alike, under Apache-2.0. The original text is at github.com/OpenBMB/MiniCPM-V/blob/main/docs/project_en.md.

In practical terms: Apache-2.0 is a permissive license that allows commercial use, modification, and redistribution. The obligations are the standard ones — retain the license and NOTICE files, state changes — and the license includes an explicit patent grant. Commercial use requires no registration with the team. The Model License section also includes a registration questionnaire, worded in the original as "we would deeply appreciate it if you could consider optionally filling out" — the operative word is optionally: it helps the team understand their user base and affects no usage rights. The Usage statement is a standard disclaimer: model-generated content does not represent the developers' views, and users bear the risks of use.

The combination — code and weights under the same license, registration optional — is refreshingly clean by open multimodal model standards: no custom clauses, no tiered use gates, one license document to read before you know where the boundaries are.

FAQ

Q1: Why is running an MoE model on a device so hard?

A1: Because an MoE model's storage requirement is not its active-parameter count. Every token can route to different experts at every layer, and the full expert weights far exceed phone memory. Loading on demand from flash works, but repeated loading stalls make decoding too slow for conversation; restricting the expert subset speeds things up but pre-decides routing outcomes and costs capability. "Lossless and smooth" means neither can be sacrificed — a three-way constraint problem of memory, storage bandwidth, and capability.

Q2: What is LES (Lookahead Expert Scheduling), and how does it work?

A2: LES uses cross-layer routing prediction to preload experts the next layer will likely need; when the real router chooses, computation prioritizes the already-prefetched experts, and compute overlaps with loading the rest. Prediction guides only prefetching and scheduling, never the compute routing — missed experts are loaded and evaluated as usual, so no capability is lost. Officially, this yields a 5x speedup on real devices.

Q3: How should the "3GB of memory runs a 35B model" figure be understood?

A3: It is runtime memory, not full-weight size — and not a device requirement. The 35B-A3B activates only 3 billion parameters per token; with layer-wise quantization, expert weights stay on flash and stream into memory on demand. The official figure is 13+ tokens/s decoding with 3GB of memory on an iPhone 17 Pro Max. The official app's hardware table separately recommends 12GB+ device RAM for 35B-A3B (validated on a 12GB iPhone 17 Pro Max) and 4GB+ for the 1B — devices with less RAM get a smaller expert cache and slower speeds. Distinguish the two figures when citing.

Q4: How does MiniCPM-V 4.7 relate to the previous 4.6?

A4: MiniCPM-V 4.6, released on 2026-05-11, is a dense architecture and joined the official Ollama model library on 2026-06-25. Version 4.7 is the series' first MoE model — an architectural generational change. Officially, the 1B variant continues to improve on 4.6's core capabilities, while 35B-A3B pushes capability density in its tier to a new level. Both generations share series techniques such as mixed 4x/16x visual token compression.

Q5: Can the MiniCPM-V 4.7 weights be used commercially?

A5: Yes — based on the official document's original text, not inference. The Model License section of docs/project_en.md states that the model weights and code are open-sourced under Apache-2.0, which permits commercial use, modification, and redistribution, with the obligations of retaining the license and NOTICE and stating changes; the official registration questionnaire is optional and is not a precondition for commercial use. The "permissive code, separately licensed weights" split seen in some open models does not exist here — though for any specific project, treat the latest official document as the final word.

Closing Thoughts

Fitting a 35-billion-parameter MoE into 3GB of memory comes down to three techniques: lookahead expert scheduling, layer-wise quantization, and speculative sampling. The signal beyond the engineering: the barrier to running large models on devices is shifting from "how much VRAM do you have" to "how well can you schedule." With fully open weights, inference code, and fine-tuning code under a permissive license, this on-device multimodal path can finally be taken home and reused. If you want to install it yourself — phone to desktop — the hands-on deployment routes are covered in the companion SOP piece in this series.

This article is AI-assisted and human-edited. Last updated: 2026-10-15

FAQ

Why is running an MoE model on a device so hard?
Because an MoE model's storage requirement is not its active-parameter count. Every token can route to different experts at every layer, and the full expert weights far exceed phone memory. Loading on demand from flash works, but repeated loading stalls make decoding too slow for conversation; restricting the expert subset speeds things up but pre-decides routing outcomes and costs capability. "Lossless and smooth" means neither can be sacrificed — a three-way constraint problem of memory, storage bandwidth, and capability.
What is LES (Lookahead Expert Scheduling), and how does it work?
LES uses cross-layer routing prediction to preload experts the next layer will likely need; when the real router chooses, computation prioritizes the already-prefetched experts, and compute overlaps with loading the rest. Prediction guides only prefetching and scheduling, never the compute routing — missed experts are loaded and evaluated as usual, so no capability is lost. Officially, this yields a 5x speedup on real devices.
How should the "3GB of memory runs a 35B model" figure be understood?
It is runtime memory, not full-weight size — and not a device requirement. The 35B-A3B activates only 3 billion parameters per token; with layer-wise quantization, expert weights stay on flash and stream into memory on demand. The official figure is 13+ tokens/s decoding with 3GB of memory on an iPhone 17 Pro Max. The official app's hardware table separately recommends 12GB+ device RAM for 35B-A3B (validated on a 12GB iPhone 17 Pro Max) and 4GB+ for the 1B — devices with less RAM get a smaller expert cache and slower speeds. Distinguish the two figures when citing.
How does MiniCPM-V 4.7 relate to the previous 4.6?
MiniCPM-V 4.6, released on 2026-05-11, is a dense architecture and joined the official Ollama model library on 2026-06-25. Version 4.7 is the series' first MoE model — an architectural generational change. Officially, the 1B variant continues to improve on 4.6's core capabilities, while 35B-A3B pushes capability density in its tier to a new level. Both generations share series techniques such as mixed 4x/16x visual token compression.
Can the MiniCPM-V 4.7 weights be used commercially?
Yes — based on the official document's original text, not inference. The Model License section of docs/project_en.md states that the model weights and code are open-sourced under Apache-2.0, which permits commercial use, modification, and redistribution, with the obligations of retaining the license and NOTICE and stating changes; the official registration questionnaire is optional and is not a precondition for commercial use. The "permissive code, separately licensed weights" split seen in some open models does not exist here — though for any specific project, treat the latest official document as the final word.

Related

Open Source

How a Pure-C Engine Runs a 744B MoE on an Ordinary PC

JustVugg/colibri is a hierarchical MoE inference engine written in pure C (Apache-2.0, 47,567 stars per a November 4, 2026 GitHub API check): hot experts live on the GPU via CUDA or Vulkan, warm ones in RAM, cold ones stream from NVMe. Thirteen engines read published GGUF files directly, with a floor of 8 GB RAM and 22 GB disk, installed in three steps via START-HERE.bat on Windows or start-here.sh on Linux and macOS. The README demo shows GLM-5.2 at 744B MoE int4 ready in 32 seconds with 9.9 GB resident. Its engine list does not include Hy4.

Nov 4, 20269 min read
Open Source

M3.1 Flash Weights Are Out, GitHub Comes Up Empty

On October 23, 2026, MiniMax open-sourced M3.1 Flash weights: a native multimodal MoE with 428B total and 23B active parameters, a 1M-token context, MSA model-wide sparse attention (arXiv:2606.13392), and DSpark speculative decoding replacing EAGLE-style MTP. The weights live in two places: Hugging Face at MiniMaxAI/MiniMax-M3.1-Flash (public, not gated) and ModelScope at MiniMax/MiniMax-M3.1-Flash (page and API both verified 200 on release day); GitHub has no repo named after the model (github.com/MiniMax-AI/MiniMax-M3.1-Flash returns 404; the org's 37 repos include only the kernel library MiniMax-AI/MSA). The license, per the LICENSE file itself: MiniMax Community License - free for non-commercial use; commercial use requires visible attribution Built with MiniMax M3.1-Flash; annual revenue above USD 20 million requires prior written approval (api@minimax.io), with commercial use defined broadly to cover API resale and fine-tuned deployment. Three throughput calibers coexist: over 100 trillion tokens in the first 72 hours (official announcement via media), 200T over the 13 anonymous days (official, Oct 19), and 56.7T on OpenRouter during anonymity (via Zhidx, single platform); officially, agentic usage cost fell 70 percent versus the predecessor. Ecosystem: ten chip vendors including Huawei Ascend, T-Head, and Kunlunxin plus vLLM, SGLang, and FlagOS; Kunlunxin completed Day 0 while most others are still pushing first-day integration. The only official M3-to-M3.1 Flash note: keep the core MoE, change three things - sparse attention in the first three layers, QAT, and DSpark. Division of labor: the Oct 21 unmasking piece covers the anonymous saga and pricing; this one covers weights landing, repos, and the license.

Oct 24, 20269 min read
Open Source

Mistral Large 4: Model Is Live, Weights and License Are Not

Mistral announced Large 4 (nicknamed Le Chonk) in public preview on October 6, 2026, with the API open the same day: 1.05T total parameters activating 52B per token (the HF repo name Mistral-Large-4.0-1T05-A52B uses this figure; 49B excludes the embedding and output layers), a 1.6B vision encoder, training across 160+ languages, and 3,800 Grace Blackwell GPUs in Mistral's own European datacenters; the API context is listed at 1M while OpenRouter and AA test at 512K. Pricing is $1.36/$0.14/$4.18 per million tokens list, with an undated half-price column of $0.68/$0.07/$2.09 on the same docs page. But as of October 23 the open weights have not landed: the HF repo is a countdown page with zero downloadable files, the license field reads coming soon, the docs render only the word Open, and Artificial Analysis records isOpenWeights as false, classifying it as proprietary. GTC Berlin on October 21 reconfirmed the end-of-October weight release via the Mistral-NVIDIA industrial alliance. Self-reported benchmarks: DeepSWE v1.1 61.7 percent, Cybench 93 percent, and an 82 percent vulnerability reproduce-and-patch score where third-party data shows Opus 5.5 refuses 98.5 percent and GPT-6 Astra 100 percent of the same tasks; AutomationBench 59.9 despite the model card's own chart giving GLM-5.3 a higher 62.2. The third-party picture differs: AA Intelligence Index 38 (Opus 5.5 at 58), Omniscience at -5.3 meaning bare recall fabricates more than it admits, and 33rd of 45 on the vals index, roughly 9-12 months behind the frontier. Community reaction splits between 'about Kimi K3 level' performance doubts, an OpenRouter price roughly 10x DeepSeek's, and genuine EU data-residency defense. Running it locally takes an estimated 400GB+ of fast memory - today only a 512GB Mac Studio M5 Ultra theoretically qualifies.

Oct 23, 20269 min read