Field SOP
Field SOP

One Missing BOS, Every Output Breaks: A 280B Base Setup SOP

Take ERNIE 5.5 Flash Base from pre-download compliance checks to a first successful run: start with three questions from LICENSE v1.0 (territory excluding the EU, UK and South Korea, the 50 million dollar related annual revenue threshold, the Powered by ERNIE notice and one-time notice to Baidu_ERNIE@baidu.com, separate prior written authorization above the threshold, plus the distillation ban and withheld training data); choose between the FP8 and BF16 checkpoints; run the official Transformers quick start with add_special_tokens=True so the BOS token is present; serve the base model through the completions endpoint with the official SGLang or vLLM examples (--tp 4, 131072, up to 1048576) and verify with curl using prompt rather than messages; then find the continued-training entry point in NVIDIA-NeMo/Megatron-Bridge under examples/models/ernie55/. Three traps: a missing BOS, treating a Base checkpoint as a chat model, and misreading territory or revenue conditions.

Published November 6, 20269 min read
<!-- ernie-55-flash-base-setup-sop | sop | One Missing BOS, Every Output Breaks: A 280B Base Setup SOP -->

Pull a 280B weight set onto a local machine, run a few commands, and watch the first token appear. What blocks a first deployment is usually not compute but three earlier questions: where the license lets you use these weights, in what capacity, and whether the request is shaped right. After Baidu released ERNIE 5.5 Flash Base on November 5, 2026, community commands mostly covered serving, while the license boundary and input-format traps rarely got explained together. This article follows the official model card and LICENSE text in five steps, from a pre-download self-check to the first verification request. Commands are quoted as given, without inventing hardware numbers.

Division of labour. The hotspot piece covers the release event, architecture claims and license boundary; this one sticks to procedure. The comparison piece covers how to read that self-reported benchmark table and how to choose among four base models; this one compares no scores. For general memory sizing, see llm-local-deploy-vram-sop; this article is about one weight set and its official example configuration. For general LoRA and QLoRA fine-tuning, see llm-fine-tuning-sop; here I only point at the official continued-training entry point. Put simply, this answers whether the steps get you a running service, not whether you should use the model or how to squeeze VRAM.

One reminder: a responding service and commercial clearance are different things. Step 1 restates LICENSE clauses, not legal advice; whether your use falls inside the license is yours to judge.

Step 1: Read the LICENSE before you download, three checks

Do not click download yet. These weights do not use a common open-source license; they ship under a custom ERNIE 5.5 Flash Base Model License Agreement v1.0, and the Hugging Face repo is tagged license: other. The clauses below are restated from the LICENSE text retrieved on November 6, 2026, with that verification point noted, as clause restatement only, not legal advice. Walk through three checks before pulling the weight set.

Check one, is the place of use inside the Territory. The agreement defines Territory as all geographical areas worldwide other than the European Union, the United Kingdom and South Korea. Section 5.3 states that the licensed materials, derivative models and their outputs must not be used, reproduced, modified, distributed or displayed outside the Territory. This only restates the clause's scope.

Check two, is Annual Revenue of Related Products or Services above US$50 million. Section 1.7 defines it as the same Operator's aggregate revenue over the 12 months preceding any assessment time from all related products or services using the licensed materials or derivative models; it is assessed continuously. Section 4.2 permits commercial use without a model license charge at or below US$50 million, provided you fulfill the Section 3.2 attribution obligations and send a one-time use notice; revenue exactly equal to 50 million falls here. Section 4.3 requires separate prior written commercial authorization before starting commercial use above US$50 million, and states that a use notice, application, pending reply or ongoing negotiation does not confer above-threshold rights. The one-time notice to Baidu_ERNIE@baidu.com under 4.2 is not that written authorization.

Check three, can your product prominently display Powered by ERNIE and send that notice. Section 3.2 requires any commercial product or service using the licensed materials or derivative models to clearly and prominently display the exact wording Powered by ERNIE in at least one readily accessible location on the user interface, official website, about page or product documentation; for APIs, backend services or model deliveries without an end-user interface, prominent display in developer, API or delivery documentation suffices. Section 6.3 grants only limited permission for that wording. The one-time notice goes to Baidu_ERNIE@baidu.com before the first commercial use, identifying the Operator, model name and version, and related products or services. It states plainly that this notice does not constitute the 4.3 written authorization.

Other clauses are easy to skip. Section 5.5 prohibits using the licensed materials, any derivative model, or any of their outputs or results to improve any other AI model or machine-learning system other than the Model or its derivative models, the distillation ban. Section 1.4 states that unreleased training code, data, inference code or other materials do not become deliverables, so you receive weights, parameters, configurations, tokenizers and accompanying materials, not a training recipe. Section 8.3 requires that after termination you cease affected use, copying and distribution and delete or destroy copies of the weights and derivative models. All of the above is clause restatement, not a legal conclusion. For a comparison across five agreements, see openweight-license-comparison-review.

Step 2: Pick the weights and the mirror, do not grab the wrong build

The official model card download table lists two builds: the FP8 version baidu/ERNIE-5.5-Flash-Base and the BF16 version baidu/ERNIE-5.5-Flash-Base-BF16, both safetensors, published simultaneously on Hugging Face and ModelScope. Official figures are 280B total parameters and 14B activated per token. Choose by precision and hardware: FP8 saves bandwidth and VRAM footprint, while BF16 expresses numbers more directly. The official side gives no recommendation that one build is better, so do not write this trade-off as an official conclusion.

One boundary has to hold: the official model card publishes neither a minimum VRAM nor a minimum GPU count, so do not fill either in from experience. Memory sizing is a general method, not an official figure here. Verifying file hashes after download is likewise general engineering common sense, not a step in the model card's download procedure. On the mirror, Hugging Face and ModelScope use the same repo names; pick whichever is more stable. The real trap is the suffix: the BF16 repo name ends in -BF16, so do not mix the two builds, and do not configure torch.bfloat16 against the FP8 build. Keep the repo name and local cache path straight, because both serving commands below reuse the same model identifier.

Step 3: The smallest runnable check, load the weights with Transformers

Before reaching for a serving framework, confirm the weights load and generate with a few lines of Transformers. The official Quick Start reads:

python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "baidu/ERNIE-5.5-Flash-Base-BF16"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto")
model_inputs = tokenizer(["ERNIE 5.5 Flash hybrid attention"], add_special_tokens=True, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**model_inputs, max_new_tokens=256)[0], skip_special_tokens=True))

Loading uses AutoModelForCausalLM.from_pretrained, precision is set with torch_dtype=torch.bfloat16, device_map="auto" spreads layers across available devices, and max_new_tokens=256 caps generation length. The example model name is baidu/ERNIE-5.5-Flash-Base-BF16, matching bfloat16; switching to the FP8 build means adjusting dtype rather than copying this line verbatim.

The key line is add_special_tokens=True. The official text states this is a Base model and every input must begin with a BOS token; keeping the flag True adds the start marker automatically. Set add_special_tokens=False and the input loses its BOS, generation drifts, and the continuation stops being coherent. It looks like a broken model but is really an unmet input convention. Output is a prefix continuation, not an answer.

Step 4: Serve with SGLang or vLLM, then verify over completions

Both paths come from the Deployment section of the official model card. Pick one.

Start with SGLang. The official prebuilt image is lmsysorg/sglang:dev-ernie-55-flash; the docker command mounts the local Hugging Face cache into the container, with ernie55 as the example container name:

bash
docker run -d -it --name ernie55 \
  --gpus all \
  --network host \
  --ipc host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  lmsysorg/sglang:dev-ernie-55-flash

Launch the server:

bash
sglang serve \
  --model-path baidu/ERNIE-5.5-Flash-Base \
  --tp 4 \
  --attn-dp-size 4 \
  --context-length 131072 \
  --max-running-requests 1024 \
  --host 0.0.0.0 \
  --port 30000

The values --tp 4, --attn-dp-size 4, --context-length 131072, --max-running-requests 1024 and --port 30000 come from the official example. Note that --tp 4 is an official example configuration, not a minimum hardware requirement; the card does not say four GPUs are required.

Now vLLM. Create a Python 3.12 virtual environment, install the pinned version, then install the official plugin repository vLLM-ERNIE:

bash
uv venv --python 3.12 ernie5-venv
source ernie5-venv/bin/activate

uv pip install vllm==0.32.0
uv pip install git+https://github.com/baidu/vLLM-ERNIE.git

Launch the server:

bash
vllm serve baidu/ERNIE-5.5-Flash-Base \
  --kv-cache-dtype fp8 \
  --enable-expert-parallel \
  --data-parallel-size 4 \
  --max-model-len 131072 \
  --port 30000 \
  --moe-backend humming

Here --kv-cache-dtype fp8, --enable-expert-parallel, --data-parallel-size 4, --max-model-len 131072 and --moe-backend humming are quoted as given, and --data-parallel-size 4 is likewise an official example configuration rather than a minimum hardware requirement. Both commands set context length to 131072, while the model card states both servers accept up to 1048576 tokens for long-context inference, so do not mistake the command-line value for the model ceiling.

Once the service is up, verify it. Because this is a Base model, the server exposes the completions endpoint, not chat/completions; the request body uses prompt rather than messages, and no chat template applies. The official curl reads:

bash
curl http://localhost:30000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "baidu/ERNIE-5.5-Flash-Base",
    "prompt": "Large language models are",
    "max_tokens": 256,
    "temperature": 0
  }'

The request hits /v1/completions, passes the prefix through prompt, and sets max_tokens to 256 and temperature to 0. A coherent continuation means weights, framework and request format line up. For general API integration, see hy4-tokenhub-api-integration-sop; here the point is the completions plus prompt convention of a Base model.

Step 5: The right entry point for continued training

For continued training, the official blog points at examples/models/ernie55/ in the NVIDIA-NeMo/Megatron-Bridge repository. A GitHub API snapshot of that directory on November 6, 2026 shows four files: README.md, benchmark_d_layer_indexer.py, convert_sft_jsonl.py and slurm_train.sh. The official side provides a continued-training example entry point and a script list, not a turnkey full training recipe. How to assemble the cluster, how many nodes, and which accelerators are not verified in that directory listing, so do not fill in specific cluster-scale numbers from press retellings.

General LoRA and QLoRA are generic parameter-efficient fine-tuning methods, not an official recipe for this weight set; they are covered in llm-fine-tuning-sop and treated as general methods here. Before starting, go back to Step 1 and confirm the license covers your intended use, especially the Section 5.5 ban on using outputs to improve other AI systems.

Three traps that catch almost everyone

First, the missing BOS. Setting add_special_tokens to False leaves the Base model input without its start token and the output drifts. Keeping that flag True in the Step 3 tokenizer call is the most easily overlooked convention here.

Second, treating the Base build as a chat model. Its official positioning is a pretraining-end checkpoint, without mid-training or post-training, and it is not a chat model. Use completions plus prompt, do not copy messages out of a chat template, and do not invent one. The official text says a post-trained ERNIE 5.5 Flash will follow but gives no date; do not invent a timeline or force conversation behaviour out of the Base build.

Third, misreading territory and revenue. The Territory excludes the European Union, the United Kingdom and South Korea. Revenue above US$50 million requires separate prior written authorization, and a one-time notice is not that authorization. The Section 5.5 ban, Section 1.4 training-data limits and Section 8.3 weight deletion are often overlooked together. Equating it runs with I can ship it commercially is the most common and most expensive misjudgement.

Platform and scope boundaries

This stack is Linux, Docker and CUDA, and Windows cannot run these SGLang and vLLM commands natively. Using WSL2 to host a Linux environment and then install Docker is a general workaround, not a step written in the official README, so do not look to the model card when troubleshooting that environment.

To say it once more: a running service only proves the weights, framework and request format line up; commercial eligibility is decided by the LICENSE clauses in Step 1. This article restates clauses, reproduces steps, and is not legal advice. Check version pins such as vllm==0.32.0 against the repository on download day.

FAQ

Q1: The service runs, so can I use it commercially? A1: No. Running it is engineering; commercial eligibility is licensing. The three checks in Step 1 still apply: is the place of use inside the Territory, is annual revenue above US$50 million, and can you prominently display Powered by ERNIE while sending a one-time notice to Baidu_ERNIE@baidu.com. Above the threshold you need separate prior written authorization; an application or notice does not constitute it.

Q2: FP8 or BF16, and did the official card publish a minimum VRAM? A2: Decide by precision and hardware. FP8 saves bandwidth and VRAM footprint, BF16 is more direct numerically, and the official side gives no recommendation. The model card publishes neither a minimum VRAM nor a minimum GPU count, so do not fill either in. Hash verification after download is general engineering common sense, not an official step.

Q3: I sent messages like a chat template, why did it error or produce odd output? A3: Because it is a Base model. The server exposes /v1/completions, the body uses prompt rather than messages, and no chat template applies. Keep add_special_tokens=True so every input carries a BOS token; turn it off and the output drifts. These are two sides of the same mistake.

Q4: Does --tp 4 mean I need at least four GPUs, and what is the context ceiling? A4: Both --tp 4 and --data-parallel-size 4 are official example configurations, not minimum hardware requirements, and no minimum GPU count is published. Both launch commands set context length to 131072, while the model card says both servers accept up to 1048576 tokens for long-context inference. The command-line value is not the model ceiling.

Q5: Can I run this on Windows, and when does the post-trained version arrive? A5: This Linux, Docker and CUDA stack is not supported natively on Windows; WSL2 is a general workaround, not an official README step. For the post-trained version, the official text only says it will follow, with no date, so do not invent one. If you need dialogue before then you must do continued training or adaptation yourself, which leads back to the license and the Step 5 entry point.

Related reading

Pick by whichever step you are stuck on: the hotspot piece for the release event, architecture claims and license boundary; the comparison piece for reading the self-reported table and choosing among four base models; the October 23, 2026 piece for placing this license among five agreements; the general VRAM SOP for memory sizing; the LoRA and QLoRA SOP for parameter-efficient tuning; and the API integration SOP for wrapping the service as a public API.

This article is AI-assisted and human-edited. Last updated: 2026-11-06

FAQ

The service runs, so can I use it commercially?
No. Running it is engineering; commercial eligibility is licensing. The three checks in Step 1 still apply: is the place of use inside the Territory, is annual revenue above US$50 million, and can you prominently display Powered by ERNIE while sending a one-time notice to Baidu_ERNIE@baidu.com. Above the threshold you need separate prior written authorization; an application or notice does not constitute it.
FP8 or BF16, and did the official card publish a minimum VRAM?
Decide by precision and hardware. FP8 saves bandwidth and VRAM footprint, BF16 is more direct numerically, and the official side gives no recommendation. The model card publishes neither a minimum VRAM nor a minimum GPU count, so do not fill either in. Hash verification after download is general engineering common sense, not an official step.
I sent messages like a chat template, why did it error or produce odd output?
Because it is a Base model. The server exposes /v1/completions, the body uses prompt rather than messages, and no chat template applies. Keep add_special_tokens=True so every input carries a BOS token; turn it off and the output drifts. These are two sides of the same mistake.
Does --tp 4 mean I need at least four GPUs, and what is the context ceiling?
Both --tp 4 and --data-parallel-size 4 are official example configurations, not minimum hardware requirements, and no minimum GPU count is published. Both launch commands set context length to 131072, while the model card says both servers accept up to 1048576 tokens for long-context inference. The command-line value is not the model ceiling.
Can I run this on Windows, and when does the post-trained version arrive?
This Linux, Docker and CUDA stack is not supported natively on Windows; WSL2 is a general workaround, not an official README step. For the post-trained version, the official text only says it will follow, with no date, so do not invent one. If you need dialogue before then you must do continued training or adaptation yourself, which leads back to the license and the Step 5 entry point.

Related

Field SOP

Xing4.0-29B-Intent Local Deploy: 4B Active, Which Command?

Getting Xing4.0-29B-Intent running hinges on a detail that trips most people up: the official deployment instructions do not live in the intent repository but in the base model repository. The intent README only says to replace the model path with the Xing4.0-29B-Intent weights and set --served-model-name to Xing4.0-29B-Intent, so the full vLLM and SGLang launch commands must be taken from the Xing4.0-29B-A4B base repo. Several flags are hard requirements: --trust-remote-code is mandatory because model_type is the in-house xing4_0 architecture, neither llama nor qwen; --tensor-parallel-size 2 implies at least two cards; --max-model-len 262144 matches the 256K context; --reasoning-parser xing4_0 and --tool-call-parser xing4_0 supply the matching parsers; --speculative-config uses the mtp method; and --gpu-memory-utilization 0.90 with --max-num-seqs 4 is the official example ratio. The weights ship as 41 safetensors shards totaling 62.43 GB, and the only official quantizations are FP8 (33.17 GB, e4m3) and GGUF (a single IQ4_NL file at 17.75 GB) - there is no AWQ, GPTQ, or INT8 safetensors. Minimum VRAM is not published at all; the official line is to configure GPU memory, parallelism, and context length for your environment. What the base repo does spell out: bfloat16 is required because the mHC path errors out on FP16 or quantized weights, --pipeline-parallel-size greater than 1 is not yet supported, the CUDA mHC path needs tilelang installed, and the host needs Docker 24.0 or newer with driver 535 or newer. The Ascend route explicitly wants 910B2 NPUs with at least two cards, with three official quay.io images (none of which bundle the weights). Consumer and CPU paths appear only in the base llama.cpp docs: int4 weights quantized with IQ4_NL come to about 19 GB and fit on a single consumer GPU, with an RTX 3090 24 GB as the example. The SDK is just an httpx HTTP client and is not on PyPI, and structured output comes in two layers - the model's native category plus args array, and SDK-guided JSON Schema constraints through vLLM response_format. The license is Apache-2.0, yet the weight side has no standalone LICENSE file, the repository LICENSE is an unfilled placeholder template, and the original terms of the 61 upstream datasets still need checking.

Oct 22, 20269 min read
Field SOP

Qwen3.8-Flash-Next Full-Stack Deployment SOP: 125B Main Model plus 51B N-gram Embeddings, Three Tiers from Hosted API to Apple Silicon

A three-tier route for taking Qwen3.8-Flash-Next from "it runs" to "it runs cheaply". The managed tier needs no ops: the QwenCloud API speaks both OpenAI and Anthropic specs, and QwenWork's Standard mode is powered by this model. For self-hosted serving, four commands quoted verbatim from the official README: transformers serve (--continuous-batching), SGLang (--tp-size 4 --context-length 262144 --reasoning-parser qwen3 --tool-call-parser qwen3_coder), vLLM (--tensor-parallel-size 4 --max-model-len 262144 --enable-auto-tool-choice) and TokenSpeed, all exposing an OpenAI-compatible API at localhost:8000/v1. Local and edge paths include GGUF builds via llama.cpp, mlx-vlm on Apple Silicon, and Unsloth. The engineering detail most worth remembering: the extra 51B of N-gram embeddings can be offloaded to host memory and overlapped with model compute through async prefetch. Because the README gives no official VRAM baseline, this SOP refuses to guess a hardware floor and marks it as "defer to the official recipe and your own measurements". Also covers the trade-offs of YaRN extrapolation to 1M, fine-tuning framework choices (Unsloth, Swift, Llama-Factory) and seven pitfalls - the first being that the GitHub repo ships no LICENSE file, so check the model page before commercial use.

Aug 30, 202612 min read
Field SOP

Tencent 770B Flagship Self-Hosting SOP: Eight H100s Can't Even Fit the FP8 Weights - Official Baseline Is 16x B200

A complete SOP for self-hosting Tencent's 770B flagship Hy4 preview. Cold water first, via the VRAM math: FP8 weights run about 770GB, and the official vLLM recipe states the baseline is 16xB200 or 8xB300 (weights + KV cache) - 8xH100 (640GB) cannot even fit FP8 weights, since 49B active params save compute but all 770B weights must sit in VRAM. Both deployment routes quote the official README verbatim: the vLLM prebuilt image (MTP speculative decoding num_speculative_tokens=3, FLASHMLA_SPARSE attention backend, hy_v4 tool/reasoning parsers) and the SGLang prebuilt image (NEXTN speculation, tp-size 8). Includes OpenAI-compatible invocation (temperature 0.9 / top_p 1.0; no_think to skip deep reasoning and save output tokens), AngelSlim self-quantization, the finetune pipeline, 7 pitfalls and a 10-item launch checklist; if you skip self-hosting, use Tencent Cloud TokenHub/OpenRouter or the two-week free tier on WorkBuddy/CodeBuddy.

Aug 29, 202612 min read