Compare models, inference software and memory requirements for your workload, with published results and practical guides.
Running a language model on your own computer or server can support chat, coding, translation or answers grounded in documents. Data confidentiality, control over service dependencies and predictable costs are common reasons to consider local deployment. The range of models and execution methods makes the choice less straightforward: which model is sufficient for the task, what fits the available hardware, and when does upgrading the infrastructure make a useful difference?
This guide brings those decisions together. Its interactive tables compare small and large language models, related specialist models, hardware requirements and runtime software, with links to the supporting sources. The accompanying articles explain the practical advantages and limits of each approach, connecting the task and required quality to the resources and budget needed to deliver it.
Using this collection
Where should I start?
Open a table to compare options; read the articles in each path to understand the concepts and choices.
Interactive guide
Start with the key questions; add deployment details later to refine the plan.
Specifications come from model cards, configurations and versioned software documentation. Quality and speed results retain their reporter and source conditions; this site has not run independent model benchmarks.
Candidates cover distinct roles: text generation, coding, images and documents, embedding and reranking. Model size and a multilingual label do not replace evidence for the selected task and language.
The hardware table and most current speed reports focus on NVIDIA. CPU, Apple Silicon and AMD execution depends on the software and backend; the guide lacks matched speed results for ranking all these platforms. Software routes and supported hardware
Memory estimates combine weight-file bytes, KV and a reserve for each device. The conventional KV formula is not applied to hybrid, MLA or unknown architectures. Assumptions are available in the memory section.
If a request exceeds 272,000 input tokens, rates for its entire input and output become $4 / $0.2 / $5 / $15. Tool fees are separate; Persian quality and performance claims need independent evaluation.
Prototype: NVIDIA lists a free endpoint. The base model card specifies 64K; the API page rounds it to 66K. “Download not available” describes this platform only; public BF16 and GGUF weights are published separately on Hugging Face. Persian quality and regional access are unverified.
685.397 billion stored elements · Mixture of experts (MoE) Count stored in the checkpoint, including additional components; not active parameters per generated token.
8 billion Active during prefill · 16 billion Active during decode · approximately 552 billion Main backbone · approximately 196 billion Engram conditional memory · Mixture of experts (MoE) · hybrid attention Total parameter count unverified
approximately 26 billion nominal · approximately 25.2 billion Language component in the publisher's table · Mixture of experts (MoE) Total parameter count unverified
approximately 2.3 billion Effective count excluding the embedding table · approximately 5.1 billion Language component including embeddings · approximately 0.3 billion Audio encoder · Dense Total parameter count unverified
Text, Image, Audio, Video ← Text
131,072 tokens
Local text, image and audio processing with small Gemma 4
753.864 billion stored elements · Mixture of experts (MoE) Count stored in the checkpoint, including additional components; not active parameters per generated token.
753.33 billion stored elements · Mixture of experts (MoE) Count stored in the checkpoint, including additional components; not active parameters per generated token.
128,000 tokens Publisher-declared 128K; long-context quality has not been measured.
Shared Pro/Fast vector space: offline Pro indexing and online Fast queries with matching dimensions and output type. This does not imply compatibility with Embed 4 indexes.
128,000 tokens Publisher-declared 128K; long-context quality has not been measured.
Shared Pro/Fast vector space: offline Pro indexing and online Fast queries with matching dimensions and output type. This does not imply compatibility with Embed 4 indexes.
Documented starting points, not a quality ranking.
Spanish retrieval
E5-Small is a smaller baseline; Large-Instruct scores higher in this Spanish MIRACL report. The Qwen results here are multilingual aggregates, not Spanish-specific scores.
When queries and documents are both Spanish; identify cross-language retrieval separately.
The 51.2-to-53.7 difference establishes neither final-answer quality nor coverage of every Spanish variety.
English retrieval
Treat Large and Large-Instruct as separate variants: Large scores higher on English MIRACL in this report. For a reranking step, use Qwen’s comparison with a common candidate pool.
When target-language evidence matters more than a multilingual average.
The Instruct label does not guarantee an improvement on every task.
Adding reranking
Start the size comparison with 0.6B. Consider 4B and 8B when their task-specific gains justify a second-stage resource budget; 8B does not lead on every metric.
When relevant documents reach the candidate pool but rank poorly; candidate count is part of the design.
Reranking cannot recover a document absent from its input pool.
Coding problems or code edits?
For an instruction-following assistant, distinguish repository editing from coding problems. StarCoder2-3B serves code completion/FIM; HumanEval does not measure chat or editing-agent quality.
Once the interface is known: editor completion, multi-file changes, or solving a programming problem.
MoE active parameters do not determine resident weight memory.
A small model for a bounded task
LFM is a small candidate for extraction and bounded workflows; MiniCPM and Granite also have reasoning and tool-use evidence. Actual weight size can differ from the rounded size in a model name.
When task scope, output format and memory constraints are defined.
A strong small-model score establishes neither lower latency nor laptop feasibility at maximum context.
Selecting a Spanish-language generator
Salamandra supplies direct evidence for several Spanish tasks. Keep it alongside general multilingual candidates; the available data does not identify a best Spanish chatbot.
When Spanish evidence must be distinguished from a general multilingual claim.
Language coverage is not evidence for every region or professional domain.
Intent and text classification
Use MassiveIntent and MTOP evidence for intent classification. An embedding model is one component of the classifier; the result also depends on the classifier and its training data.
For a defined set of labels; the published evidence supports an initial shortlist.
An embedding model or bare ParsBERT is not a ready-to-use chatbot.
SmolLM3-3B
A 3B model with two thinking modes. For short replies, choose non-thinking mode and an explicit output budget.
Tool calling, Text generation, Document-grounded generation, Structured extraction, Programming
Prototype a small assistant with function calling
The largest SmolLM2 variant listed supports a function-calling format as well as instruction following; the two smaller variants do not inherit that feature.
Primarily an English model. The host application executes functions and must validate arguments.
Text generation, Document-grounded generation, Structured extraction, Reasoning, Programming, Tool calling
Small assistant with selectable thinking mode
SmolLM3 3B offers direct-answer and reasoning modes, a training context of 65,536 tokens and published training details.
Longer context requires YaRN. The card names six native languages including German, while repository tags list eight different languages; neither list includes Persian. Requires Transformers 4.53 or later., Count thinking tokens toward response time and context.
Text generation, Document-grounded generation, Structured extraction, Reasoning, Programming, Tool calling
Prototype chat with the smallest Qwen3
Dense Qwen3 0.6B offers thinking and non-thinking modes; an official Q8_0 file is available.
Limit the output budget; the family name does not establish the problem-solving ability of larger variants., Count thinking tokens toward response time and context.
Text generation, Document-grounded generation, Structured extraction, Reasoning, Programming, Tool calling
Small text assistant with reasoning control
Qwen3-1.7B is among the sub-2B options listed; its message template can select direct-answer mode.
In thinking mode, reasoning tokens count toward cost and output length. This record's official GGUF is Q8_0., Count thinking tokens toward response time and context.
Text generation, Document-grounded generation, Structured extraction, Reasoning, Programming, Tool calling
Text generation and analysis with dense Qwen3
Qwen3-14B is a dense model with two answer modes; official GGUF variants offer a choice of weight precision.
Q4_K_M and Q8_0 packages are available; their speed and quality have not been tested by this guide., Count thinking tokens toward response time and context.
Tool calling, Text generation, Document-grounded generation, Structured extraction, Reasoning, Programming
Tool-oriented assistant with MoE architecture
Qwen3-30B-A3B uses selected experts and the Qwen-Agent pattern for connecting tools.
Active parameters do not replace total weight size in memory estimates; execution requires compatible message templates and parsers., For simple answers, disable thinking where supported or limit the output budget., Count thinking tokens toward response time and context.
Reasoning, Programming, Text generation, Document-grounded generation, Structured extraction, Tool calling
Multi-step analysis with a larger dense Qwen3
Unlike 30B-A3B, Qwen3-32B is dense; reasoning mode can be chosen according to question complexity.
Compare direct-answer and thinking modes under matched output budgets and conditions; this overview is not a quality ranking., This is a general model; the specialized Coder has different training and tool formats., For simple answers, disable thinking where supported or limit the output budget.
Guide's analytical recommendation, Publisher-documented use
Text generation, Document-grounded generation, Structured extraction, Reasoning, Programming, Tool calling
Chat and text tasks with thinking control
Qwen3-8B is a general dense model with 8.2B declared parameters; its message template selects direct answers or reasoning.
Official Q4_K_M and Q8_0 files are available for local use; this guide has not measured their quality or runtime memory., Retrieval, document insertion and citation checks belong to the system; this relationship is not a RAG or Persian benchmark., Count thinking tokens toward response time and context.
Programming, Tool calling, Text generation, Document-grounded generation, Structured extraction, Reasoning
Repository editing with a tool-using assistant
Qwen3-Coder-30B-A3B-Instruct is tuned for code generation, repository editing and coding-agent workflows, with a native context of 262,144 tokens.
This is a non-thinking variant. Use Coder's tool-calling format; it is distinct from general Qwen3-30B-A3B., The host application supplies file, testing and execution tools., This checkpoint produces direct answers only; it has no separate thinking mode.
Guide's analytical recommendation, Publisher-documented use
Qwen3-Reranker 0.6B is tuned to score query–document relevance with task instructions.
Requires the reranker template and yes/no scoring; do not use the chat or embedding-generation path., Versioned llama.cpp compatibility: RANK input above n_ubatch requires b11223+ containing fix #28876 and causal runtime attention. b11223 is an experimental prerelease; this fix does not certify full multimodal /rerank or /embedding support, long-context quality or production readiness.
Instruction-aware reranking of candidate documents
Qwen3-Reranker-4B is the middle option; the query, retrieval instruction and document all contribute to relevance scoring.
Pass only candidate documents to this stage; more query–document pairs increase second-stage cost., Versioned llama.cpp compatibility: RANK input above n_ubatch requires b11223+ containing fix #28876 and causal runtime attention. b11223 is an experimental prerelease; this fix does not certify full multimodal /rerank or /embedding support, long-context quality or production readiness.
Qwen3-Reranker-8B is the largest reranker in this collection; it scores query–document relevance.
To choose between this and 4B, compare retrieval quality and latency on your documents; size alone is not a ranking., Versioned llama.cpp compatibility: RANK input above n_ubatch requires b11223+ containing fix #28876 and causal runtime attention. b11223 is an experimental prerelease; this fix does not certify full multimodal /rerank or /embedding support, long-context quality or production readiness.
Small Qwen3.5-2B is presented for prototyping and task-specific fine-tuning; it accepts visual inputs alongside text.
Larger family variants' quality does not transfer to this 2B model., For simple answers, disable thinking where supported or limit the output budget., Count thinking tokens toward response time and context.
The 35B-A3B variant uses an integrated text–image foundation and hybrid architecture, offering a candidate for multi-step workflows.
Hosted Qwen3.5-Flash has different tools and default context; API specifications do not transfer to local weights., Use the multimodal settings and image limits documented for these weights., For simple answers, disable thinking where supported or limit the output budget., Count thinking tokens toward response time and context.
Guide's analytical recommendation, Publisher-documented use
Text and image processing at four billion parameters
Qwen3.5-4B belongs to the integrated text–image generation; it is a smaller-than-9B candidate for document information extraction.
Scanned documents require the multimodal path; the text context limit does not determine how many images can be processed., For simple answers, disable thinking where supported or limit the output budget., Count thinking tokens toward response time and context.
Qwen3.5 9B combines text and images in a dense architecture.
Compare against 4B with the same document data and image settings; distinguish language and vision parameter counts., For simple answers, disable thinking where supported or limit the output budget., Count thinking tokens toward response time and context.
Qwen3.8-27B is a dense text–image model with thinking control; its announcement emphasizes coding, research and multi-step work.
Promised cloud-service features, such as built-in tools and default context, are not guarantees for local weights., Image dimensions and frame sampling are part of the usage conditions., For simple answers, disable thinking where supported or limit the output budget., Count thinking tokens toward response time and context.
Guide's analytical recommendation, Publisher-documented use
Reasoning, Text generation, Document-grounded generation, Structured extraction, Programming
Distilled reasoning on a Llama 70B base
This checkpoint distills R1 onto a Llama base and is the largest distilled variant listed.
Both the Llama base license and distillation publisher's terms matter; estimate weight size and reasoning-chain length separately., For simple answers, disable thinking where supported or limit the output budget.
Reasoning, Text generation, Document-grounded generation, Structured extraction, Programming
Problem solving with a mid-sized R1 distillation
DeepSeek-R1-Distill-Qwen-14B uses Qwen2.5-14B and R1-generated data; its weights are not the full R1 model.
Reported numbers apply to this variant and evaluation settings; avoid unmatched comparisons with direct-answer modes., For simple answers, disable thinking where supported or limit the output budget.
Reasoning, Text generation, Document-grounded generation, Structured extraction, Programming
Analysis and coding with a dense 32B distillation
The Qwen-32B variant distills R1 onto a Qwen2.5 base.
Declared reasoning ability is not tool calling or automatic code execution; the application must provide tools and answer validation., For simple answers, disable thinking where supported or limit the output budget.
Reasoning, Text generation, Document-grounded generation, Structured extraction, Programming
Problem solving with the 7B R1 distillation
Based on Qwen2.5-Math-7B; distinct from the later R1-0528 distillation onto Qwen3.
Keep this repository's template and tokenizer; comparison with the 8B variant requires a shared evaluation., For simple answers, disable thinking where supported or limit the output budget.
Tool calling, Text generation, Document-grounded generation, Structured extraction, Reasoning, Programming
Connect reasoning and tools in agent workflows
DeepSeek-V3.2 combines DSA sparse attention with agent-oriented post-training; its chat template differs from earlier versions.
Use V3.2-specific templates and execution paths; Speciale results do not transfer to this model., For simple answers, disable thinking where supported or limit the output budget., Count thinking tokens toward response time and context.
DeepSeek-V4.1-Flash processes text and images with CED architecture; active parameter counts differ between prefill and decode.
Active parameters are not a single fixed count; KV compression and a specialized execution path are part of this architecture., Use the CED-specific execution path; this relationship is not an OCR evaluation result., For simple answers, disable thinking where supported or limit the output budget., Count thinking tokens toward response time and context.
Guide's analytical recommendation, Publisher-documented use
Gemma 4 26B-A4B is an expert model with thinking control and 262,144 tokens of context; its architecture differs from small E2B.
This variant has no audio input; active parameters differ from all model weights., Requires the image processor path; E2B audio support does not transfer to this model., For simple answers, disable thinking where supported or limit the output budget.
Guide's analytical recommendation, Publisher-documented use
Local text, image and audio processing with small Gemma 4
Gemma 4 E2B is presented for on-device execution and also supports audio input. E2B is an effective count, not all weights.
Use total counts and actual files for memory calculations; audio and image paths need their matching processors., For simple answers, disable thinking where supported or limit the output budget., Count thinking tokens toward response time and context.
Document-grounded generation, Structured extraction, Text generation, Programming, Tool calling
Generate answers from retrieved documents
Granite 3.3 2B Instruct is a small business-oriented model supporting RAG, summarization, text extraction and thinking mode.
The system must retrieve documents and place them in the prompt; the model is not a search engine. Persian is not among its 12 declared languages., Validate output against the application's schema; requesting a format alone does not guarantee valid JSON.
Guide's analytical recommendation, Publisher-documented use
Devstral Small 2 targets software engineering and tool-assisted code inspection and editing; this generation also accepts images.
This repository's Instruct weights are FP8; use the Mistral format and dependencies for this revision, not generic chat-model settings., Agent tools and Mistral templates must match the model revision., Requires this revision's image processor and vision components; image count, resolution and frames affect memory use.
Guide's analytical recommendation, Publisher-documented use
Mistral Small 3.1 24B Instruct adds image processing and long context to Small; Persian appears in the publisher's language list.
Tools and JSON output need a matching template and parser; do not mix Base and Instruct results., Configure the parser and schema validation in the application layer.
Guide's analytical recommendation, Publisher-documented use
Document-grounded generation, Text generation, Structured extraction, Reasoning, Programming, Tool calling
Document-grounded answers with a reasoning budget
Nemotron Nano 9B v2 combines Mamba-2 with attention; reasoning mode and its token budget are controllable.
Retrieved documents must be supplied externally; Persian is not among the six declared languages, and the backend must support the hybrid architecture., For simple answers, disable thinking where supported or limit the output budget., Count thinking tokens toward response time and context.
Tool calling, Text generation, Document-grounded generation, Structured extraction, Reasoning
Adjustable-reasoning agent with the larger gpt-oss
gpt-oss-120b is an MoE model with 117B total and 5.1B active parameters; expert weights are released in MXFP4.
Correct execution requires harmony format; the host supplies browser or Python tools. Publisher memory claims are not site measurements., For simple answers, disable thinking where supported or limit the output budget., Count thinking tokens toward response time and context.
Reasoning, Text generation, Document-grounded generation, Structured extraction, Tool calling
Local reasoning with the smaller gpt-oss
gpt-oss-20b has 21B total and 3.6B active parameters, targets local or specialized use and offers three reasoning levels.
Requires harmony format and an MXFP4-compatible backend; do not substitute the 20B name for actual weight counts in calculations., For simple answers, disable thinking where supported or limit the output budget.
Tool calling, Text generation, Document-grounded generation, Structured extraction, Reasoning, Programming
Coding agent with preserved reasoning
GLM-4.7-Flash is a 30B-A3B-class MoE model; its card describes preserving thinking between turns for multi-step agents.
vLLM and SGLang paths depend on specific development versions., For simple answers, disable thinking where supported or limit the output budget., Count thinking tokens toward response time and context.
Multimodal agent for coding from visual designs and document analysis
—
For simple answers, disable thinking where supported or limit the output budget., Count thinking tokens toward response time and context., Requires this revision's image processor and vision components; image count, resolution and frames affect memory use.
Multimodal assistant for analysis, code and documents
—
For simple answers, disable thinking where supported or limit the output budget., Count thinking tokens toward response time and context., Requires this revision's image processor and vision components; image count, resolution and frames affect memory use.
Small model for prototyping and task specialization
—
For simple answers, disable thinking where supported or limit the output budget., Count thinking tokens toward response time and context., Requires this revision's image processor and vision components; image count, resolution and frames affect memory use.
For simple answers, disable thinking where supported or limit the output budget., Count thinking tokens toward response time and context., Requires this revision's image processor and vision components; image count, resolution and frames affect memory use.
For simple answers, disable thinking where supported or limit the output budget., Count thinking tokens toward response time and context., Requires this revision's image processor and vision components; image count, resolution and frames affect memory use.
Base encoder for training classification and entity extraction
—
For llama.cpp use b11532+; older builds evaluated hidden_activation=gelu with the tanh approximation. Existing GGUFs need no reconversion; retest outputs from older builds., Bidirectional encoder; token output or trained task head
An Iberian-language reference candidate with published Spanish task results. The 2B instruction model is a language-specific comparison point; these results do not establish a universal winner.
An Iberian-language reference candidate with published Spanish task results. The 7B instruction model is a language-specific comparison point; these results do not establish a universal winner.
A compact candidate for reasoning and tool workflows, with about 2.52 billion parameters reported in repository metadata. Separate the developer’s evaluations from relayed Artificial Analysis scores. This package establishes no Spanish or Persian task quality.
A compact candidate for extraction and bounded on-device tasks. The publisher advises against programming and knowledge-intensive use. Spanish is declared, but this package has no task-level Spanish quality score. It uses a custom license.
A Granite reasoning model with thinking and non-thinking modes. The 3B label is a size name; repository metadata reports about 3.66 billion parameters. Keep the native 128K context separate from the claimed extension to 512K.
Scores query relevance for text, images and video; reranks retrieved candidates with up to 32K task tokens.
Versioned llama.cpp compatibility: RANK input above n_ubatch requires b11223+ containing fix #28876 and causal runtime attention. b11223 is an experimental prerelease; this fix does not certify full multimodal /rerank or /embedding support, long-context quality or production readiness.
Scores query relevance for text, images and video; reranks retrieved candidates with up to 32K task tokens.
Versioned llama.cpp compatibility: RANK input above n_ubatch requires b11223+ containing fix #28876 and causal runtime attention. b11223 is an experimental prerelease; this fix does not certify full multimodal /rerank or /embedding support, long-context quality or production readiness.
Xiaomi’s multimodal model for coding, tool use and long inputs; the RL checkpoint has downloadable weights with MIT license metadata.
About 1.02T total and 42B active parameters; weight files occupy 534.1 GiB. This checkpoint does not fit on one 24 or 48 GB GPU., Artificial Analysis Intelligence Index v4.3.2: 46 for the MiMo-V2.6-Pro service; this is neither a Persian evaluation nor a local RL-checkpoint test.
Local multimodal retrieval for RAG; does not generate answers
Local multimodal embedder with official Apache-2.0 weights; the previous EmbeddingGemma remains a separate entry.
Before selecting for Persian text RAG, compare retrieval on a Persian corpus with identical queries/documents at 768 and 256 dimensions; Persian quality and actual memory are unknown.
Text generation, Programming, Reasoning, Tool calling, Image understanding
Reasoning, coding and image analysis
Multimodal Muse Spark Open with 816B total and 42B active; downloads require access approval and the custom license.
The Muse Model License has a Model-as-a-Service condition: aggregate revenue above US$20 million in a consecutive 12-month period requires notifying Meta and a separate agreement before offering the model in that business. Accept the terms and review the usage policy.
Text generation, Programming, Reasoning, Tool calling, Image understanding, Structured extraction
Reasoning, coding and document/video understanding
Hybrid Mamba-2/Transformer/MoE vision-language model with text, image and video input and text output.
The card uses vLLM nightly commit 3f1bcfd82bb2e93fcd364fc7a5491718453a77e2; this does not establish stable 0.31 support. Ollama 0.40.5 announces Super support.
Text generation, Programming, Reasoning, Tool calling, Image understanding, Structured extraction
Local agents, coding and image understanding
74B multimodal model with 9B active and an official IQ2 package targeting 24 GB; full service memory is not established.
24GB is a package name and publisher target, not a guarantee for full 256K context. Budget weights, projector, KV cache, drafter and overhead separately., The vLLM example uses nightly-77154ca6116ae34cbfe314beef1016bde9b63989, not verified stable v0.31. llama.cpp 0.7.0 includes Bolt support.
Text generation, Programming, Reasoning, Tool calling, Image understanding
Reasoning, coding, agents and image input
Separate 17 October DeepSeek-V4.1-Flash snapshot with Safetensors and MIT; the previous release and its results remain intact.
No Jinja template is included; use the documented reference encoder and deepseek-recipe. Earlier runtime support is not inherited by snapshot 1017., Architecture support for deepseek41 was added in llama.cpp b11870; snapshot 1017 with a converted artifact and chosen backend still requires validation.
Text generation, Reasoning, Tool calling, Programming
Reasoning and local agents
Text reasoning MoE with 24.0B total, 2.45B active and 131,072 context; official FP8, GGUF and MLX variants exist.
12 October is the base repository creation date, not a verified release date for every variant. 2.45B active does not determine the size of all 24B weights., LFM Open License v1.0 is not unrestricted commercial use; review the US$10 million annual-revenue threshold, legal-entity definition and nonprofit exception.
Text generation, Structured extraction, Tool calling
Chat, decisions and speech transcription
Granite Switch 4.1 combines a dense base with decision/Guardian adapters and a speech encoder in one package.
Chat, typed decisions and Guardian depend on the selected adapter/encoder route; this is not merely a classifier., The official vLLM route requires the Granite Switch plugin. llama.cpp b11700 adds decision support after 0.7.0; it is not attributed to 0.7.0.
Text generation, Structured extraction, Tool calling
Chat, decisions and speech transcription
Granite Switch 4.1 combines a dense base with decision/Guardian adapters and a speech encoder in one package.
Chat, typed decisions and Guardian depend on the selected adapter/encoder route; this is not merely a classifier., The official vLLM route requires the Granite Switch plugin. llama.cpp b11700 adds decision support after 0.7.0; it is not attributed to 0.7.0.
MiniCPM-V 4.7 1B accepts text, images and video for visual understanding, OCR and document analysis. Official distributions include measured file sizes.
The 262144-token limit comes from configuration; quality over that full length is unverified. GGUF needs a compatible projector; DSpark is an auxiliary model. iPhone memory and speed claims do not generalize to all GPUs.
MiniCPM-V 4.7 35B-A3B accepts text, images and video for visual understanding, OCR and document analysis. Official distributions include measured file sizes.
The 262144-token limit comes from configuration; quality over that full length is unverified. GGUF needs a compatible projector; DSpark is an auxiliary model. iPhone memory and speed claims do not generalize to all GPUs.
Classification, routing, scoring and agent decision checks
The multimodal d3 decision model accepts text or JSON with images and video, returning choice, yes/no or probabilistic scores. It does not generate free text or extract arbitrary values.
Download the model and its readout together. The Semantic Router route is in the development channel and does not establish direct support in the general vLLM engine. Ollama package size is download size; runtime VRAM and KV usage are unmeasured.
Classification, routing, scoring and agent decision checks
The multimodal d3 decision model accepts text or JSON with images and video, returning choice, yes/no or probabilistic scores. It does not generate free text or extract arbitrary values.
Download the model and its readout together. The Semantic Router route is in the development channel and does not establish direct support in the general vLLM engine. Ollama package size is download size; runtime VRAM and KV usage are unmeasured.
Classification, routing, scoring and agent decision checks
The multimodal d3 decision model accepts text or JSON with images and video, returning choice, yes/no or probabilistic scores. It does not generate free text or extract arbitrary values.
Download the model and its readout together. The Semantic Router route is in the development channel and does not establish direct support in the general vLLM engine. Ollama package size is download size; runtime VRAM and KV usage are unmeasured.
Classification, routing, scoring and agent decision checks
The multimodal d3 decision model accepts text or JSON with images and video, returning choice, yes/no or probabilistic scores. It does not generate free text or extract arbitrary values.
Download the model and its readout together. The Semantic Router route is in the development channel and does not establish direct support in the general vLLM engine. Ollama package size is download size; runtime VRAM and KV usage are unmeasured.
A Gemma-4-E4B small model for local conversation, multi-turn tool use and device interfaces; 8B total and 4B effective. Official BF16 and GGUF weights are available.
NVIDIA specifies text and 64K. Ollama advertises images and 128K for its package; this unresolved difference does not enable the model’s multimodal filter. The main GGUF is 6,180,446,208 bytes; 4GB is neither file size nor a VRAM guarantee. DFlash is auxiliary and needs extra memory.
A multimodal reranker with a bidirectional Ministral3 language tower and Pixtral vision encoder; it scores text queries paired with text, image or mixed documents.
Use Transformers 5.20.0 for images; 5.17–5.19 with Flash Attention 2 may return incorrect outputs or CUDA errors. Multimodal vLLM 0.32.0 uses /classify with query/document messages, not /score or /rerank. Output is a logit, not an embedding or calibrated probability.
Actual weight file size + KV cache + runtime reserve. GGUF scenarios use FP16 KV cache. The default GPU reserve is 2 GiB per card; the CPU reserve is 4 GiB. These reserves are planning assumptions.
MoE calculations include all weights. Nominal GPU capacity is used as a GiB budget; deployment planning should use the device's reported free memory. The sum of several GPUs' memory is not unified memory.
SmolLM2 FP32 scenarios use FP32 for both weights and KV cache. Scenarios above the selected file's context limit are excluded, including Aya Expanse 32B with the selected GGUF file's 8,192-token limit.
“—” means no information is recorded, not that the model lacks the capability.
Use the tool below to compare concurrency and context length against your model and GPU count. The accompanying article explains the calculations and how to design a service capacity test. What do 100 concurrent users mean for an AI assistant?
From user count to a memory budget
Which GPU can hold this workload?
Choose a model and workload. If you have not selected a GPU, the tool suggests candidates to test. A memory fit is an estimate; speed and service quality still need testing.
Memory assumptions and actual free space
Illustrative model: 8 billion parameters, raw BF16 weights, 32 layers, 8 KV heads and a head dimension of 128. Current geometry: 32 layers, 8 KV heads, head dimension 128.
The default 2 GiB reserve is the table’s planning assumption, not a measurement or an overhead ceiling. Quantized weights do not imply a low-precision KV cache: check engine support and cache accuracy separately. This full-attention estimate assumes no savings from prefix caching, sliding-window attention, MLA or hybrid architectures.
First candidate for comparison: 1 × GeForce RTX 5090
Fits the memory estimate
Budget of the busiest GPU24.9 GiB
Memory class per GPU32 GiB
Memory headroom per GPU7.1 GiB
Theoretical active-request limit by memory15
Weights 14.9 GiB
KV 8 GiB
Reserve and other components 2 GiB
With balanced distribution, the busiest GPU is assumed to handle 8 requests. Each GPU loads the complete weights; queues and uneven distribution can change this estimate.
For initial screening, the table’s nominal memory class is treated as a GiB budget. Before deployment, enter the device’s actual free memory. The theoretical limit above does not establish capacity at an acceptable latency.
Candidates come from the site’s current GPU list, ordered by the fewest devices and then memory capacity. Price, speed and package compatibility with each engine still need checking. When more devices are needed, the suggested count appears beside the GPU name.
Suggestions exclude announced products, unofficial boards, unified system memory and pooled multi-chip capacity. For AMD and Intel, check package, kernel and engine support on the exact model and operating system.
Explore user load and token demand
User requests per second
0.833
Model calls per second
0.833
Input demand, tokens/second
3,583.33
Output demand, tokens/second
275
User requests in flight; Little’s law
6.67
In steady state, Little’s law includes the queue and all workflow stages. It does not automatically set the running-inference slider. Input and output are counted separately; the assumed response time is not a prediction by this tool.
These ratios are only an initial screen. Stage interference, P95, bursts and goodput need testing under a mixed workload. Enter measured capacity for the entire configuration; the tool does not multiply speed by GPU count. Changing the model, deployment or request length clears the measurements so you can enter results for the new settings.
Section four · two separate views
Inference and serving software
04.1
Data view
Software comparison
Compare software for local inference and model serving.
72results out of 72 rows
Advanced filters 36 controls
All rows
Row comparison and conditions
Comparison rule: Versions can be displayed together. Interface comparisons keep the backend fixed; full-stack comparisons may vary it.
Use × beside a heading to hide its column. Open row details for sources and conditions.
Hidden columns:
Versioned inference and serving software comparison
Compare
Details
What is it for?
Practical advantage
Selection condition
Get started
Ollama · v0.40.2
Inference engine / library, API server, Model manager
Local setup, development and small services
Model downloads, management and API setup in one tool
Local and cloud execution are separate; OLLAMA_NO_CLOUD=1 disables cloud routes. Stable / Latest Ollama v0.40.2: only approved architectures run on MLX by default on Apple Silicon: Qwen3.8, Qwen3.6, Qwen3.5, Gemma 4, Nimble, Tev1, Clef, Clef Flash and EmbeddingGemma 2. Do not generalize to all models, sizes, quantizations, inputs or operating systems; verify each route’s compatible tag and package separately. Decision models return typed outputs, not chat, tools or thinking. Persian quality, speed and actual memory for this route are untested. On the first run of each older model, an upgrade runs in the background while the original temporarily remains on disk. Check free disk space before upgrading; automatic backup removal is planned for a future release. The Windows/Clef Flash requirement remains v0.40.1+.
Inference engine / library, API server, Model manager
Download, manage and run models locally through an API
Text generation, Embedding generation, Streaming output, Structured output Stable; historical release Ollama v0.40.0: only approved architectures run on MLX by default on Apple Silicon: Qwen3.8, Qwen3.6, Qwen3.5, Gemma 4, Nimble, Tev1, Clef, Clef Flash and EmbeddingGemma 2. Do not generalize to all models, sizes, quantizations, inputs or operating systems; verify each route’s compatible tag and package separately. Decision models return typed outputs, not chat, tools or thinking. Persian quality, speed and actual memory for this route are untested.
Stable; historical release Ollama v0.40.0: only approved architectures run on MLX by default on Apple Silicon: Qwen3.8, Qwen3.6, Qwen3.5, Gemma 4, Nimble, Tev1, Clef, Clef Flash and EmbeddingGemma 2. Do not generalize to all models, sizes, quantizations, inputs or operating systems; verify each route’s compatible tag and package separately. Decision models return typed outputs, not chat, tools or thinking. Persian quality, speed and actual memory for this route are untested.
Continuous batching and KV-cache management; OpenAI-compatible API
Historical release; GGUF requires the separately installed vllm-gguf-plugin; support for this format is experimental. GHSA-x6mc-67gf-chw4: conditional vulnerability in >=0.24.0,<0.30.0. The Python frontend with Qwen2-VL/Qwen3-VL video sampling accepts media_io_kwargs.video.max_frames and fps, allowing memory DoS; requests can also select the sampler. --limit-mm-per-prompt limits items, not frames; on affected versions --api-key alone does not guard /tokenize or /invocations. The Rust frontend rejects this input and is unaffected by this specific issue only. Upgrade to 0.30.0 is the declared fix; gateway authentication and resource limits are additional controls. GHSA-85xf-c7hm-whqw: malformed structured output can terminate shared EngineCore; declared range <0.30.0, but the report confirms testing on 0.25.1 without establishing a lower bound. GHSA-5fj9-pfhr-6j48: enabled Rust frontend before 0.30.0; arbitrary HTTP methods accumulate Prometheus labels and memory, even on /tokenize 405 responses. GHSA-58v5-2m8f-94pr: >=0.23.0rc2,<0.30.0; video-capable models accepting media_io_kwargs permit GLMGA selection and excessive pre-decode CPU/memory work; a valid key is required when API-key authentication is enabled. The latter three advisories also declare 0.30.0 patched.
Language and multimodal model serving on one GPU or a cluster
Text generation, Continuous batching, Prefix caching, Structured output Tool parsers and quantization kernels depend on the model architecture. Starting with 0.5.20, CUDA 12 packages and images are no longer published; 0.5.19 is the last release for this path. Previously published images remain available. Responses storage is off by default. Retrieval, previous_response_id and background requests require --enable-response-store; this option cannot be enabled in prefill/decode-disaggregated (PD) deployments.
Tool parsers and quantization kernels depend on the model architecture. Starting with 0.5.20, CUDA 12 packages and images are no longer published; 0.5.19 is the last release for this path. Previously published images remain available. Responses storage is off by default. Retrieval, previous_response_id and background requests require --enable-response-store; this option cannot be enabled in prefill/decode-disaggregated (PD) deployments.
Shared gateway for multiple providers and model services
Connects provider APIs and model servers
A newer stable API gateway release; generation happens in the connected service. Do not apply Enterprise features or licensing to the open-source core.
Maintenance of text-generation serving for supported models
Text generation, Model sharding across GPUs (conditional), Streaming output, Continuous batching The repository has been archived and read-only since March 21, 2026.
The repository has been archived and read-only since March 21, 2026.
Local generation and quantization of compatible models on Apple silicon
Text generation (conditional), Streaming output (conditional), Prefix caching (conditional), Model sharding across GPUs (conditional) A direct route for local generation and quantization on Apple silicon with compatible models. Do not double-count unified memory as RAM plus VRAM. The README’s macOS 15 requirement concerns large-model memory wiring, not every MLX LM feature.
A direct route for local generation and quantization on Apple silicon with compatible models. Do not double-count unified memory as RAM plus VRAM. The README’s macOS 15 requirement concerns large-model memory wiring, not every MLX LM feature.
Four security advisories fixed; review upgrade conditions
Stable; fixes GHSA-x6mc-67gf-chw4, GHSA-85xf-c7hm-whqw, GHSA-5fj9-pfhr-6j48 and GHSA-58v5-2m8f-94pr per official advisories. Upgrading from 0.29: GPTQ activation ordering using g_idx was removed and g_idx is ignored; review dependent checkpoints. Plain vllm serve requires --enable-scale-out; VLLM_ENABLE_SCALE_OUT_ENDPOINTS was removed. launch render and --tokens-only are exceptions. Default packages use CUDA 13.0; separate CUDA 12.9 wheels/images exist. Check GPU/driver compatibility with the selected package. Older benchmark results are not assigned to this release.
RPC protocol 7; clients and servers must be compatible.
For RPC, the client and every server must share the major version, and the server minor must not exceed the client minor. Align both builds; do not connect v0.4.1 protocol 6 to v0.5.0 protocol 7. Prerelease fixes and capabilities are listed under their own builds.
FP32 accumulation in F16 dot products; overflow fix
Prerelease b11262 fixes CPU computation correctness. In the AVX512-FP16 path, FP16 accumulation in F16 dot products could overflow and change logits; b11262 accumulates in FP32. Retest older-build output on b11262+ containing fix #29545 before relying on this path. This is not evidence of a speed increase, and the prerelease is not a production default.
Stable release; typed decisions via /v1/systemone for Nimble, Tev1, Clef and Clef Flash
Clef/Clef Flash text and image input; choice, noul and score; decision capability
Stable v0.35.1: /v1/systemone accepts choice, noul and score questions for compatible decision models; Clef and Clef Flash score text and images together. Declare CAPABILITY decision for a compatible model; clients must not offer it for chat, tools or thinking. The endpoint does not generate free-form text. The release notes mention updates to llama.cpp and MLX but do not establish backend-by-backend compatibility. For Nimble and Tev1, Safetensors alone does not establish that the same file runs with SystemOne; verify the compatible artifact separately. No local run or Persian quality evaluation was performed.
Prerelease; IQ4_NL memory-safety fix for CUDA/HIP/MUSA. This path requires b11310+ containing #29683. When the element count is a multiple of QK4_NL but not QK_K, the final block could read beyond the input and write beyond the output buffer; QK_K multiples are unaffected by this bug. In the author’s debug test with 192 elements, out-of-bounds writes fell from 128 to zero bytes (CUDA H20-3e and MUSA S5000); HIP shares the code but was not separately tested in that report. As reviewed on 1 October 2026, the sources announce no CVE or exploitability. This is not evidence of increased speed or changed memory requirements.
Layer-input order; separate KV streams and tensor split
Prerelease; restores layer-input order for DFlash/DFlash2 across multiple sequences. For concurrency above one, especially with separate KV streams, use b11307+ containing #29019; the fix also accommodates tensor split. Batch splitting could deliver another token’s hidden state to the drafter; the fix restores original order. In the author’s SPEED-Bench run with Qwen3.8-27B and DFlash2, both Q4_K_M at concurrency=16, mean acceptance length rose from 3.3531 to 3.9211 (+16.94%). This does not establish end-to-end speed, Persian quality or performance on other models and hardware. Issue #27117 provides context, not confirmation that every reported case is fixed. No local runtime test; the prerelease is not a production default.
This build supports Clef text only; v0.6.0 adds image input
b11371 adds initial text-only Clef support. Stable v0.6.0 adds images with compatible GGUF and projector files; this build’s limit does not apply to newer versions or Ollama v0.35.1. b11371 remains a prerelease; local performance and Persian quality are unverified.
Stable llama.cpp / llama-server with Clef text and image decisions
Classification (conditional) Stable v0.6.0 adds Clef image input through /v1/systemone. Use GGUF with the updated template and a compatible projector (--mmproj); older files may need re-conversion. The whole prompt must fit --ubatch-size; images use data URLs. Responses are choice, noul or score; a Clef server does not generate free text. This release also includes Vulkan FlashAttention fix #29988. Persian quality, speed and full runtime memory remain untested.
Stable v0.6.0 adds Clef image input through /v1/systemone. Use GGUF with the updated template and a compatible projector (--mmproj); older files may need re-conversion. The whole prompt must fit --ubatch-size; images use data URLs. Responses are choice, noul or score; a Clef server does not generate free text. This release also includes Vulkan FlashAttention fix #29988. Persian quality, speed and full runtime memory remain untested.
fixes an out-of-bounds access in Vulkan FlashAttention
Vulkan + FlashAttention: shared-memory write fix
b11424 includes #29988 for an out-of-bounds shared-memory write in Vulkan FlashAttention; the official report describes an NVIDIA DeviceLost crash. For the affected path, use b11424+ containing this fix or stable v0.6.0 and retest the workload. This is not evidence of higher speed, a CVE or a proven exploit; VRAM estimates are unchanged.
minimum llama.cpp route for five K2 Horizon models with official GGUF
Text generation (conditional), Tool calling (conditional) Models launched on September 3; b11454 adds execution support on October 6 and remains a prerelease. Scope is the five official GGUF models; no 375B-A23B route is attributed. -sm tensor is disabled. ZWNJ/ZWJ and Windows fixes are not Persian-quality evidence; speed and actual memory are untested.
Models launched on September 3; b11454 adds execution support on October 6 and remains a prerelease. Scope is the five official GGUF models; no 375B-A23B route is attributed. -sm tensor is disabled. ZWNJ/ZWJ and Windows fixes are not Persian-quality evidence; speed and actual memory are untested.
minimum build containing both the memory-safety and ModernBERT GELU fixes
b11529+ checks the offset in ggml_acc_impl before narrowing it to int32_t to prevent overflow-induced out-of-bounds writes. The PR report also mentions negative offsets arriving via RPC GRAPH_COMPUTE; the final patch does not directly validate RPC parameters. b11532+ uses exact GELU matching Transformers for ModernBERT with hidden_activation=gelu.
The reviewed sources announce no CVE or real-world exploitation; this fix is not evidence of a speed increase. Existing ModernBERT GGUFs need no reconversion; retest outputs from older builds. The author tested only CPU/F32 on gte-reranker-modernbert-base: maximum activation error fell from 5.8e-2 to 7.4e-6; GPU was not tested and CPU speed changes were within run-to-run noise. These figures are not a direct ModernBERT-base evaluation.
minimum build containing the embedding-graph ordering fix after the mixed-batch regression
Fix #30160 orders GET_ROWS before other input operations to reduce unnecessary graph splits and CPU execution. Use b11537 or newer for mixed-batch/embedding and as the starting point for retesting MTP and multi-GPU paths; verify model support separately.
One user report on Linux, SYCL and two Arc Pro B70 cards with Qwen3.8-Flash-Next / UD-IQ3_XXS: about 365 pp and 24 tg recovered to 582.23 pp and 32.75 tg. The corrected numbers were measured on the PR branch, not an independent b11537 binary test. Input details, repetition count and a separate MTP test are not fully reported; do not generalize to every GPU.
SYCL MXFP4 MoE optimized decode; #29809 uses arithmetic decoding and weight reordering. The author reports gpt-oss-20b on one B70 improving from 55.84 to 106.64 tok/s and gpt-oss-120b on two B70 cards from 33.27 to 67.14 tok/s.
The PR author’s original test used Windows 11, oneAPI 2026.0, an EPYC 7402P CPU and Flash Attention enabled; five repetitions for 20B and two for 120B. These are decode results; do not extrapolate to prefill, Linux or other GPUs. b11540 contains the fix, but the numbers come from the PR branch; the site has not independently tested them.
llama-server context-isolation fix for pinned id_slot with RAM prompt caching
A request with a pinned id_slot now waits without loading cached context into a busy slot. Configuration requirement: pinned id_slot + RAM prompt cache → b11552+. Keep this combination out of multi-user service until upgraded and the canary acceptance test passes.
Context isolation failed; the release and PR report no CVE or real-world disclosure incident. Limited upstream test: M2 Ultra/Metal, Qwen3.8-27B-UD-Q4_K_M plus mmproj, --parallel 10, --kv-unified, --no-cache-idle-slots, --cache-ram 8192 and id_slot=7. b11146: 30 simultaneous pairs, 1 crossing and 9 corrupted responses. PR branch (build 11550): 60 pairs, 0 crossings and 0 corrupted responses; one round hit a 60-second client timeout. b11552 contains the fix; these are not release-binary measurements or a guarantee for every configuration. Impact without slot pinning remains unknown.
Stable serving with MoE expert caching and LiliCorr
--moe-cache-mib sets the cache budget for host experts; this release uses RPC protocol 8 and supports tensor split.
Before connecting over RPC, check protocol 8 compatibility on both sides. Direct execution defaults to port 9931; Docker still uses port 8080 internally. The chat session API and part-based format changes may require client updates. Beam support in b11701 and Granite support in b11700/11707 arrived after this release.
The 0.40.2 background upgrade was paused in 0.40.3 for performance; do not extend that behavior to 0.40.5. Cloud decision models differ from local execution. Bolt 24GB does not guarantee full context or total memory; verify the exact package and version.
Stable vLLM 0.31.0; version- and input-scoped serving
GHSA-4xqp-c3mv-qff7 fixed; declared affected range below 0.31.0
The advisory describes malformed /inference/v1/generate input hanging serving while /health stays healthy: declared affected <0.31.0, patched >=0.31.0, with a 0.29 proof of concept. Review multimodal kwargs, tokenizer_mode=slow, cache changes and other 6 October advisories separately. New-model nightly support is not attributed to this stable release.
Stable SGLang 0.5.21; versioned CUDA and ROCm routes
Version-scoped MiMo 2.6 and GLM 5.3 Flash support on MI355X
ROCm images are GPU-family-specific; MI355X support does not certify all AMD GPUs. Reported speedups are not generalized to other models or conditions; historical results are retained.
Stable SGLang 0.5.22: serving configuration migration and hybrid-model state correctness
Python ≥3.11; an unpinned install on Python 3.10 may silently select an older release. Use --attn-dp-size N instead of --enable-dp-attention; the old combination is translated with a warning. --dp-size counts replicas; DeepSeek CP temporarily retains its attention-DP meaning, to be rejected after 2026-12-31. Combining replicas with attention DP, or both spellings, fails at startup. --disaggregation-decode-host-receive-threshold reverses its encoding: 1=disabled (default), 0=always receive on host. tc_piecewise was removed; --cuda-graph-backend-prefill tc_piecewise and tc_compiler settings fail at startup.
Separate dates: GitHub release on 19 October 2026 at 02:55 UTC; PyPI package available since 16 October. Runtime correctness: fixes incorrect Mamba checkpoint tracking when speculative decoding and structured output are combined, which could resume hybrid models from the wrong state (PR #43029). Security: a Dumper fix is unconfirmed; the eval() path remains in tag 0.5.22. Do not treat it as fixing the earlier vulnerability. Confidence in the release, requirements and changes is high; performance and Persian quality were not tested here. This record is not an untested upgrade recommendation; OEM-specific BOM/QPLs and internal wizard code are not fully covered.
From b11722, Flash Attention can process multiple sequences on Hexagon with --no-kv-unified.
PR #30218 tests IQ-9075 EVK / Hexagon v73 with Llama-3.2-1B and Qwen3.5-2B Q4_0. The route requires --no-kv-unified. These results do not validate every Snapdragon or a single-session speedup; test on your device before use. No local runtime test was run.
gguf-split and export-lora now detect incomplete tensor reads and stop with an error.
A reused buffer could previously fill unread bytes with data from the preceding tensor, producing corrupt output that appeared successful. Fix #30497 rejects that case; input checksum verification is still required. The fix does not imply problems with valid files or change model quality. No local runtime test was run.
LFM2 and LFM2.5-VL image preprocessing aligned with Transformers
Images are stretched to the tile grid, and resizing and dimension rounding follow the Python reference. Reconvert mmproj with the updated converter to store processor_config.json settings; b11767 changes are included.
Before the fix, layout differed for 31,926 sizes among about 16.8 million cases up to 4096×4096. In the 720×1600 example, tokens fell from 2812 to 2298. Other image shapes can increase token count and runtime. PR output tests mainly cover 450M models and the d1 family; the LFM2.5-VL-3B route remains experimental and needs tests on your images. This note does not certify the DSpark model. No local runtime test was run.
Text embeddings with Nemotron-3 and Nemotron-3.5 Embed
Convert Embed 1B/8B BF16 weights to GGUF and run them with llama-embedding. This is a user conversion; no official ready-to-download GGUF is recorded.
A published comparison for one 1B text input reports cosine similarity 0.999847 against PyTorch. This measures implementation agreement, not retrieval quality or correctness for every input. The evidence does not show end-to-end image or 8B tests. This route does not cover Rerank-8B or NVFP4. No local runtime test was run.
coopmat1 requires a cooperative-matrix build and device/driver INT8 support; GGML_VK_DISABLE_COOPMAT must not disable it.
Formats: q4_0, q4_1, q5_0, q5_1, q8_0, q3_k, q4_k, q5_k, q6_k, mxfp4, nvfp4 and iq4_nl. On RDNA4, MUL_MAT excludes q4_1, q5_1, q4_k, q5_k and nvfp4; MUL_MAT_ID also excludes nvfp4. Publisher tests include RDNA3.5, without validating every AMD GPU. Before relying on b11160 results, repeat correctness and then performance tests on b11224+ with fix #28956.
On Blackwell CUDA, NVFP4/MXFP4 layers marked W4A16 can run as W4A8. FP4 weights do not imply four-bit activations.
GGML_CUDA_MMQ_PREC=auto follows metadata; q4 and q8 override it. q4 may accelerate prompt processing at an accuracy cost, so test it before choosing a default. This change is not part of stable v0.5.0.
Long inputs for causal Qwen3 and Qwen3-VL rerankers
From b11223+ with fix #28876, RANK input with causal attention can span multiple physical batches.
Previously, input above n_ubatch, default 512 tokens, was rejected. Bidirectional cross-encoders still need one physical batch. This does not complete multimodal /rerank or /embedding support; deployment needs controlled testing.
Fix #28956 corrects mul_mat and mul_mat_id reads for some strided cache and expert views.
The PR reports an endless SheetSage2 loop and a Qwen3 AR --no-fa issue. On b11224+, compare output with CPU/CUDA under matching settings and explicit numerical tolerances. Older b11160 results also need retesting. This fix does not establish a speedup or failures in every model.
From b11240+, /v1/embeddings accepts OpenAI-style typed input; supported modalities depend on the model.
Each {"content": [...]} object produces one embedding; bare content arrays are rejected and the legacy API remains available. KV prefix reuse is disabled for embedding/rerank, so remeasure capacity and speed. The Qwen3-VL-Embedding-2B/8B route is experimental and does not certify multimodal Reranker support.
b11279 introduced support in #27773; experiments also need the later fix #29745.
A CPU/ThreadSanitizer data race was reported after merging. b11279 does not contain fix #29745: verify its inclusion in your build and retest output. This note does not establish GLM-5.3-BF16 support. No local runtime test was run.
Intel XMX fix for short batches and speculative verification
Refactored multi-column kernels, input alignment and fused gate/up; fixed NaNs for row counts not divisible by four.
Published performance evidence is limited to Arc Pro B70 and oneAPI; it does not generalize to all Intel GPUs. Local correctness and speed were not tested.
Registers the model and projector using the MiniCPM-V 4.6 conversion path; this alone does not establish a public checkpoint or model license.
OCR text may appear in reasoning_content; use --reasoning off or reasoning_effort: "none" for content. This runtime fix does not establish public model weights or license.
Changes operand elements from __bf16 to short without altering bits; fixes builds for gfx1150/1151 and gfx1200/1201 with newer HIP/ROCm and LLVM.
For these targets with a newer toolchain, use b11847 or later. The PR author tested only gfx1151; not every RDNA3/RDNA4 GPU was tested. GPU specifications and memory estimates are unchanged.
Inference engine / library, API server, Model manager
Historical stable; d3, Phonon speech and cross-platform Flint
d3 at /v1/systemone; Phonon at /v1/audio/speech on Apple Silicon; Flint on Linux and Windows; OpenAI/Anthropic streaming error reporting.
Phonon is Apple-Silicon-only here and is not a Persian model. d3 returns decisions, not free text. Generation errors should no longer be hidden after partial stream output; no local test was run.
Inference engine / library, API server, Model manager
Stable / Latest; adds E4B and DFlash
nemotron-3.5-bolt:e4b with DFlash; retains the d3, speech and Flint additions from 0.40.9.
NVIDIA specifies text and 64K. Ollama advertises images and 128K for its package; this unresolved difference does not enable the model’s multimodal filter. The main GGUF is 6,180,446,208 bytes; 4GB is neither file size nor a VRAM guarantee. DFlash is auxiliary and needs extra memory.
In rdma_recv, a receive larger than the remaining buffer is rejected before memcpy; the receive slot is reposted and the connection is closed.
Scoped to RPC over RDMA, not the HTTP API. The PR author reproduced the bug and fix using ASan and SoftRoCE; real-world exploitation and independent local testing are not established.
Disables --split-mode tensor for mellum and flint; mistral4 was already denied. Splitting a row produced partial maxima rather than the full-row maximum needed for MoE rescaling.
This restriction is specific to these architectures and this split mode; it does not prohibit all MoE models or all multi-GPU methods. No independent multi-GPU runtime test was performed.
Adds the deepseek41 architecture in commit 74bca39; conversion writes the Engram table as BF16 and removes its MXFP8 path.
Architecture support does not validate snapshot 1017 with a specific artifact and backend. Official GGUF, direct MXFP8 support, Persian quality, speed and runtime memory are unverified; not all PR checks were reported successful.
Inference engine / library, API server, Model manager
Preserve model sampler settings and fix API behavior
When temperature or top_p is omitted, compatible endpoints retain the model package defaults. Fixes cover compressed web-search responses, /v1/responses content and MLX top-k sampling.
Clients omitting sampler parameters now use the model’s own settings; account for this when reproducing outputs.
This tag documents PyTorch 2.14, Triton 3.8, CUDA 13.0 artifacts, default ROCm 10 images and /v1/decisions. CUDA 12.9 artifacts are no longer published.
Prerelease rc3; migration from 0.32.0 changes the CUDA and PyTorch packages.
“—” means no information is recorded, not that the model lacks the capability.
Evaluation results
Evidence for model selection
How selection and comparison work
On Spanish MIRACL, nDCG@10 rises from 51.2 for Small to 53.7 for Large-Instruct. On English, Large scores 52.9 versus 51.5 for Large-Instruct; Persian scores are 59.0 and 59.4. A multilingual average cannot replace the target-language comparison.
Two displayed numbers are not automatically rankable. Check the model variant, benchmark version, language, generation mode and protocol. A reported score difference is neither a quality ratio nor a statistical-significance claim.
Distilled Model Evaluation table; generation settings immediately preceding the table
Named distilled models in one publisher table; MATH-500 pass@1 uses the reported sampling protocol. No quantized-artifact or statistical-superiority claim.
83.9 %0..100 as printed
Reasoningtest
DeepSeekPublisher report
Conditions and sourceMATH-500 · pass@1: 83.9%Publisher report · DeepSeek· Reasoning· en
Evaluation mode
Reasoning
Model named in the report
DeepSeek-R1-Distill-Qwen-1.5B
Benchmark / version
MATH-500
Metric and unit
pass@1 · Percent
Source document commit
ad9f0ae0864d7fbcd1cd905e3c6c5b069cc8b562
Language
en
Maximum output tokens
32,768
Temperature
0.6
top-p
0.95
Samples per question
64
Dataset split
test
Score scale
0..100 as printed
Access date
2026-09-15 (Gregorian)
Publisher score; not suitable for ranking across different sources or directly inferring cost/performance.
Tested weight commit and precision are unreported; the result applies only to the named variant.
The 64-sample protocol estimates pass@1 for sampled evaluations; it has not been extended to CodeForces rating.
Distilled Model Evaluation table; generation settings immediately preceding the table
Named distilled models in one publisher table; MATH-500 pass@1 uses the reported sampling protocol. No quantized-artifact or statistical-superiority claim.
93.9 %0..100 as printed
Reasoningtest
DeepSeekPublisher report
Conditions and sourceMATH-500 · pass@1: 93.9%Publisher report · DeepSeek· Reasoning· en
Evaluation mode
Reasoning
Model named in the report
DeepSeek-R1-Distill-Qwen-14B
Benchmark / version
MATH-500
Metric and unit
pass@1 · Percent
Source document commit
1df8507178afcc1bef68cd8c393f61a886323761
Language
en
Maximum output tokens
32,768
Temperature
0.6
top-p
0.95
Samples per question
64
Dataset split
test
Score scale
0..100 as printed
Access date
2026-09-15 (Gregorian)
Publisher score; not suitable for ranking across different sources or directly inferring cost/performance.
Tested weight commit and precision are unreported; the result applies only to the named variant.
The 64-sample protocol estimates pass@1 for sampled evaluations; it has not been extended to CodeForces rating.
Distilled Model Evaluation table; generation settings immediately preceding the table
Named distilled models in one publisher table; MATH-500 pass@1 uses the reported sampling protocol. No quantized-artifact or statistical-superiority claim.
94.3 %0..100 as printed
Reasoningtest
DeepSeekPublisher report
Conditions and sourceMATH-500 · pass@1: 94.3%Publisher report · DeepSeek· Reasoning· en
Evaluation mode
Reasoning
Model named in the report
DeepSeek-R1-Distill-Qwen-32B
Benchmark / version
MATH-500
Metric and unit
pass@1 · Percent
Source document commit
711ad2ea6aa40cfca18895e8aca02ab92df1a746
Language
en
Maximum output tokens
32,768
Temperature
0.6
top-p
0.95
Samples per question
64
Dataset split
test
Score scale
0..100 as printed
Access date
2026-09-15 (Gregorian)
Publisher score; not suitable for ranking across different sources or directly inferring cost/performance.
Tested weight commit and precision are unreported; the result applies only to the named variant.
The 64-sample protocol estimates pass@1 for sampled evaluations; it has not been extended to CodeForces rating.
Distilled Model Evaluation table; generation settings immediately preceding the table
Named distilled models in one publisher table; MATH-500 pass@1 uses the reported sampling protocol. No quantized-artifact or statistical-superiority claim.
92.8 %0..100 as printed
Reasoningtest
DeepSeekPublisher report
Conditions and sourceMATH-500 · pass@1: 92.8%Publisher report · DeepSeek· Reasoning· en
Evaluation mode
Reasoning
Model named in the report
DeepSeek-R1-Distill-Qwen-7B
Benchmark / version
MATH-500
Metric and unit
pass@1 · Percent
Source document commit
916b56a44061fd5cd7d6a8fb632557ed4f724f60
Language
en
Maximum output tokens
32,768
Temperature
0.6
top-p
0.95
Samples per question
64
Dataset split
test
Score scale
0..100 as printed
Access date
2026-09-15 (Gregorian)
Publisher score; not suitable for ranking across different sources or directly inferring cost/performance.
Tested weight commit and precision are unreported; the result applies only to the named variant.
The 64-sample protocol estimates pass@1 for sampled evaluations; it has not been extended to CodeForces rating.
Distilled Model Evaluation table; generation settings immediately preceding the table
Named distilled models in one publisher table; MATH-500 pass@1 uses the reported sampling protocol. No quantized-artifact or statistical-superiority claim.
06
Data view
Small and specialized complementary models
Compare specialized roles, input limits, vector dimensions and results for the selected language.
Test shown in the result column: MIRACL · nDCG@10 · en
This selection applies to result columns across tables. Execution conditions are available in each model’s details.
multilingual, af, am, ar, as, az, be, bg, bn, br, bs, ca, cs, cy, da, de, el, en, eo, es, et, eu, fa, fi, fr, fy, ga, gd, gl, gu, ha, he, hi, hr, hu, hy, id, is, it, ja, jv, ka, kk, km, kn, ko, ku, ky, la, lo, lt, lv, mg, mk, ml, mn, mr, ms, my, ne, nl, no, om, or, pa, pl, ps, pt, ro, ru, sa, sd, si, sk, sl, so, sq, sr, su, sv, sw, ta, te, th, tl, tr, ug, uk, ur, uz, vi, xh, yi, zh en: Published result
multilingual, af, am, ar, as, az, be, bg, bn, br, bs, ca, cs, cy, da, de, el, en, eo, es, et, eu, fa, fi, fr, fy, ga, gd, gl, gu, ha, he, hi, hr, hu, hy, id, is, it, ja, jv, ka, kk, km, kn, ko, ku, ky, la, lo, lt, lv, mg, mk, ml, mn, mr, ms, my, ne, nl, no, om, or, pa, pl, ps, pt, ro, ru, sa, sd, si, sk, sl, so, sq, sr, su, sv, sw, ta, te, th, tl, tr, ug, uk, ur, uz, vi, xh, yi, zh en: Published result
multilingual, af, am, ar, as, az, be, bg, bn, br, bs, ca, cs, cy, da, de, el, en, eo, es, et, eu, fa, fi, fr, fy, ga, gd, gl, gu, ha, he, hi, hr, hu, hy, id, is, it, ja, jv, ka, kk, km, kn, ko, ku, ky, la, lo, lt, lv, mg, mk, ml, mn, mr, ms, my, ne, nl, no, om, or, pa, pl, ps, pt, ro, ru, sa, sd, si, sk, sl, so, sq, sr, su, sv, sw, ta, te, th, tl, tr, ug, uk, ur, uz, vi, xh, yi, zh en: Published result
approximately 0.57 billion nominal Total parameter count unverified
8,192 tokens
1,024-dimensional vector
multilingual, af, am, ar, as, az, be, bg, bn, br, bs, ca, cs, cy, da, de, el, en, eo, es, et, eu, fa, fi, fr, fy, ga, gd, gl, gu, ha, he, hi, hr, hu, hy, id, is, it, ja, jv, ka, kk, km, kn, ko, ku, ky, la, lo, lt, lv, mg, mk, ml, mn, mr, ms, my, ne, nl, no, om, or, pa, pl, ps, pt, ro, ru, sa, sd, si, sk, sl, so, sq, sr, su, sv, sw, ta, te, th, tl, tr, ug, uk, ur, uz, vi, xh, yi, zh en: Published result
af, am, ar, as, az, be, bg, bn, br, bs, ca, cs, cy, da, de, el, en, eo, es, et, eu, fa, fi, fr, fy, ga, gd, gl, gu, ha, he, hi, hr, hu, hy, id, is, it, ja, jv, ka, kk, km, kn, ko, ku, ky, la, lo, lt, lv, mg, mk, ml, mn, mr, ms, my, ne, nl, no, om, or, pa, pl, ps, pt, ro, ru, sa, sd, si, sk, sl, so, sq, sr, su, sv, sw, ta, te, th, tl, tr, ug, uk, ur, uz, vi, xh, yi, zh en: Published result
How large a language model does your task actually need?
Choose between specialized models, small generators and larger LLMs by separating task quality, language evidence, retrieval, memory and serving capacity—with published comparisons and a worked memory example.
Choose task-relevant benchmarks, preserve language and protocol scope, build representative examples, and compare quality, tools, latency and cost without mistaking fluency or a multilingual average for task success.
RAG, CAG, KAG, fine-tuning and instruction tuning: how do they differ, and which should you choose?
When a model gives an unsatisfactory answer, does it need more training, or simply access to the right information? Through practical analogies, this article compares RAG, CAG and KAG with fine-tuning and instruction tuning: retrieving documents, caching knowledge, reasoning over relationships and changing model behavior. A comparison table and real project scenarios help distinguish missing knowledge from unsuitable behavior before committing to training, and identify the method or combination that fits the task.
Choosing the generator, embedding model and reranker for enterprise RAG
A practical three-part model stack for document assistants: small-model candidates, retrieval quality, vector and KV budgets, document preparation and a diagnostic table for finding the failing component.
LLMs on RTX 4090: what changes between 24 and 48 GB?
Exact weight files, a worked KV-cache budget, replicas versus model splitting, and the point at which professional or datacenter GPUs become useful for inference and adaptation.
An eight-bit model format does not define its execution path. Examine kernels, memory and output quality before choosing hardware for language-model inference.
AirLLM and layer-wise inference: large models, little VRAM, and the cost of waiting
Where streamed weights live, how transfer time limits generation, why KV still matters, and which research, batch and adapter workloads can benefit from layer-wise execution.
Ollama, vLLM, SGLang or llama.cpp: choosing an inference engine
How scheduling, KV cache, model formats and operational controls change the choice of inference engine, with a worked memory example and a reproducible comparison method.
Why the fastest GPU does not necessarily deliver the fastest response
More compute and a higher token rate do not always mean a faster response. This article examines time to first token versus completion time, memory and concurrency, the division of work between GPUs and LPUs, and communication costs—so infrastructure choices reflect the capacity to serve requests at the required quality and latency.
From one user to enterprise LLM serving: when to change the model, replicas or GPU
Diagnose queues, memory and availability separately. Worked replica and cloud-capacity examples show how to grow a service while preserving quality, latency and failure tolerance.
What do 100 concurrent users mean for an AI assistant?
User count alone does not define an AI assistant’s capacity. Worked examples and an interactive tool connect request rates, token lengths, queues and KV cache to GPU selection and acceptance testing.
The real cost of running an LLM: buy hardware, rent a GPU or use an API?
A worked USD scenario for API usage, GPU rental and ownership, including retries, accepted answers, operations, energy and conditional break-even calculations.
Ollama · v0.40.2 
SGLang · v0.5.20 
llama.cpp / llama-server · v0.4.1 
Text Embeddings Inference (TEI) · v1.9.4 
AirLLM · v4.0.0 
LiteLLM · v1.103.1 
Open WebUI · v0.11.4 
LM Studio · 0.4.24 Build 1 
KTransformers · v0.7.1 
