Kimi K3: Engineering a 2.8 Trillion Parameter Frontier

Introduction: The New Axis of Scaling

The launch of Kimi K3 represents a pivotal moment in the evolution of open-source artificial intelligence. As the world’s first open-source model in the 3-trillion-parameter class (specifically 2.8T), Kimi K3 is not merely an exercise in size, but a strategic push along two distinct scaling axes: the pre-trained foundation scale and test-time scaling (reasoning effort).

While the industry has recently pivoted toward test-time scaling, the open-source ecosystem has historically lagged in foundation scale, creating a gap between open weights and the most powerful proprietary systems. Kimi K3 bridges this by combining an unprecedented 2.8T parameter architecture with sophisticated reinforcement learning (RL) designed for a 1-million-token context. While we currently trail the most powerful proprietary models—Claude Fable 5 and GPT-5.6 Sol—Kimi K3 establishes a new frontier for open intelligence, outperforming all other models in our evaluation suite.

Kimi K3 Key Specifications:

• Total Parameters: 2.78 Trillion
• Activated Parameters: 104.2 Billion
• Context Window: 1 Million tokens (Native)
• Modality: Native Multimodal (MoonViT-V2)
• Architecture: Hybrid Kimi Delta Attention (KDA) & Block Attention Residuals (AttnRes)
• Efficiency: 2.5x overall scaling efficiency improvement over Kimi K2

---

The Architecture of Information Flow: Three Dimensions of Scaling

To manage a model of this magnitude, we moved beyond standard Transformer scaling. Kimi K3 optimizes information flow across sequence length, network depth, and model width using an algorithm-system co-design approach.

Dimension 1: Sequence Length via Hybrid KDA-Gated MLA

Traditional softmax attention is restricted by a KV cache that grows linearly with sequence length. Kimi K3 utilizes a 3:1 mixing ratio: three layers of Kimi Delta Attention (KDA) followed by one Gated Multi-head Latent Attention (MLA) layer.

The "physics" of KDA involves a channel-wise forget gate and delta-rule recurrence. A critical innovation here is Lower-Bounded Decay. Previous linear attention models suffered from serial bottlenecks in diagonal-tile computation because the reciprocal cumulative decay could overflow. By using a scaled sigmoid to bound log-decay ($g_{\min} = -5$), we ensure every retention factor satisfies $\alpha > e^{-5}$. This keeps the cumulative log-decay over a 16-token tile within the $(-80, 0)$ range, ensuring reciprocal rescaling factors stay within the BF16 dynamic range ($e^{80}$). This allows us to utilize dense Tensor Core matrix multiplications for all causal tiles, bypassing the serial bottlenecks of prior architectures.

Dimension 2: Network Depth via Block Attention Residuals (AttnRes)

In standard architectures, information is compressed into a single state through sequential residual accumulation. AttnRes allows each layer to selectively retrieve representations from all preceding layers using learned pseudo-queries. To minimize memory and communication overhead, we partition the 93 layers into 8 blocks of 12 layers. This reduces cross-stage communication in pipeline parallelism and drops the overhead from $O(Ld)$ to $O(Nd)$.

Dimension 3: Model Width via Stable LatentMoE

Kimi K3 employs an extremely sparse Mixture of Experts (MoE) framework, activating 16 out of 896 routed experts. Optimization is further stabilized by Per-Head Muon, which applies Newton-Schulz orthogonalization to attention projections independently for each head, equalizing learning dynamics across the 2.8T scale.

---

Overcoming the Challenges of Extreme Sparsity

Sigmoid Tanh Unit (SiTU-GLU)

To control activation outliers in a 2.8T model, we replaced SwiGLU with SiTU-GLU. SwiGLU is unbounded, increasing the risk of activation explosion. SiTU-GLU applies a smooth cap via a scaled tanh function:

$$\text{SiTU-GLU}(x) = \left[ \beta_1 \tanh\left(\frac{W_g x}{\beta_1}\right) \odot \sigma(W_g x) \right] \odot \left[ \beta_2 \tanh\left(\frac{W_u x}{\beta_2}\right) \right]$$

We set $\beta_1=4$ (gate branch) and $\beta_2=25$ (up branch), preserving the SwiGLU-like local response while strictly bounding growth to $\beta_1 \beta_2 = 100$.

Quantile Balancing (QB)

Traditional auxiliary losses fail at the 896-expert scale. We use Quantile Balancing, an auxiliary-loss-free method that derives expert-specific biases from router-score histograms. This histogram estimator recovers the pooled global batch quantile across sharded tokens without massive communication overhead, ensuring perfectly balanced expert loads during training.

Native Vision: MoonViT-V2

Unlike Kimi K2.5, we trained our vision encoder, MoonViT-V2 (0.4B parameters), entirely from scratch. Initializing from pre-trained SigLIP models caused persistent gradient spikes and instability at this scale. By training with next-token prediction from the outset, we achieved stable optimization and superior alignment for structural and textual visual cues.

---

Infrastructure: The Physics of Running a 3T Model

• MoonEP (Expert Parallelism): We developed a "Perfectly Balanced" scheme using dynamic redundant experts (with a proven upper bound of $E/R$ per rank). This allows for static computation shapes, eliminating host-device synchronization stalls on the critical path and enabling zero-copy communication.
• Million-Token RL (AgentENV): Training agentic models at 1M context requires persistent environments. Our AgentENV system uses Firecracker microVMs that support sub-second pause/resume and fork operations. We achieved a 6.5x memory overcommit ratio, allowing the system to maintain complex states across trajectories spanning millions of tokens.
• Speculative Decoding (EAGLE-3): We repurposed the model’s native Multi-Token-Prediction (MTP) layer into a draft model for speculative decoding. By optimizing the LK loss (negative log of the acceptance rate), we achieved significant inference speedups without compromising the 2.8T backbone's precision.

---

Scaling Efficiency & Frontier Performance

The integration of KDA, AttnRes, and Stable LatentMoE resulted in a 2.5x overall scaling efficiency improvement over Kimi K2.

Case Study: GPU Compiler Development (MiniTriton)

Kimi K3 autonomously developed MiniTriton, a compact Triton-like compiler. It features a tile-level IR layer and a PTX code-generation pipeline. Crucially, MiniTriton is stable enough to sustain end-to-end nanoGPT training, with a loss curve that closely tracks the PyTorch reference. Its Tensor Core matmul path approaches 90% of the hardware roofline, rivaling cuBLAS.

Case Study: Chip Design & Research

• Autonomous Chip Design: In a 48-hour run, Kimi K3 designed an inference chip prototype. The design integrates 1.46M standard cells and achieves a decode throughput of 8,700 tokens/s in RTL simulation.
• Computational Astrophysics: Kimi K3 reproduced the "I-Love-Q" universal relations in two hours—a task normally requiring weeks for a human researcher. It cross-validated 20+ papers and synthesized an interactive HTML dashboard.
• Interactive Industry Research: The model generated a comprehensive website detailing 42 years of AI ASIC history, synthesized from over 11,000 pages of quarterly reports.

Vision-in-the-Loop

Native multimodal support enables iterative "vision-in-the-loop" workflows. Kimi K3 edited its own teaser video, handling 56 source clips with frame-accurate beat synchronization—a task that typically requires days of professional human editing.

---

Conclusion: The Open Frontier

Kimi K3 is currently the #1 model on the WebDev Arena and #2 on the Vals Index. By releasing the full model weights, we aim to accelerate global research into the 3-trillion-parameter regime.

Key Limitations & Transparency:

• Thinking History Sensitivity: Generation quality may become unstable if the full historical thinking content is not passed back in multi-turn sessions.
• Excessive Proactiveness: Due to training on long-horizon tasks, the model may occasionally make proactive decisions for the user; we recommend explicit behavioral constraints in system prompts.

Full model weights will be available by July 27, 2026. Developers can access Kimi K3 today via the Kimi API or our GitHub repository to begin building the next generation of agentic intelligence.