Kimi K3: Open Frontier Intelligence and Implementation Framework

Executive Summary

The Kimi K3 project introduces the world’s first open-source model in the 3-trillion-parameter class, designed for frontier-level intelligence in coding, knowledge work, and complex reasoning. With 2.8 trillion total parameters and 104 billion activated parameters per token, Kimi K3 achieves an approximately 2.5× improvement in scaling efficiency over its predecessor, Kimi K2.

Critical Takeaways:

• Scale and Architecture: A Mixture-of-Experts (MoE) model built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), supporting a native visual understanding and a 1-million-token context window.
• Performance Positioning: While trailing the most powerful proprietary models (Claude Fable 5 and GPT-5.6 Sol), Kimi K3 consistently outperforms other open and proprietary models across long-horizon coding, agentic, and vision benchmarks.
• Reinforcement Learning (RL) Focus: Post-training emphasizes reinforcement learning across multiple "thinking effort" levels (low, high, max), allowing for autonomous execution and compositional generalization.
• Infrastructure Innovation: Implementation of "MoonEP" for perfectly balanced expert-parallel training and "AgentENV" for resumable, high-fidelity microVM sandboxes to support million-token agentic trajectories.
• Availability: Full model weights are scheduled for release by July 27, 2026. The model is currently accessible via the Kimi API and specialized agent platforms.

---

Technical Architecture and Scaling Efficiency

The model's architecture is engineered to scale information flow across sequence length, network depth, and model width.

Hybrid Attention Mechanism

Kimi K3 utilizes a 3:1 layerwise hybrid of linear and global attention:

• Kimi Delta Attention (KDA): Extends delta-rule recurrence with a channel-wise forget gate. It uses a lower-bounded decay parameterization that allows diagonal and off-diagonal tiles to use dense Tensor Core matrix multiplications, eliminating traditional computational bottlenecks.
• Gated Multi-head Latent Attention (MLA): Provides unrestricted global content interaction. Kimi K3 applies "No Position Encoding" (NoPE) to MLA layers, relying on KDA for position-sensitive sequence mixing.

Depth and Width Innovations

• Attention Residuals (AttnRes): Unlike standard residual connections that accumulate information uniformly, AttnRes allows each layer to selectively retrieve representations from all preceding layers or block-level representations.
• Stable LatentMoE: Scales channel mixing to 896 routed experts (16 activated per token).
  • SiTU-GLU (Sigmoid Tanh Unit): Controls activation outliers while preserving the positive response of SwiGLU.
  • Quantile Balancing (QB): An auxiliary-loss-free routing mechanism that sets expert biases based on router-score quantiles to ensure perfectly balanced token loads.

Native Vision Integration

MoonViT-V2 is a 27-layer vision transformer trained from scratch using next-token prediction. This design avoids the instabilities associated with grafting pre-trained contrastive encoders (like SigLIP) onto language models. It supports high-resolution inputs up to 3584×3584 pixels through temporal pooling and pixel-shuffle operations.

---

Performance and Capability Analysis

Kimi K3 establishes a new open frontier, particularly in "vision-in-the-loop" scenarios where the model iterates between code execution and visual feedback.

Coding and Engineering

The model demonstrates "long-horizon" coding capabilities, navigating massive repositories and orchestrating terminal tools autonomously.

• GPU Compiler Development: The model successfully built "MiniTriton," a compact Triton-like compiler from scratch, including a custom DSL frontend and PTX code-generation pipeline.
• Kernel Optimization: In 24-hour sandbox sessions, Kimi K3 optimized GPU kernels, reducing AttnRes latency by over 50% and outperforming GPT-5.5 and Claude Opus 4.8.
• Chip Design: In a 48-hour autonomous run, Kimi K3 designed, optimized, and verified an RTL inference-chip prototype using open-source EDA tools.

Knowledge Work and Research

Kimi K3 advances end-to-end knowledge synthesis through interactive visualizations and deep research.

• Case Studies:
  • ASIC Industry Research: Created an interactive website covering 42 years of industry data via 120+ rounds of recursive self-improvement.
  • Astrophysics: Reproduced "I–Love–Q" universal relations, cross-validating 20+ papers and generating 3,000+ lines of Python code in two hours.

Benchmark Comparisons

---

Infrastructure and Implementation

Scaling a 2.8T model with 1M context required custom algorithm-system co-designs.

Training Infrastructure

• MoonEP: A perfectly balanced expert-parallel training scheme. It uses online planning to migrate redundant experts (bounded at E/R per rank), ensuring all ranks perform identical computation. This eliminates host-side kernel-launch overheads caused by dynamic shapes.
• Memory Management: Utilizes a unified activation manager with FP8 quantization and "remote offloading," where activations are moved to the memory of other Pipeline Parallel (PP) ranks to balance memory usage.

Agentic RL Environment (AgentENV)

To support long-lived agent trajectories, the "AgentENV" sandbox system was developed:

• MicroVM Isolation: Uses Firecracker microVMs for high-fidelity isolation, preventing kernel panics or deadlocks during aggressive agent exploration.
• Resumability: Supports pause/resume and incremental checkpointing (latencies as low as 49ms). This allows sandboxes to be "paused" while waiting for model inference, saving 98% of potential sandbox lifetime resources.

Inference Serving

• KDA-Aware Prefix Caching: Unifies KDA recurrent states and MLA KV caches into a single paged pool. It allows for fine-grained prefix hits (at 512-token boundaries) even within large 6144-token physical blocks.
• Speculative Decoding: Uses an EAGLE-3-style draft model fine-tuned from the pre-trained Multi-Token Prediction (MTP) layer.

---

Access, Pricing, and Usage

Model Access Requirements

Access to Kimi K3 is restricted to "flagship" tier users. Unlocking the model requires a minimum top-up of $1.00. Account tiers (determining concurrency, RPM, and TPM) are based on cumulative top-up amounts.

Pricing Structure

Kimi K3 utilizes pay-as-you-go pricing with separate rates for cache hits and misses:

• Input (Cache Hit): $0.30 per 1M tokens.
• Input (Cache Miss): $3.00 per 1M tokens.
• Output: $15.00 per 1M tokens.

Operational Limits

• Thinking Mode: Always enabled; cannot be turned off. Users can only adjust "Thinking Effort" (Low, High, Max).
• Context Window: Up to 1,000,000 tokens.
• Fixed Parameters: Temperature (1.0), Top_p (0.95), Presence Penalty (0), and Frequency Penalty (0) are fixed at the system level for Kimi K3.
• Vision Input: Does not support public URLs; requires base64 encoding or local file IDs (ms://<file-id>).