Stanford’s Marin 535B AI Model Training Is Live-Streamed Openly
Stanford professor and Simile AI founder Percy Liang is putting one of the most expensive and technically complex stages of modern AI development under public observation.
Through the Marin Open Lab, Liang and collaborators have launched open pretraining of Marin 535B-A23B, a 535-billion-parameter Mixture-of-Experts (MoE) model with 23 billion active parameters per token.
Rather than publishing only the final model or selected research results, the project is exposing the training process itself. The approximately three-month “hero run” is being streamed and documented with live loss curves, data-mixture information, hardware telemetry, training configurations, engineering artifacts, and experimental logs.
The initiative represents a more radical interpretation of open-source AI: open development. Researchers can observe not only what the model eventually becomes, but how a frontier-scale model behaves while it is being built.
🧠 Marin 535B-A23B: The Hero Training Run #
The scale of the Marin project places it firmly in the frontier-model training category.
| Parameter / Metric | Specification |
|---|---|
| Model Architecture | Marin 535B-A23B, Mixture-of-Experts |
| Total Parameters | 535 billion |
| Active Parameters per Token | 23 billion |
| Training Dataset | 18.75 trillion tokens |
| Training Mix | 80% pretraining, 20% midtraining |
| Compute Hardware | 11 NVIDIA GB200 NVL72 racks |
| GPU Count | Approximately 792 GB200 GPUs |
| Estimated Compute | ~2.7 × 10²⁴ FLOPs |
| Training Duration | Approximately 3 months |
The distinction between total and active parameters is fundamental to the architecture.
Although Marin contains 535 billion parameters, only approximately 23 billion are activated for an individual token. This MoE design allows the model to maintain a very large parameter capacity without requiring every parameter to participate in every forward pass.
A multi-stage scaling strategy #
Before launching the full-scale training run, the Marin team prepared a four-rung scaling ladder.
The smaller models range from approximately 1.6B-A61M to 27.7B-A1.2B, allowing researchers to evaluate scaling behavior before committing the full compute budget to the largest run.
This methodology is particularly important at frontier scale because training failures can become extraordinarily expensive once hundreds of billions of parameters and hundreds of GPUs are involved.
Smaller-scale experiments provide an opportunity to test architecture, optimization settings, data mixtures, and scaling assumptions before increasing compute expenditure.
🔬 Opening the AI Pretraining Black Box #
Most frontier-model development remains highly proprietary.
Companies typically keep training datasets, data mixtures, hyperparameters, loss curves, infrastructure failures, and optimization decisions confidential. Even when final model weights or research papers are released, much of the engineering process remains invisible.
Marin takes the opposite approach.
Data pipeline transparency #
The project exposes information about how training data is constructed and mixed.
Researchers can inspect domain-level composition, processing pipelines, and sampling strategies rather than treating the training corpus as an opaque input.
This is valuable for understanding how changes in data composition affect optimization and model performance.
Live loss tracking #
Training loss is being exposed in real time through public dashboards.
Instead of seeing a polished loss curve after training has completed, researchers can observe the optimization process as it unfolds, including unexpected spikes, changes in convergence behavior, and deviations from predicted scaling trajectories.
This makes the project a live experiment in large-scale optimization.
Open engineering artifacts #
The project also exposes engineering infrastructure that is normally hidden from external researchers.
These artifacts include:
- Training code
- Hyperparameter configurations
- Experimental logs
- Issue tracking
- Scaling experiments
- Pre-registered loss forecasts
- Hardware and training information
The combination creates a much more complete record of how a frontier model is trained.
📈 Scaling Laws Become a Live Experiment #
One of the project’s most valuable scientific components is the ability to compare theoretical scaling predictions against an actual frontier training run.
Researchers can establish expected loss trajectories before training and then compare those predictions with real measurements.
This provides a direct test of assumptions about:
- Compute scaling
- Dataset composition
- Model size
- Optimization efficiency
- Training stability
- MoE routing behavior
- Convergence rates
At smaller scales, deviations may be relatively inexpensive to investigate. At the 535B scale, the consequences of an incorrect assumption can translate into enormous compute costs.
The Marin approach effectively turns the entire training run into a large-scale reproducible experiment.
🛰️ Public Observation Infrastructure #
The project uses multiple public channels to expose different parts of the training process.
Dataset information #
A public Google Cloud Storage portal provides access to information about domain-level data mixtures and related dataset details.
This gives researchers visibility into the composition of the material being used to train the model.
Real-time training dashboard #
Training loss and scaling-law measurements are available through Weights & Biases (W&B) dashboards.
The live monitoring system allows observers to track the model’s optimization trajectory while training is still underway.
Open-source repository #
The Marin codebase, issue tracker, and experimental records are maintained publicly through the project’s GitHub repository.
This allows external researchers to inspect implementation details, follow technical discussions, and potentially reproduce individual experiments at smaller scales.
🌐 Open Development Instead of Open Weights #
Traditional open-model releases generally follow a relatively simple pattern: a research organization develops the model privately and then releases some combination of weights, code, documentation, or evaluation results.
Marin goes further by exposing the development process itself.
The distinction is important.
Open weights show the final product.
Open development shows how the product was created.
That includes the failures.
If an optimization run becomes unstable, researchers can potentially observe the event rather than seeing only a cleaned-up description afterward. If a particular hyperparameter configuration performs poorly, the evidence can remain visible in the training history.
For AI research, this creates a valuable record of negative results that would normally disappear from published papers and polished model releases.
🧪 Community Debugging in Real Time #
The live nature of the project also changes how research collaboration works.
External researchers can monitor training behavior and investigate anomalies while the experiment is still running.
One reported example involved a sudden norm spike around training step 500. Community members analyzed the behavior and linked it to router_bias, an MoE token-balancing heuristic interacting with the model’s optimization dynamics.
The significance is not simply that one anomaly was explained.
It demonstrates a different research workflow in which a large distributed community can inspect training behavior, propose hypotheses, and discuss potential causes while the underlying experiment is still active.
This effectively turns model training into a collaborative debugging environment.
🎓 Connecting Frontier Engineering With AI Education #
The Marin Open Lab also complements Liang’s Stanford teaching work, particularly CS336: Language Models From Scratch.
The course focuses on the mechanics behind modern language models, while Marin provides an unusually large-scale real-world example of those principles operating in production-like research infrastructure.
This creates a bridge between two traditionally separate environments:
Academic instruction → research implementation → frontier-scale training
Students and researchers can move beyond theoretical discussions of optimization, scaling laws, data mixtures, and MoE routing and observe how those concepts behave when deployed across hundreds of GPUs.
💻 Why the Marin Experiment Matters #
The most important contribution of Marin may not ultimately be the model itself.
A 535-billion-parameter model is an impressive engineering achievement, but the project’s broader scientific value comes from exposing the process used to train it.
Frontier AI development currently contains substantial amounts of tacit knowledge: how teams respond to loss spikes, how they diagnose hardware failures, how data mixtures evolve, and how scaling predictions hold up against actual training.
Much of that knowledge never enters academic literature.
By making the training process observable, Marin can create a detailed empirical record of frontier-model engineering that other researchers can study, critique, and build upon.
🚀 A New Model for Frontier AI Research #
Marin 535B-A23B represents more than another large language model.
It is an experiment in whether frontier-scale AI development can be conducted with an unprecedented level of transparency.
The project exposes the model architecture, data strategy, scaling experiments, loss behavior, infrastructure, and engineering decisions while the training run is still in progress.
That approach has obvious costs. Public experimentation can expose failures, create additional operational complexity, and reveal technical information that conventional AI labs would normally treat as proprietary.
But it also creates something difficult to obtain from closed development: a detailed, real-time record of how a frontier model is actually built.
If the Marin Open Lab approach proves sustainable, it could establish a new standard for open AI research—one where the scientific artifact is not merely the final model, but the entire path from scaling experiments and data preparation to optimization, failure analysis, and final training.