Single Token Geometry: Manifold Tearing in MoE Training

# Single Token Geometry 02: Manifold Tearing

Chinese Edition: 单标几何 02: 流形撕裂 (https://x.com/BetaTomorrow/status/2084751162994307258)

I was preparing Single Token Geometry: Data Complexity, the second entry in this series , when DeepSeek V4 dropped. I set it aside immediately.

What caught my attention wasn’t the benchmark numbers, impressive as they are. It was a paragraph buried in Section 4.2.3 of the technical report:

> “We identified that the occurrence of spikes is consistently tied to outliers in the MoE layers, and the routing mechanism itself appears to exacerbate the emergence of these outliers.”

And then, with unusual candor:

> “Although a comprehensive theoretical understanding of their underlying mechanisms remains an open question for now, we are sharing them openly to foster further exploration by the community.”

I appreciate DeepSeek’s transparency here. They found three empirical fixes that work. They admitted they don’t fully understand why. And they published anyway. That intellectual honesty is rare, and it’s exactly the kind of opening that geometric thinking is built for.

This post is my attempt to supply part of what they left open: a geometric account of what training instability is in a system like DeepSeek V4, and why their mitigations work.

The thesis is simple and I want to state it plainly upfront:

Loss spikes in MoE training are not optimization failures. They are manifold tears.

## What Is a Manifold Tear?

Before we can call something a tear, we need to be precise about what a manifold is — and what it requires.

A manifold is a space that is locally Euclidean. The canonical example is the surface of the Earth: globally curved and non-Euclidean, but zoom in far enough on any point and it looks flat, like a patch of ℝ². More formally, a topological manifold requires that every point has a neighborhood homeomorphic to ℝⁿ. A smooth manifold further requires that the transition maps between overlapping neighborhoods , called charts , are differentiable. A Riemannian manifold additionally equips the space with a metric tensor, giving you notions of distance and curvature.

The key requirement at the foundation of all of this is continuity. Not just continuity :smoothness. The whole apparatus of differential geometry, and crucially the whole apparatus of gradient-based optimization, assumes that the space being traversed is smooth enough to support derivatives everywhere.

A manifold tear is a violation of this assumption. Precisely: a tear occurs when a map between manifold regions fails to be continuous,  when a point has no well-defined image, or when nearby points are sent to regions that are not nearby in the target space. A tear is not a large deformation. A large deformation of a manifold is still a manifold. A tear is a discontinuity , the manifold’s local Euclidean structure breaks down at the torn point.

In the context of a transformer’s residual stream, we can define this operationally:

> A manifold tear in a transformer’s residual stream is a discontinuity in the layer-to-layer transport map, induced by a discrete routing decision that is inconsistent with the local geometry of the representation manifold at that point.

Now let’s see exactly how that happens in DeepSeek V4 — and how the three mitigations each address a different stage of the same failure cascade.

## The Geometry of the Residual Stream

Think of a single token passing through a transformer. At each layer, its hidden state is a point in ℝᵈ , a high-dimensional vector. As the token passes through successive layers, this point traces a trajectory through representation space. The claim that this trajectory lives on a manifold is implicit in how we train: gradient descent assumes a smooth loss landscape, backpropagation assumes differentiable operations everywhere, and the expressive power of deep networks comes precisely from learning smooth, structured transformations of this space.

In a standard transformer, every token passes through the same FFN at each layer. The map from layer l to layer l+1 is the same for all tokens. The manifold deforms, but it deforms continuously — the same function applied everywhere.

In a Mixture-of-Experts (MoE) transformer like DeepSeek V4, this changes fundamentally. At each MoE layer, a router examines the token’s hidden state and routes it to one or more experts, a discrete, Top-k selection from hundreds of possible FFNs. The composite map from hidden state to new hidden state is now:

> hidden state → routing decision → expert application → new hidden state

The routing decision is discontinuous by construction. It is a discrete selector. A token sitting at position x in representation space gets routed to expert E₁. A token at position x + ε , infinitesimally close, might get routed to expert E₂. These two experts were trained on different regions of the manifold. Their outputs are not guaranteed to be close to each other.

This means the layer-to-layer transport map has discontinuities at routing boundaries. The manifold is already torn, structurally, at every expert boundary. This is not a bug — it is the price of the MoE architecture’s efficiency. But it is a latent geometric fragility that becomes catastrophic under the right conditions.

## The Failure Cascade

The paper’s observation, that spikes are tied to outliers in MoE layers, and that routing exacerbates them, describes a specific failure cascade. Geometrically, it unfolds in three stages:

Stage 1: Local curvature spike. An activation in an MoE expert grows very large. The SwiGLU gate or linear component produces an extreme value. This is not yet a tear — it is a region of very high curvature on the representation manifold. The local geometry is still technically defined, but it is poorly conditioned. Small changes in input produce large changes in output. The optimizer’s gradient step, calibrated for normal curvature, begins to overshoot.

Stage 2: Chart inconsistency. The router, updating synchronously with the backbone, now operates on a shifted manifold. At step t, the backbone parameters θₜ define a representation space Mₜ. But the router, which just updated, is making routing decisions as if the manifold were Mₜ₋₁. A token at position x on Mₜ gets routed as if it were on a manifold that no longer exists. This is a chart inconsistency: the coordinate chart (the routing assignment) no longer matches the local geometry of the point being mapped. The token is being sent to the wrong expert for where it actually lives in the representation space.

Stage 3: Tear amplification. The misrouted token enters an expert that wasn’t trained on its region of the manifold. The expert produces an outlier output — a point far from where the token should be in the next layer’s representation space. This discontinuous jump is the tear. And if the residual mapping matrices are expansive (spectral norm > 1), the tear grows as it propagates through subsequent layers. By layer L, a token that was slightly misrouted has been thrown far from its correct manifold region. The loss spike is the accumulated geometric damage becoming visible in the training objective.

This is the cascade: high curvature → chart inconsistency → routing boundary discontinuity → tear amplification → loss spike.

DeepSeek V4’s three mitigations each interrupt this cascade at a different stage, see Mathematical Considerations for DeepSeek V4 for more detail (https://open.substack.com/pub/deepmanifold/p/mathematical-considerations-for-deepseek)

## What the Paper Leaves Open

The authors write:

> “Although a comprehensive theoretical understanding of their underlying mechanisms remains an open question for now...”

The geometric framing offered here suggests what that theoretical understanding might look like: a formal account of how routing boundary discontinuities propagate through residual streams, how spectral norm bounds on residual mappings contain this propagation, and how temporal consistency conditions in discrete selectors can be enforced as a parallel transport rule.

This is not a complete theory. It is a vocabulary, a set of geometric concepts precise enough to ask the right questions. The manifold is torn by construction in every MoE layer. The question is whether that tear propagates catastrophically or stays bounded. DeepSeek V4’s three mitigations are three answers to that question, each operating at a different scale.

Single Token Geometry is the project of taking seriously the idea that what happens to one token, passing through one forward pass, is already geometrically rich enough to explain phenomena we currently attribute to optimization dynamics. Loss spikes are one example.

## Manifold Tearing Is Not DeepSeek’s Problem Alone

It is worth being precise about scope. Manifold tearing is not unique to DeepSeek, or to MoE architectures, though MoE does make it worse, by introducing routing boundaries that are discontinuous by construction. The pathology is more general.

We noticed and discussed this in our 2024 paper, Deep Manifold Part 1: Anatomy of Neural Network Manifold — specifically Section 3.6, Learning Transformation:

> “This explains the zig-zag pattern observed in the loss curve during the slow decline stage of almost all foundation model training, suggesting that training is struggling to converge — a hidden bottleneck identified in our analysis. A model is considered to have converged effectively when the standard deviation of its loss values stabilizes at less than 5.”

That zig-zag pattern is not noise. It is not a quirk of the optimizer or the learning rate schedule. High loss deviation is an indication of manifold tearing , the loss surface is not smooth but discontinuous, and the optimizer is crossing those discontinuities rather than descending through them. The standard deviation of the loss is, in this framing, a roughness measure of the representation manifold. When it stabilizes, the manifold has settled into a geometry the optimizer can navigate. When it spikes, the manifold is tearing.

This means manifold tearing is visible in almost every foundation model training run ever published. It has been attributed to many things — learning rate warmup, batch size scheduling, data curriculum, optimizer hyperparameters. These attributions are not wrong. But they are proximate causes. They describe the conditions under which tearing is more or less likely. They do not identify the root cause.

The root cause, beyond model architecture, is data.

Specifically: data complexity. The representation manifold that a model learns is not chosen by the architect , it is induced by the training data. A dataset with discontinuous structure, sharp distributional boundaries between domains, conflicting label geometries, or extreme token frequency imbalances, induces a representation manifold with discontinuities baked in from the first forward pass. The model is not tearing a smooth manifold. It is trying to learn a manifold that was never smooth to begin with. The routing boundaries in MoE are a second-order effect; the data boundaries are primary.

This is the argument of the next article in this series.

Single Token Geometry: Data Complexity will ask what it means, geometrically, for data to be complicit in manifold tearing: why certain data compositions make smooth representation manifolds impossible, and what that implies for how we should think about dataset curation, not as an engineering convenience, but as a geometric necessity. (https://x.com/BetaTomorrow/status/2050461737363951997)

Manifold tearing will continue to appear throughout this series. DeepSeek V4 gave us a precise, honest, and unusually well-documented instance of it. The mitigations they found are real and they work. But they are defenses against a pathology whose origin sits upstream of the architecture — in the data the model is asked to learn from, and in the geometry that data imposes on the representation space before training even begins.

## Closing

In classical differential geometry,  "tearing" implies an external force acting upon a pre-existing, well-defined surface. In a forward problem, the manifold is usually predefined. The geometry is given first, and the computation moves on top of it. Because the coordinate structure is fixed, manifold tearing is not usually the central issue.

Learning is different. Learning is an inverse problem. The manifold is not given; it has to be learned from data while the model is simultaneously moving on, traverses it. That means the geometry, the coordinates, and the transport map are all changing during training. 

'Tearing' in this sense is a topological failure of convergence. It occurs when the model forces conflicting logical constraints into the same dimension before local homeomorphy is achieved. The result is not a break in a known geometry, but a failure of the geometry to ever become continuous—leading to the routing instabilities and activation outliers we observe in complex models.

So manifold tearing is not just a failure to move correctly on a known manifold. It is a failure that occurs while the manifold itself is being formed. That is why MoE routing, activation outliers, and residual amplification can become so unstable: the token is not merely crossing a boundary; the boundary is still being learned.

Two primary blind spots prevent us from fully understanding neural networks, which remain a 'black box' to many. These are: 1) the failure to view learning as an inverse problem, and 2) the failure to treat neural networks as forms of numerical computation.

- Single Token Geometry 01: Topology (https://x.com/BetaTomorrow/status/2045685439156437196)

- Single Token Geometry 03: Data Complexity (https://x.com/BetaTomorrow/status/2050461737363951997)

- Mathematical Considerations for DeepSeek V4 (https://x.com/BetaTomorrow/status/2084846064201261375)

This article is now listed under Single Token Geometry Series  in Deep Manifold, Two Years Later: 2024–2026. (https://x.com/BetaTomorrow/status/2080057647160824276)

- What is Deep Manifold ? (https://x.com/BetaTomorrow/status/2070113382347461038)

- Neural Fixed-Point Field (https://x.com/BetaTomorrow/status/2075335417189257256)

- Learning Is an Inverse Problem (https://x.com/BetaTomorrow/status/2063266723336499444)

- Deep Manifold Model Scaling (https://x.com/BetaTomorrow/status/2090563679666491438)

- Early Validations of Deep Manifold (https://x.com/BetaTomorrow/status/2081606689506447748)

- Deep Manifold in the Real World (https://x.com/BetaTomorrow/status/2072044191023329753)