Flattened vs. Hierarchical SODA Models

SODA answered how to scale a flattened discrete-audio LM. It did not answer whether flattening is the right thing to scale. So: at matched training compute, does a CSM/Moshi-style hierarchical factorization model speech better?

SODA turns each 80 ms audio frame into 8 Mimi RVQ tokens (1 semantic + 7 acoustic) and flattens them into one long stream, which a single decoder models with uniform cross-entropy. Our scaling study derived compute-optimal allocation rules for that setup. But an allocation rule only covers the design space it was fit on—if a different factorization has a different frontier, the rules move with it. Testing that is the natural next step on our own result.

The alternative, such as CSM (Sesame) and Moshi (Kyutai), factorizes hierarchically: a backbone transformer whose positions are “steps” (one text token, or one whole audio frame summed from its codebook embeddings), plus a small depth transformer that predicts the remaining acoustic codebooks inside each frame. It is what much of the field builds. Comparisons between the two families, though, tend to differ in data, tokenizer and compute budget as well as in factorization, which makes the contribution of the factorization itself hard to isolate. We built this campaign so the two arms differ in exactly one thing.

There is also a standing practical reason to want the answer before scaling further. The hierarchy carries a KV cache of length T instead of 8T, and decodes audio for roughly 7× fewer FLOPs per generated second. If it is also competitive in quality, that changes what is worth serving.

Two positions this study could support:

• Flattened. Within-frame left-to-right attention over all 8 codebooks is worth what it costs, and a single head over a 144,644-token vocabulary is a strong joint model.
• Hierarchical. Spending sequence positions 8× more cheaply frees backbone capacity for long-range structure, and a small depth transformer suffices for within-frame acoustics. We do not apply a delay pattern, keeping the comparison against the flattened arm as tight as possible.

• A single NLL number cannot rank the two families. An NLL is an average of per-token bits, so it needs weights: how much should a text token count against a late acoustic one? Scoring the same four models three different ways reverses the ranking — flattened wins by 17% when every token counts equally, but under Moshi-style or semantic-only weighting every hierarchical run beats every flattened run.
• The families split along a clean axis. At matched compute the hierarchy roughly halves ASR word error and wins every semantic, lexical and text benchmark that resolves above chance; the flattened model wins every acoustic NLL component and speaker similarity.
• Generation reliability is structural. Every hierarchical run terminates on all 1,088 zero-shot TTS prompts; four of the five flattened runs — at both budgets — fail to terminate on 24–71% of them, getting stuck emitting silence instead of closing the utterance, which was an issue we encountered in the flattened SODA models.
• A graded per-codebook loss weighting is a Pareto improvement over Moshi. Moshi weighting is flat: text and semantic tokens count 100, and every acoustic codebook counts 1. Grading it instead — weighting acoustic codebook k by 100^(1−k/7), so the early codebooks keep most of the weight and the late residuals are starved — matches or beats it on every semantic metric while recovering most of its generation deficit. This is the recipe we scaled.
• And the split survives at scale. Carrying the recipe to SODA-Hier (1.1B, 1.17×10²⁰ FLOPs), it beats a flattened model trained with 1.5× its compute (SODA-Flat-1.8e20) on ASR and every text/lexical task—and loses to one trained with 0.77× its compute (SODA-Flat-9e19) on SALMon and both TTS metrics.

Two arms over the same Mimi RVQ tokens (8 codebooks × 2048 entries, 12.5 Hz), trained on utterance-level interleaved speech and transcripts:

• Arm F (flattened) — the SODA architecture: one Qwen3 decoder over the flat 144,644-vocab stream, 4096-token windows (≤41 s of audio).
• Arm H (hierarchical) — a Qwen3 backbone whose positions are steps, with a unified 130,308-way head predicting the next step's text/semantic token, plus a small Qwen3 depth transformer with per-codebook 2048-way heads for the 7 acoustic codebooks. 1024-step windows (~82 s of audio).

The comparison is only meaningful when the other moving parts are controlled: the same corpus (15.7B flat tokens ≈ 42k audio-hours), the same tokenizer and id space, the same Qwen3 blocks, and the same solver-derived batch/learning-rate/step rules with no per-run tuning. Critically, both arms define normalized joint distributions over the same token sequences—we assert flat_tokens == steps + 7 × frames per document at preprocessing.

Three loss recipes appear below, differing only in how much weight each token type carries in the cross-entropy. (i) Uniform weights every token equally—the published SODA recipe, which means 7 of every 8 audio tokens are late-RVQ acoustic residuals. (ii) Moshi weights text and semantic tokens 100 and acoustic tokens 1, concentrating capacity on content at the expense of acoustic detail. (iii) Decay is the graded middle ground introduced in Finding 3: acoustic weights fall geometrically across the 7 codebooks instead of dropping off a cliff.

One scope note. A full isoflop suite is expensive—SODA's own required 64 models—and we have not run one for the hierarchical arm. What follows is a compute-matched comparison at the smaller budgets where such a comparison is affordable, plus a single scaled-up run of the recipe it selects. So we can say which factorization is better at these budgets; we cannot draw a hierarchical compute-optimal frontier.

Held-out NLL is a proper scoring rule and both arms model identical token sequences, so the comparison is mathematically sound. The catch is what “the NLL” even means when 7 of every 8 audio tokens are late-RVQ acoustic residuals. A scalar NLL is an average of per-token bits, and an average needs weights: you have to decide how much a text token counts relative to a semantic one, relative to a late acoustic one. Uniform cross-entropy is one such choice of weights; a weighted cross-entropy is another. Training under a weighted loss is exactly maximum likelihood under the matching weights.

So we held the four trained 2×2 models fixed and re-scored each one three times, changing nothing but the weights used to average its per-token bits: uniform (every token counts equally), Moshi (text and semantic tokens count 100, acoustic tokens 1) and semantic-only (codebook 0 of each frame, nothing else). Those are deliberately the same three shapes as the three training losses — which is the whole problem. Mean nats per scored token, lower is better:

So “which factorization reaches lower NLL” has no weight-free answer: each model wins under the weights its own training loss optimized, which makes any single-number comparison across losses close to circular. The fix is to stop reporting a scalar and report the NLL vector (text, semantic, per-codebook acoustic). Its components turn out to be excellent capability predictors—across architectures and losses, not just within one. Spearman correlations over the 12-run matrix:

How this relates to the scaling result we found previously. SODA's headline method finding—that capabilities are predictable from validation loss—replicates here, and it stands. What this work adds is its scope. That relationship holds within a model family scored under its own training measure, which is exactly the setting it was established in (flattened + uniform CE, varying scale). It does not license comparing a scalar NLL across different training losses or architectures: there, the number you get is a choice of importance measure, and changing that choice reverses the ranking. Within the flattened family the original practice behaves exactly as before (ρ = +1.00 for NLL → sWUGGY); the hierarchical family breaks under uniform NLL and is restored by scoring it under its own training measure (sWUGGY +0.09 → +0.77, tWUGGY +0.14 → +0.89).

The headline pair at 3×10¹⁸ FLOPs and d=768. The hierarchical model wins nearly every capability: it roughly halves ASR word error at both zero and two shots, edges TTS-WER, and leads sWUGGY, tStoryCloze, tBLIMP and tWUGGY (per-run numbers in the table below). The flattened model's wins are all acoustic: uniform-scored SALMon (68.6 / 66.7), TTS speaker similarity (0.332 vs 0.230), and every acoustic NLL component (cb1–cb7 at 3.58–4.41 vs 4.57–4.94 nats), which is also what drives its 17% uniform-NLL advantage. Roughly 72% of that advantage is architectural rather than a loss-weighting effect (P1b). sBLIMP and sStoryCloze sit at chance for both, as in the original study at this budget.

Generation behavior separates the arms beyond the scores, and the pattern is arm-wide rather than pair-specific. The flattened model fails to produce a terminating generation on 24% of TTS prompts, and even its surviving transcripts are deletion-dominated (18.2% DEL vs 10.9% SUB—truncated or incomplete speech), where the hierarchical model terminates on all 1,088 and errs by substitution (17.8% SUB, 7.7% DEL). The non-terminating cases share a characteristic failure mode: instead of ending the utterance the flattened model gets stuck in silence and runs on to the generation cap. Every hierarchical run in the campaign terminates on 100% of prompts; four of five flattened runs drop 24–71%.

Architecture and loss weighting are two ways of doing the same thing: de-prioritizing acoustic tokens. Crossing them gives a 2×2 that is strongly non-additive (interaction ≈ −65 bits per audio-second).

On the flattened arm, weighting is a large, double-edged lever. Adding Moshi weighting buys most of the hierarchical arm's text and semantic profile—ASR 37.1 → 28.5, tBLIMP 62.0 → 64.6—at the cost of nearly the entire acoustic-NLL advantage (91 of the 96-bit gap). And the cost is not confined to likelihoods: termination falls from 76% to 35% of prompts, and TTS-WER on even the surviving third doubles, to 66.7.

On the hierarchical arm it is a small lever for likelihood capabilities and a large one for generation. Removing it costs some ASR (19.4 → 22.5) and buys back 26 bits of acoustic NLL—but TTS improves dramatically, to 17.8 WER at 0.330 similarity, the campaign's best TTS model on both axes, matching the flattened arm's speaker similarity at half its word error.

The asymmetry is the insight. Once the architecture has delegated acoustics to a depth module, down-weighting them in the loss has little left to do; on the flattened arm the loss is the only such mechanism, so it does all the work and pays the full acoustic price. Generation is where the substitution breaks—synthesis is the one task where the acoustic axis is load-bearing, and both moshi-weighted cells pay for their tilted measure with worse, less reliable generation.

Moshi and uniform are the two extremes of one knob, and the 2×2 shows each buying one axis at the other's expense. P5 asks whether something in between can beat both — rather than simply landing halfway. Instead of a flat 100/100/1 or 1/1/1, the acoustic codebooks get geometrically decaying weights:

All three recipes on the hierarchical arm at 3×10¹⁸ and d=768, so only the weighting differs:

Decay takes the whole semantic column outright and recovers most of the generation ground Moshi weighting gives up. The last two columns show why: it models cb1 as well as uniform weighting does, and pays for that only on the highest-order residual, which little downstream depends on. This is the recipe we carried into the scale-up.

These are smaller probes than the isoflop suite behind the SODA paper — three widths, three depth sizes and two budgets, one run each. They settled the design choices the campaign needed, but they are observations at the budgets we ran rather than fitted trends.

• Width. Under an isoflop budget, wider means fewer steps. Across d = 512/768/896 neither arm's NLL moves much — a few bits per audio-second, on a curve that looks flat rather than peaked — so we cannot locate either arm's optimum from these runs, and we do not claim a width trend. The capability metrics vary more over the same range (the hierarchical d=896 run is the campaign's best model, ASR 16.1 / 15.1), but with one run per point and no seed replicates we cannot separate a width effect from run-to-run variation. Settling this needs the isoflop treatment, which we have not run for the hierarchical arm. The one difference too large to be noise is on the flattened arm, where TTS termination falls 100% → 76% → 29% as width grows.
• Depth-transformer size. Shrinking the depth transformer from the dd384L4 default to dd256L2 costs 15 bits per audio-second; growing to dd512L6 buys 1.4 — the default sits at the knee. Likelihood metrics are nearly insensitive, and the small variant even posts the best hierarchical sWUGGY and tBLIMP: everything the paired benchmarks measure lives in the backbone. Generation is the exception, and it is where an under-sized depth module is paid for (TTS-WER 41.0 against 29.6 for the default). Small, but not below the knee.
• Budget. From 1×10¹⁸ to 3×10¹⁸ both arms improve on essentially every axis, and no ordering between them changes, on any metric — including the TTS split, where the hierarchy has the better WER and the flattened model the better speaker similarity at both budgets. Two budgets is a check rather than a slope: it says the cheap pilots predict the more expensive ones here, not how the gap behaves much further out.
• Decode cost. Analytic, so unlike the rest of this section it does not depend on run count at all. The flattened model pays its 144,644-way head on all 8 tokens of every frame and carries a KV cache of length 8T; the hierarchy pays a 130,308-way head once per frame plus 2,048-way heads within it, at KV length T. At the P1 model sizes that is 47.4 GFLOPs per generated audio-second against 6.7 — a 7.0× advantage for the hierarchy, which depends only on the factorization and so carries over to the scale-up unchanged.

The campaign's recommendation—hierarchical factorization, graded decay weighting, wide backbone, small depth—was selected on 1×10¹⁸/3×10¹⁸ pilots. SODA-Hier asks whether it survives an order of magnitude more compute. It is essentially a scaled-up P5: the same arm and the same recipe at 1.1B parameters (a d1536/L15 backbone of ~990M plus a dd1152/L4 depth transformer of ~122M), trained on a 396k-hour Yodas+Emilia corpus for 1.168×10²⁰ FLOPs—38.9× the campaign's 3×10¹⁸ point—and evaluated on the identical battery.

Keep the scale in perspective. At 1.17×10²⁰ FLOPs this is a larger ablation, not a large-scale run: SODA's published run used 1.3×10²² FLOPs, about 110× more. SODA-Hier extends the campaign's compute ladder by an order of magnitude and shows the recipe holds there.

Two results deserve emphasis. ASR more than halves, 18.9 → 7.7 zero-shot; the campaign's best ASR model managed 16.1, so the scaled run is at less than half that error on the same test set with identical decode parameters. And the acoustic axis moves with scale: the campaign's honest summary was that flattened models were the better acoustic models, with speaker similarity their strongest evidence (best flattened run 0.332, every hierarchical run 0.19–0.33). SODA-Hier reaches 0.362—above every flattened run in the campaign—while also posting the best TTS-WER. But those flattened runs are 3×10¹⁸ models. The harder test is against flattened models that are also scaled up.

The campaign could only answer the flat-versus-hierarchical question at 1×10¹⁸/3×10¹⁸. Our original isoflop sweep supplies the missing large-scale reference: flattened models trained near compute-optimally at their scale, evaluated with the same harness. SODA-Hier's budget falls between two of those points, and the parameter counts bracket too (851M / 1.1B / 1.23B):

• SODA-Flat-9e19 — 851M (d1408/L14), 0.77× SODA-Hier's training budget.
• SODA-Flat-1.8e20 — 1.23B (d1664/L17), 1.5× SODA-Hier's training budget.

That gives a reading rule. Where SODA-Hier beats the 1.5× model it is clearly better—it wins against more flattened compute than it had. Where it loses to the 0.77× model it is quite likely worse—it loses against less. Between the brackets is a draw.

The campaign's axis split survives at scale, against compute-optimal opponents, on both sides. SODA-Hier is clearly better on every semantic, lexical, text and ASR metric that resolves—beating even the flattened model with 1.5× its compute—and quite likely worse on the acoustic and generation-quality axes, losing SALMon and both TTS metrics to even the 0.77× model. The flattened arm's home turf extends, at compute-optimal scale, to TTS intelligibility as well as speaker similarity. The observation above—that SODA-Hier beats the campaign flats on TTS—does not carry over to flattened models that are also scaled up.

We note that SODA-Hier is not compute-optimal at its budget. Its width was anchored to CSM rather than isoflop-optimized, while the SODA-Flat models are near-optimal at theirs. This handicap makes the hierarchy's wins conservative and its losses ambiguous.