Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws

Scaling laws have made language-model performance predictable from model size, data, and compute, but they typically treat the optimizer as a fixed training detail. We show that this assumption misses a fundamental axis of representation scaling: how effectively the optimizer converts added FFN width into utilized spectral capacity. Using eigenspectra of feed-forward network representations, measured through soft and hard spectral-ranks, we find that the same Transformer architecture realizes markedly different spectral scaling laws when trained with different optimizers. Holding architecture and width schedule fixed, AdamW exhibits weak hard-rank scaling (β=0.44) on rare-token (TAIL) representations where learning is known to be hardest, whereas Muon achieves linear scaling (β=1.02) in the same regimes, a 2.3times increase in the scaling exponent. This difference is not reducible to validation loss: AdamW configurations can match low-rank Dion variants in perplexity, under extended training, while exhibiting sharply different spectral geometry, demonstrating that matched loss does not imply matched representation structure. Hard--soft rank asymmetry further reveals that optimizers differ not only in how much capacity is realized, but also in how that capacity is structured across eigenmodes. To disentangle optimizer effects from architectural ones, we compare against architectural interventions (e.g., attention rank and positional encoding), and find that optimizer-induced spectral shifts often exceed the architectural effects. These results suggest optimization as a first-class axis of representation scaling, motivating optimizer--architecture co-design.

한 줄 요약

**[Scaling Laws / Optimizer-Architecture Co-Design]** 동일 Transformer가 옵티마이저에 따라 다른 spectral scaling law 보임 — AdamW는 TAIL representation에서 hard-rank β=0.44, Muon은 β=1.02로 2.3× 큰 스케일링 지수, matched loss여도 spectral geometry는 sharply 다름.

핵심 기여도

Scaling law가 모델 크기·데이터·연산으로 language model 성능을 예측 가능하게 했지만 옵티마이저를 보통 fixed training detail로 다룬다는 한계 지적.
이 가정이 representation scaling의 근본 축 — 옵티마이저가 추가된 FFN width를 얼마나 효과적으로 utilized spectral capacity로 변환하는지 — 를 놓침을 입증.
FFN representation의 eigenspectra(soft/hard spectral-rank)로 측정 시 동일 Transformer 아키텍처가 옵티마이저별로 markedly 다른 spectral scaling law를 실현함 발견.
학습 hard-rank scaling 비교 — AdamW는 rare-token(TAIL) 표현에서 weak(β=0.44), Muon은 동일 영역에서 linear(β=1.02), 2.3× 큰 스케일링 지수.

핵심 아이디어

Validation loss가 같아도 옵티마이저는 spectral geometry를 sharply 다르게 형성하며, FFN의 utilized spectral capacity는 architecture·width 조절보다 옵티마이저 선택에 더 강하게 의존한다 — optimization을 representation scaling의 first-class 축으로 격상해 architecture와 co-design해야 한다.

기술적 접근법

**방법론**: FFN representation eigenspectra를 통한 spectral-rank scaling 분석.
**핵심 기법**: (1) Soft·hard spectral-rank 정의, (2) 동일 Transformer architecture·width schedule 고정 후 옵티마이저만 변경, (3) AdamW·Muon·Dion variant 비교, (4) Hard-soft rank asymmetry 분석으로 capacity 양·구조 분리, (5) Attention rank·positional encoding 같은 architectural intervention과 옵티마이저 효과를 비교해 분리.

주요 결과

AdamW hard-rank β=0.44 vs Muon β=1.02 (TAIL 표현), 2.3× 큰 스케일링 지수.
AdamW가 low-rank Dion variant와 perplexity 매치 가능하나 spectral geometry는 sharply 상이.
Hard-soft rank asymmetry로 옵티마이저가 capacity의 양뿐 아니라 eigenmode 분포 구조까지 결정함을 입증.
Optimizer-induced spectral shift가 architectural intervention 효과를 자주 초과.

의의 및 한계

**의의**: scaling law 연구에 옵티마이저라는 새 축 추가, matched loss ≠ matched representation의 정량 입증, optimizer-architecture co-design의 필요성 정립. **한계**: language model FFN 중심으로 multimodal·vision 일반화 추가 검증, 새 옵티마이저(Muon, Dion) 등 비교에 한정, downstream task 성능과 spectral-rank의 인과 연결은 partial.

실용적 활용

LLM 학습의 옵티마이저 선택 가이드.
Architecture-optimizer co-design.
Rare-token·long-tail 학습 진단.