Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models

Normalization layers in modern large language models (LLMs) consist of a deterministic normalization operation and a learnable scale vector. While the normalization operation has been extensively studied, the scale vector remains poorly understood despite its ubiquitous use. In this work, we present a systematic study of scale vectors in LLMs from the perspectives of expressivity, optimization, and architectural structure. First, we show empirically that although scale vectors constitute only a negligible fraction of model parameters, removing them substantially degrades LLM pre-training. Our theory further shows that, in Pre-Norm architectures, scale vectors do not increase expressivity; instead, they improve optimization through a self-amplifying preconditioning effect on subsequent linear mappings. Second, we investigate the role of weight decay for scale vectors. By distinguishing Input-Norm and Output-Norm layers, we theoretically show that weight decay is beneficial for the former but harmful for the latter, due to their distinct roles in optimization and expressivity. Third, motivated by this understanding, we propose three lightweight and complementary improvements to scale vectors: branch-specific heterogeneity, improved placement around linear mappings, and magnitude-direction reparameterization. Both theory and experiments show that each improvement yields consistent gains. Finally, we combine these improvements into a unified scale-vector strategy and evaluate it through extensive LLM pre-training experiments on dense and mixture-of-experts models ranging from 0.12B to 2B parameters, across multiple optimizers and learning rate schedules, under industrial-scale token budgets. The unified strategy consistently achieves lower terminal loss than well-tuned baselines and exhibits more favorable scaling behavior, while adding negligible parameter and computational overhead.

📋 한 줄 요약

**[LLM Scale Vector 분석]** Scale vector가 negligible 파라미터지만 제거 시 pre-training 심각 저하 — Pre-Norm에서 expressivity 아닌 self-amplifying preconditioning 효과·Input/Output-Norm 구분으로 weight decay 영향 반전, branch-specific heterogeneity + magnitude-direction reparam의 통합 전략이 0.12B-2B에서 baseline outperform.

🎯 핵심 기여도

현대 LLM의 normalization layer가 deterministic normalization과 learnable scale vector로 구성되며, normalization 연산은 광범위 연구되었지만 scale vector는 ubiquitous하게 사용됨에도 poorly understood 임을 지적.
LLM scale vector의 systematic study — expressivity·optimization·architectural structure 관점.
Scale vector가 모델 파라미터의 negligible 비중이지만 제거 시 LLM pre-training이 substantially degrade. Pre-Norm에서 expressivity 증가는 없으나 subsequent linear mapping의 self-amplifying preconditioning 효과로 optimization 개선.
Input-Norm·Output-Norm 구분으로 weight decay가 전자에 유익·후자에 해롭다는 이론 결과 도출. 이 이해로부터 3 lightweight·complementary improvement 제안 — branch-specific heterogeneity, linear mapping 주변의 개선 배치, magnitude-direction reparameterization. 통합 전략을 dense·MoE 모델 0.12B-2B에서 industrial-scale token budget·multiple optimizer·LR schedule로 광범위 pre-training 평가, 일관 lower terminal loss·favorable scaling 달성.

💡 핵심 아이디어

Negligible 파라미터의 scale vector가 LLM 학습에 critical한 이유는 expressivity가 아닌 self-amplifying preconditioning 효과이며, Input-Norm vs Output-Norm 구분으로 weight decay 영향이 반전된다는 이해가 branch-specific heterogeneity·improved placement·magnitude-direction reparameterization의 3 통합 개선으로 이어진다.

🔬 기술적 접근법

**방법론**: Scale vector의 expressivity·optimization·architecture 체계 연구 + 3-part 개선 전략.
**핵심 기법**: (1) 경험적으로 scale vector 제거의 LLM 학습 저하 입증, (2) Pre-Norm 이론 — self-amplifying preconditioning이 optimization 개선의 메커니즘, (3) Input-Norm vs Output-Norm 구분으로 weight decay 영향 반전 이론, (4) 3 개선 — branch-specific heterogeneity, linear mapping 주변 개선 배치, magnitude-direction reparameterization, (5) Dense·MoE 0.12B-2B·multiple optimizer·LR schedule에서 industrial-scale 검증.

📊 주요 결과

Scale vector 제거 시 LLM pre-training substantially 저하.
3 개선 각각 일관 gain.
통합 전략이 0.12B-2B dense·MoE에서 well-tuned baseline 대비 일관 lower terminal loss.
More favorable scaling behavior, negligible parameter·계산 overhead.

💭 의의 및 한계

**의의**: Negligible component(scale vector)의 critical 역할을 expressivity가 아닌 preconditioning으로 새 해명, weight decay·Norm 위치의 정밀 이론, industrial-scale 검증으로 실용 가치, dense·MoE·optimizer·LR schedule 전반의 robustness. **한계**: 0.12B-2B 범위 외 더 큰 모델 효과 추가 검증, 이론의 Post-Norm 등 다른 architecture 일반화 여지, 3 개선 각각의 contribution 분리 ablation 명시 부족.

🚀 실용적 활용

LLM pre-training의 lightweight 성능 boost.
Normalization layer 설계 원리.
Dense·MoE 모델의 weight decay 전략 가이드.