Colored Noise Diffusion Sampling

Diffusion models achieve state-of-the-art image synthesis, with their generative trajectories fundamentally exhibiting a spectral bias, resolving low-frequency global structures early and high-frequency fine details later. Conventional stochastic differential equation (SDE) solvers fail to account for this dynamic, naively injecting uniform white noise throughout the entire process and misusing the finite energy budget. In this work, we establish a mathematical framework that reconsiders SDE inference as a targeted, frequency-decoupled energy transfer. Leveraging this framework, we introduce Colored Noise Sampling (CNS), a novel, training-free stochastic solver. Rather than injecting uniform white noise, CNS utilizes a dynamic, timestep- and frequency-dependent schedule that more efficiently allocates injected energy toward structurally unresolved frequency bands. By actively exploiting the model's inherent spectral bias, CNS systematically steers the generated distribution toward the true data manifold. Extensive experiments demonstrate that CNS significantly outperforms standard ODE and SDE baselines as a strictly plug-and-play, inference-time sampler substitution across diverse architectures (SiT, JiT, FLUX). Compared to standard sampling on ImageNet-256, CNS achieves substantial unguided FID reductions, improving from 8.26 to 6.27 on SiT-XL/2, 32.39 to 26.69 on JiT-B/16, and 11.88 to 8.31 on JiT-H/16, while yielding consistent relative FID improvements with Classifier-Free Guidance. Project page is available at https://hadardavidson.github.io/CNS/.

📋 한 줄 요약

**[Diffusion 추론 / Colored Noise]** CNS가 SDE inference를 frequency-decoupled energy transfer로 재정의, timestep·frequency-dependent noise schedule로 spectral bias 능동 활용 — ImageNet-256 unguided FID SiT-XL/2 8.26→6.27, JiT-B/16 32.39→26.69, JiT-H/16 11.88→8.31.

🎯 핵심 기여도

Diffusion 모델이 SOTA 이미지 합성 달성, 생성 trajectory가 fundamentally spectral bias 보임 — 저주파 global structure는 빨리 해결되고 고주파 fine detail은 늦게 해결됨.
기존 SDE solver가 이 dynamic을 고려 못하고 전 과정에 uniform white noise를 naively 주입해 finite energy budget을 오용함을 진단.
수학적 프레임워크 정립 — SDE inference를 targeted·frequency-decoupled energy transfer로 재고.
Colored Noise Sampling(CNS) — training-free stochastic solver. Uniform white noise 주입 대신 dynamic·timestep·frequency-dependent schedule로 주입 에너지를 structurally unresolved frequency band에 더 효율적으로 할당. 모델 내재 spectral bias를 능동 활용해 생성 분포를 true data manifold로 systematically steer.

💡 핵심 아이디어

Diffusion의 spectral bias(저주파 먼저·고주파 나중)는 uniform white noise SDE 추론으로는 finite energy budget 낭비를 초래하므로, timestep·frequency-dependent colored noise schedule로 unresolved frequency band에 에너지를 집중 주입하면 training 없이 plug-and-play로 sample 품질 대폭 향상 가능하다.

🔬 기술적 접근법

**방법론**: CNS — Colored Noise Sampling, training-free SDE solver.
**핵심 기법**: (1) SDE inference를 targeted frequency-decoupled energy transfer로 수학적 재정의, (2) Timestep·frequency-dependent dynamic noise schedule, (3) Structurally unresolved frequency band에 에너지 집중 주입, (4) 모델 내재 spectral bias 능동 활용, (5) Plug-and-play inference-time solver로 기존 architecture(SiT·JiT·FLUX)에 substitute.

📊 주요 결과

ImageNet-256 unguided FID — SiT-XL/2 8.26→6.27, JiT-B/16 32.39→26.69, JiT-H/16 11.88→8.31.
Classifier-Free Guidance 함께 사용해도 일관 relative FID 개선.
Strictly plug-and-play로 다양 architecture(SiT·JiT·FLUX) 적용.
표준 ODE·SDE baseline 대비 significant outperform.

💭 의의 및 한계

**의의**: SDE inference의 finite energy budget을 frequency 도메인으로 명시 분배하는 새 원리, training 불요·plug-and-play로 즉시 적용 가능, FID 대폭 개선이 실용 가치. **한계**: Frequency schedule 디자인의 hyperparameter 부담, 매우 다양한 도메인·해상도에서 schedule 보편성은 추가 검증, ODE 해법과 비교는 일부 시나리오에서 trade-off 가능.

🚀 실용적 활용

기존 diffusion 모델의 즉시 sample 품질 향상.
Image·video·audio diffusion sampling 일반화.
Finite compute budget의 효율적 추론.