BOHM: Zero-Cost Hierarchical Attribution for Compound AI Systems

Compound AI systems route tasks through hierarchies of specialised components. Attribution is dominated by Shapley-based methods (SHAP), which decompose a coalition value function into per-component marginal contributions and require evaluation of the system on arbitrary component subsets. That requirement fails for third-party APIs, opaque endpoints, and agentic orchestrators that concentrate routing on a few tools, leaving most coalitions un-evaluable from the deployed orchestrator. We introduce BOHM, which extracts a hierarchical attribution tree directly from the routing weights such systems already maintain: leaf attribution is the path product of root-to-leaf routing weights; level-k attribution is the induced distribution over depth-k nodes. The method has zero marginal cost, requires no access to component internals, and provides multi-resolution attribution at every level simultaneously, which flat methods cannot offer at any evaluation budget. BOHM and SHAP answer different questions and converge when the deployed router routes near-optimally. On 18 LLMs in a 3-level hierarchy over 880 LiveCodeBench problems, BOHM yields Kendall tau=0.928; SHAP reaches tau=0.980 at 9,000x more coalition evaluations per seed. On a 5-driver, 7-benchmark agentic study (35 cells, complete coverage), drivers concentrate routing on a single tool (top-share median 0.65), and cell-level tau(BOHM,SHAP) is predicted by whether the driver's top pick is the empirically best tool (mean +0.22 vs ~+0.01). On a US Census hierarchy (475 leaves, 4 levels), BOHM recovers ground-truth rankings at every level (tau up to 0.722). BOHM satisfies efficiency, monotonicity, symmetry, and weak suppression but not Shapley's additivity. It is best understood as a complementary primitive: a multi-resolution decomposition computable wherever routing state exists, whose disagreement with Shapley is itself diagnostic.

한 줄 요약

**[Compound AI / Attribution]** BOHM이 라우팅 가중치만으로 hierarchical attribution tree 추출 — zero marginal cost·black-box endpoint 호환, 18 LLM 3-level 880 LiveCodeBench에서 SHAP(9,000× cost) Kendall τ=0.980 대비 0.928 달성.

핵심 기여도

Compound AI 시스템이 specialised 컴포넌트의 hierarchy를 통해 task를 route함을 배경 소개. Attribution은 SHAP 기반 방법이 dominate — coalition value function을 per-component marginal 기여로 분해·임의 컴포넌트 부분집합 평가를 요구하지만, 이 요구는 third-party API·opaque endpoint·few tool에 라우팅을 집중하는 agentic orchestrator에서 fail함을 진단.
BOHM 도입 — hierarchical attribution tree를 시스템이 이미 유지하는 routing weight에서 직접 추출. Leaf attribution은 root-to-leaf routing weight의 path product, level-k attribution은 depth-k node의 induced 분포.
Zero marginal cost·컴포넌트 내부 접근 불필요·flat method가 어떤 evaluation budget에서도 제공 불가한 multi-resolution attribution을 모든 레벨에서 simultaneously 제공.
BOHM·SHAP이 다른 질문에 답하지만 deployed router가 near-optimally 라우팅할 때 수렴. 18 LLM·3-level hierarchy·880 LiveCodeBench problem에서 BOHM Kendall τ=0.928, SHAP은 9,000× coalition 평가 cost로 τ=0.980; 5-driver·7-benchmark agentic 연구에서 driver가 single tool에 라우팅 집중(top-share median 0.65), cell-level τ(BOHM,SHAP)은 driver의 top pick이 경험적 최고 tool인지에 의해 예측됨(평균 +0.22 vs ~+0.01). US Census hierarchy(475 leaf·4 level)에서 모든 레벨 ground-truth ranking 회복(τ up to 0.722). Efficiency·monotonicity·symmetry·weak suppression 만족, Shapley additivity는 미만족 — Shapley와의 disagreement 자체가 진단적.

핵심 아이디어

Compound AI 시스템 attribution의 SHAP 의존 요구(임의 coalition 평가)가 black-box agentic 시스템에서 fail하므로 라우팅 가중치 자체를 attribution 신호로 활용해야 하며, BOHM의 path product가 zero-cost·multi-resolution decomposition을 제공하면서 Shapley와의 disagreement는 그 자체로 라우팅 품질 진단 정보가 된다.

기술적 접근법

**방법론**: BOHM — routing weight 기반 hierarchical attribution tree.
**핵심 기법**: (1) Routing weight에서 leaf attribution을 root-to-leaf path product로 정의, (2) Level-k attribution은 depth-k node의 induced distribution, (3) Zero marginal cost — coalition 평가 불필요, (4) Multi-resolution attribution을 모든 level에서 simultaneously 제공, (5) Shapley와 비교·disagreement를 diagnostic으로 활용.

주요 결과

18 LLM·3-level hierarchy·880 LiveCodeBench: BOHM Kendall τ=0.928 vs SHAP τ=0.980(9,000× cost).
5-driver·7-benchmark agentic 연구(35 cell): top-share median 0.65, cell-level τ이 driver top pick의 정확도에 의해 예측됨(평균 +0.22 vs ~+0.01).
US Census hierarchy(475 leaf·4 level): 모든 레벨 ground-truth ranking 회복(τ up to 0.722).
4 공리(efficiency·monotonicity·symmetry·weak suppression) 만족, Shapley additivity 미만족.

의의 및 한계

**의의**: Compound AI attribution을 black-box·agentic 환경으로 확장, zero-cost multi-resolution 분해의 첫 실용 방법, Shapley와의 차이를 diagnostic primitive로 재정의. **한계**: Shapley와 다른 질문에 답하므로 직접 대체 아님, deployed router가 sub-optimal일 때 SHAP과의 gap 존재, Shapley additivity 미만족이 일부 응용에서 제한.

실용적 활용

LLM ensemble·tool routing의 attribution 분석.
Agentic orchestrator의 컴포넌트 기여도 진단.
Black-box API attribution 진단.