Rule2DRC: Benchmarking LLM Agents for DRC Script Synthesis with Execution-Guided Test Generation

Manufacturable chip layouts must satisfy thousands of geometry-based design rules, and design rule checking (DRC) enforces them by running executable DRC scripts on layouts. Translating natural language rules into correct DRC scripts is labor-intensive and requires specialized expertise, motivating LLM agents for DRC script synthesis and debugging. However, existing benchmarks have small evaluation sets and often evaluate scripts by code similarity rather than execution correctness, and prior machine learning-based methods either ignore execution feedback or require labeled test layouts as agent's input. To this end, we introduce Rule2DRC, a large-scale benchmark for DRC script coding agents with 1,000 rule-to-script tasks and 13,921 evaluation chip layouts for execution-based scoring. Rule2DRC provides an evaluation pipeline that measures functional correctness via DRC execution outcomes without requiring evaluation layouts as input to the agent. We also propose SplitTester, a tester agent for program selection that uses execution feedback to generate discriminative test cases and separate previously indistinguishable candidate scripts, substantially improving Best-of-N selection performance in this domain. We release the code at https://github.com/snu-mllab/Rule2DRC.

한 줄 요약

**[Chip Design / LLM Agent]** Rule2DRC가 1,000 rule-to-script task·13,921 실 chip layout으로 execution-based scoring 제공 — code similarity가 아닌 functional correctness 측정, SplitTester가 discriminative test로 Best-of-N 선택 substantially 향상.

핵심 기여도

Manufacturable chip layout이 thousands of geometry-based design rule을 만족해야 하고, Design Rule Checking(DRC)이 layout에 실행 가능한 DRC script를 돌려 이를 enforce함을 배경 소개.
자연어 규칙을 정확한 DRC script로 번역하는 작업이 labor-intensive·전문성 요구로 LLM agent 활용 동기.
기존 벤치마크의 한계 지적 — 평가 set이 작고 종종 execution correctness가 아닌 code similarity로 평가, prior ML 방법은 execution feedback 무시·labeled test layout을 agent 입력으로 요구.
Rule2DRC — DRC script 코딩 agent의 large-scale 벤치마크 — 1,000 rule-to-script task·13,921 evaluation chip layout으로 execution-based scoring 제공. Evaluation pipeline이 functional correctness를 DRC 실행 결과로 측정, evaluation layout을 agent 입력으로 요구하지 않음. SplitTester — program selection용 tester agent — execution feedback을 사용해 discriminative test case를 생성·이전에 indistinguishable이던 candidate script들을 분리해 Best-of-N selection 성능을 substantially 개선.

핵심 아이디어

DRC script 합성 평가는 code similarity가 아닌 execution-based functional correctness로 측정되어야 하며, evaluation layout을 agent 입력으로 요구하지 않는 평가 파이프라인과 execution feedback 기반 discriminative test 생성 tester(SplitTester)가 결합되면 LLM agent의 실용 합성 능력이 의미 있게 측정·향상된다.

기술적 접근법

**방법론**: Rule2DRC 벤치마크 + SplitTester tester agent.
**핵심 기법**: (1) 1,000 rule-to-script task·13,921 chip layout 큐레이션, (2) Execution-based scoring으로 functional correctness 직접 측정, (3) Evaluation layout을 agent 입력으로 요구하지 않는 hidden test 평가 protocol, (4) SplitTester — execution feedback으로 discriminative test case 생성, (5) Best-of-N candidate script 분리·선택 정확도 향상.

주요 결과

1,000 task·13,921 layout의 large-scale 벤치마크 제공.
SplitTester가 Best-of-N selection 성능 substantially 향상.
Code similarity 기반 평가 한계 해소.
Code: github.com/snu-mllab/Rule2DRC.

의의 및 한계

**의의**: EDA·chip design 분야의 LLM agent benchmark 표준 정립, execution-based scoring으로 평가 신뢰성 강화, SplitTester가 Best-of-N selection의 일반 보조 도구로 확장 가능. **한계**: DRC script 도메인 특화로 다른 EDA·HDL 도메인 일반화 추가 검증, 1,000 rule이 실제 chip 설계의 thousands rule을 완전 커버하지 않을 수 있음, SplitTester가 candidate 다양성에 의존.

실용적 활용

LLM 기반 EDA·chip design 자동화.
DRC script 합성·디버깅 agent 평가·개선.
Best-of-N selection 일반 LLM 코딩 agent에 응용.