Research · UAM + LUMEN · 2026
Evaluation for Taxonomies at Scale
As label spaces grow, evaluation must distinguish genuine structure from duplicated concepts, brittle boundaries, and expensive model behavior.
621–5K
category scale studied
25×
lower reported cost
2
2026 submissions
01
Problem
A taxonomy can look coherent while repeating the same idea across branches. At the same time, classification systems that work at small label counts can become costly or unstable as the taxonomy grows.
02
Why it matters
Teams use taxonomies to aggregate evidence and decide what to fix. Structural duplication distorts counts, while classification cost can make an otherwise accurate system impractical.
03
Architecture
- 01Raw anecdotes
- 02Candidate hierarchy
- 03Structural measures
- 04Scale-aware classifier
- 05Error analysis
- 06Human-readable finding
04
Technical challenges
Measuring hierarchy, not just labels
Designed analysis around parent-child structure, overlap, and duplicated concepts rather than only flat accuracy.
Holding comparisons fair
Evaluated systems across a wide category range so model quality and cost could be compared under increasing complexity.
Turning anomalies into findings
Connected quantitative structure checks to a failure mode people could inspect and reason about.
05
Tradeoffs
Interpretable measures over one composite score
Separate signals made structural failure modes visible and actionable.
Scale sweep over a single benchmark point
A model that is practical at hundreds of classes may behave differently at thousands.
Submission status stated explicitly
The portfolio does not imply acceptance or publish details that are still under review.
06
Experiments
- 01Compared hierarchy evaluation behavior across generated and human-constructed structures.
- 02Swept classification scale from 621 to 5,000 categories.
- 03Tracked model quality together with inference cost rather than optimizing either in isolation.
07
Results
UAM identified a previously hidden duplication failure mode in human-built structures.
LUMEN matched a frontier model's reported accuracy at 25× lower cost across the studied scales.
The work produced two 2026 submissions: UAM as first author and LUMEN as third author.
08
Lessons learned
- Human-authored structure is a baseline, not ground truth beyond inspection.
- Cost belongs in the scientific result when it changes deployability.
- Error analysis is most valuable when it reveals a repeatable failure class.
09
Future work
- Release public artifacts when review and confidentiality constraints allow.
- Extend structural measures to deeper and evolving hierarchies.
- Study how taxonomy quality changes downstream prioritization decisions.