← Selected work

Applied science · Public summary · 2026

Autonomous Taxonomy Systems at Amazon

The hard part was not generating a hierarchy. It was replacing a multi-notebook expert workflow with a system that calibrates itself, beats the manual baseline, and explains its own quality.

Public summary only

0.74

F1 vs. 0.71 baseline

<18h

onboarding cycle

2.7×

faster extraction

01

Problem

Creating a taxonomy for a new feedback domain required a seven-notebook workflow: scientists selected examples, tuned prompts, ran clustering, and manually stitched a three-level hierarchy. Each domain consumed five to seven days of expert time and kept scientists in every step.

02

Why it matters

Taxonomies determine which customer issues become visible to product teams. If onboarding remains expert-bound, scale stops at scientist availability; if the hierarchy overlaps or duplicates concepts, the resulting issue counts become misleading.

03

Architecture

  1. 01Raw feedback
  2. 02Self-calibrating KIE
  3. 03Four-pillar evidence
  4. 04BERTopic L3 discovery
  5. 05LLM hierarchy induction
  6. 06Grounded explainability
The public pattern is a two-phase system: mine and validate a small, diverse in-domain example set, then run cached batch extraction at scale. Structured phrases feed density-based L3 discovery; the LLM classifies those clusters into a disjoint hierarchy, while deterministic attribution computes every quality claim before the LLM renders it in plain language.

04

Technical challenges

01

Calibrating without labelled training data

The pipeline mines candidates from the target domain, checks every extraction against its source, ranks quality and structural diversity, and selects a compact few-shot set automatically.

02

Preserving context through clustering

Generic phrases collapsed unrelated products into catch-all clusters. Explicitly bracketing category and aspect fields restored the context the embedder needed to separate them.

03

Knowing where the LLM belongs

Recursive density clustering produced overlapping parent themes. The system pivoted to density discovery for L3 and constrained LLM classification for L1/L2, where model behavior was more reliable.

04

Explaining metrics without invented claims

Python identifies the exact node driving each quality score and validates paths and rankings. The LLM only translates grounded attribution into a concise explanation.

05

Tradeoffs

KIE before taxonomy generation

Directly embedding raw anecdotes lost the four distinct issue pillars and amplified noise. Structured extraction added a stage, but made downstream taxonomies separable and auditable.

Two-model calibration and extraction

A stronger judge is reserved for the quality-critical calibration step; a lower-cost extractor handles full-scale cached inference.

Algorithms decide; the LLM communicates

Deterministic code owns metric attribution and validation. The model is used for semantic labeling and readable language, not numerical truth.

06

Experiments

  1. 01Ablated KIE entirely and confirmed that raw anecdotes were too noisy to form four coherent pillar taxonomies.
  2. 02Compared zero-shot and few-shot extraction, plus in-domain and cross-domain examples; selected compact in-domain few-shot calibration to preserve domain-specific failure patterns.
  3. 03Swept calibration sizes from 10 to 1,000 examples instead of choosing the pilot size by intuition.
  4. 04Compared direct one-shot prompting, bottom-up BERTopic, and top-down Leiden against the same manual baseline and holistic evaluation framework.

07

Results

Reduced a five-to-seven-day scientist workflow to an autonomous run under 18 hours.

Improved taxonomy F1 from a 0.71 manual baseline to 0.73 with BERTopic and 0.74 with Leiden.

Reduced missing Category and Aspect extractions from roughly 23.8% to 0.7% in the documented evaluation.

Cut extraction time by 2.7× and inference cost by 53% through prompt caching and cross-region execution.

Shipped the KIE system as a production container for multi-domain, million-record workloads.

08

Lessons learned

  • LLMs were better at classifying discovered structure than discovering the structure itself.
  • Self-calibration is a product capability: it determines whether a new user can onboard a domain without a scientist.
  • An explainable metric needs deterministic attribution before fluent language.
  • A test-set improvement is not enough; production behavior still decides whether an idea survives.

09

Future work

  • Complete production launch of the grounded metrics-explainability layer.
  • Extend calibration and hierarchy evaluation across more domains and drift patterns.
  • Connect structural quality measures to downstream issue-discovery decisions.