Query Qualification Pipeline
How Solid filters millions of historical queries down to the representative set used to seed semantic models and benchmarks.
During ingestion, Solid processes raw query history from the data warehouse to extract the highest-value queries for seeding semantic models. The pipeline filters aggressively.
The Five-Stage Funnel
| Stage | Operation | Example Scale |
|---|---|---|
| Raw queries | All historical queries from the query log | 35M |
| Filter irrelevant | Remove unsuccessful queries; take unique SELECT queries (remove duplicates) | 1.7M |
| Select valuable | Keep complex queries with metrics, filters, and joins — the ones the model can learn from | 1.2M |
| Map by user role | Select reliable queries from human and machine users | 350K |
| Cluster queries | Group similar queries into clusters; select cluster representatives | 10K |
What Survives
The representative queries that survive the funnel are:
- Syntactically valid SELECT statements
- Structurally complex — containing joins, aggregations, filters, metrics
- Contributed by reliable users (human analysts or trusted automation)
- Deduplicated and clustered, so redundancy is eliminated
These representative queries are used to:
- Identify relevant tables and columns for semantic model generation
- Infer relationships between tables
- Extract metric definitions
- Seed benchmark questions, via certified SQL examples
Updated 8 days ago
Did this page help you?