Collect and Offline Process

Before Solid can answer a business question, it first has to learn how your data is actually structured and used. The offline process is the background pipeline that does this — reading only metadata (schema structure, query history, usage patterns) from your warehouses, lakes, and BI tools, never your underlying business data, and compressing millions of historical queries down to the small set that reflects real, repeated business logic. It runs on a schedule, never in the request path of a question a user asks.

What the Offline Process Produces

The offline process does not build semantic models. It builds the foundation a semantic model is built from:

• A documented catalog — tables, columns, and schemas annotated with descriptions and quality signals
• Relationships between tables inferred from real usage, not just what was formally declared
• Extracted metric definitions
• An enriched business glossary
• A searchable index over all of the above

What comes out is both a catalog and a graph. The catalog lets you search for a specific table, query, column, or metric directly. The graph — built from the relationships above — lets you start at any one of those and expand outward to the dimensions and supporting facts around it.

Semantic models are built on top of this foundation, on demand, scoped to a specific business use case — see Semantic Model Lifecycle for that process.

The Pipeline

flowchart LR
  connect["Connect"]
  collect["Collect"]
  normalize["Normalize"]
  analyze["Analyze"]
  infer["Infer"]
  rank["Rank"]
  enrich["Enrich"]
  indexNode["Index"]

  connect --> collect --> normalize --> analyze --> infer --> rank --> enrich --> indexNode

Each stage runs in sequence on a schedule — never in the request path of a question a user asks:

What Solid Collects

Column profiling happens in two passes — schema first, then, once real usage shows which columns actually matter, a second pass to sample those specifically, rather than profiling every column in the warehouse up front. See Data Warehouses, Databases, and BI Tools for the full, current list of supported connectors.

How Queries Become Understanding

A warehouse's query log is mostly noise — exploratory one-offs, broken drafts, admin housekeeping. Solid's analysis separates the small fraction that reflects real, repeated business logic from everything else: filter, break down, cluster, and keep only what's repeatable.

Rather than treat each query as a single unit, Solid looks at the pieces that build it — how tables are joined, how results are filtered, which groupings are used — and looks for repetition in those pieces across many queries. A calculation a hundred analysts independently wrote the same way is a strong signal; one person's different version is an outlier, not a pattern worth keeping. Who ran the query matters too: queries that fail to execute, or come from users whose queries rarely succeed, count for less.

For example: if ninety-nine queries compute a daily average by dividing by 365, and one divides by 360, that one is flagged as an outlier and dropped — not because Solid knows what "365" means, but because it doesn't match the pattern everyone else converged on.

The result is a dramatic compression: tens of millions of logged queries typically yield on the order of ten thousand representative ones — complex, deduplicated, and clustered. See Query Qualification Pipeline for the full funnel.

When Context and Query Patterns Conflict

The outlier filter above handles noise — one-off queries that don't match what everyone else does. A different problem is legitimate disagreement: supplied context (wikis, glossary entries, BI metric definitions) that describes a metric one way, while query history shows analysts computing it another — or two real business definitions of the same term used in different parts of the organization.

Solid does not silently pick a winner when these signals conflict. Instead, it surfaces the competing definitions and lets a modeler decide how to resolve them:

This is a human-in-the-loop step. Solid presents the evidence from each source; the modeler decides which definition belongs in a given semantic model. See Semantic Model Lifecycle for where that certification happens.

Inferring Relationships

Not every warehouse has its relationships formally declared — plenty of tables are joined constantly in practice with no foreign key ever defined to say so. Solid infers relationships from four kinds of evidence, in order of trust:

1. Detected primary keys (PK) — Solid automatically probes each table's columns to identify which columns uniquely identify every row, including combinations of columns that form a composite primary key. This happens even when no key is formally declared in the DDL.
2. Detected unique keys (UK) — Columns and column combinations that enforce uniqueness but are not the primary identifier are detected separately, since they often make valid join targets.
3. Detected foreign keys (FK) — Solid probes for columns whose values consistently appear as primary or unique keys in another table, making them strong candidates for a foreign-key relationship even without a formal constraint.
4. Observed join behavior — when two columns are joined together often enough, across enough distinct queries, that's treated as evidence of a real relationship even without a formal declaration.
5. Statistical evidence from the column profiles themselves — null rates and naming patterns typical of a key column serve as a further signal when other evidence is thin.

Where these sources disagree, actively detected key constraints win over inferred join behavior, and observed behavior wins over statistical inference alone. This is how Solid can propose a correct join path even in a warehouse where the DDL alone would leave it a guess, and how it catches relationships that exist in practice but were never formally enforced.

Deciding What's Worth Surfacing

A typical warehouse has far more tables and columns than are useful for answering business questions — staging tables, deprecated columns, one-off exports. Before anything is enriched or indexed, every asset is scored on three signals — naming, usage, and freshness — combined with any preferences you've set, and that score decides whether it's surfaced at all. This is what keeps the catalog, and eventually the semantic models built from it, focused on what's actually worth an analyst's or an agent's attention, rather than a mirror of every table in the warehouse.

Keeping the Catalog Current

The first connection runs a full pass, looking back across a 90-day window of query history by default. After that, Solid checks for updates on a schedule — commonly weekly, though it's configurable per connection — reading only what's new or changed and running it through the same filtering and analysis. Assets that disappear from a source (a dropped table, a retired report) are marked removed rather than left stale.

This refresh also powers drift detection: when a run surfaces a change relevant to an existing semantic model — a column deleted from the warehouse, or a column seeing rising query activity that is not yet in the model — Solid raises it as a recommendation for a modeler to review, never applying it automatically. See Data Drift Recommendations for how those are surfaced and resolved.

What Solid Reads — and What It Doesn't

Collection is metadata-first: schema structure, query text, and aggregate, non-PII column samples used to make categorical filters more accurate. Solid does not store your raw business data, and PII is detected and excluded at collection time — before it's written anywhere in Solid's system. Every connection uses a dedicated, least-privilege credential scoped to exactly what you've authorized.

See When Does Solid Execute SQL? and Security Architecture for the full breakdown of what Solid can access, and under what identity, at every stage.