Databricks

Overview

Solid connects to your Databricks workspace to build insights into your data lineage, usage, and quality. To do this, Solid requires access to:

• The schema of your Databricks datasets
• Query logs from the Query History table in Databricks
• System tables for metadata extraction

You have two options for connecting Databricks to Solid:

What Is Collected

Automatic Pull (Recommended)

Best for: Production use with continuous monitoring and updates.

Step 1: Create a Personal Access Token

Solid connects to Databricks using a Personal Access Token (PAT). Service Principal tokens are not currently supported.

1. In your Databricks workspace, click your username in the top-right corner and select Settings
2. Go to Developer → Access tokens
3. Click Generate new token, give it a name (e.g., solid-integration), set a lifetime, and click Generate
4. Copy and store the token securely — it will not be shown again

For detailed steps, see the Databricks personal access token documentation.

Step 2: Grant Permissions

System Tables Access

Solid extracts metadata from system tables. Follow these steps:

1. Enable System Tables in Databricks by following the Databricks system tables documentation

   • Execute the instructions for each workspace that has Unity Catalog

2. Grant permissions to the Databricks user whose token you created in Step 1:

GRANT USE CATALOG ON CATALOG system TO <your_databricks_user>;
GRANT SELECT ON system.information_schema TO <your_databricks_user>;
GRANT SELECT ON system.query TO <your_databricks_user>;

Catalog Access

Grant the user access to each catalog you want Solid to integrate. This cascades to all schemas within the catalog:

GRANT USE CATALOG ON CATALOG <CATALOG> TO <your_databricks_user>;
GRANT USE SCHEMA ON CATALOG <CATALOG> TO <your_databricks_user>;
GRANT SELECT ON CATALOG <CATALOG> TO <your_databricks_user>;

Example:

GRANT USE CATALOG ON CATALOG analytics_prod TO [email protected];
GRANT USE SCHEMA ON CATALOG analytics_prod TO [email protected];
GRANT SELECT ON CATALOG analytics_prod TO [email protected];

User List Access

To retrieve the list of workspace users, the Databricks account used must have Workspace admin privileges.

Step 3: Create SQL Warehouse

1. Create a SQL Warehouse following the Databricks SQL Warehouse documentation

2. Grant your Databricks user access to the SQL Warehouse:

   • Click the Permissions button at the top right of the SQL Warehouse configuration page
   • Add the user/service principal
   • Grant "Can use" permission

Step 4: Verify Data Access

Confirm that the SQL Warehouse has access to the catalogs, schemas, and tables you want to integrate.

Run the following commands in the Databricks SQL editor to verify access:

-- Verify system tables access
SHOW TABLES IN system.information_schema;

-- Verify catalog access
SHOW CATALOGS;

-- Verify schema access (replace <CATALOG> with your catalog name)
SHOW SCHEMAS IN <CATALOG>;

-- Verify table access (replace with your catalog and schema)
SHOW TABLES IN <CATALOG.SCHEMA>;

-- Verify detailed table information
DESCRIBE EXTENDED <CATALOG.SCHEMA.TABLE>;

Important: If you're testing with a different user token than what will be configured in Solid, the test results may not reflect the actual permissions Solid will have.

Step 5: Configure Solid

Once you've completed the setup:

1. Log into the Solid platform
2. Navigate to Settings → Integrations → Databricks
3. Enter the following information:

4. Click Test Connection
5. Select the Catalogs and Schemas you want to monitor
6. Click Save

Solid will begin analyzing your Databricks metadata.

Manual Pull

Best for: Testing Solid or if you can't grant direct access yet.

Steps

1. Export schema and query history using Python notebooks in Databricks
2. Archive the exported files (ZIP or GZIP)
3. Upload to the Solid Azure Storage container

Export Schema Information

Open a notebook in Databricks and paste this code:

from pyspark.sql import Row

# Lists to hold the schema information
schema_info = []

# Step 1: Get all schemas
schemas = spark.sql("SHOW SCHEMAS").collect()

# Iterate through each schema
for schema in schemas:
    schema_name = schema["databaseName"]
    
    # Step 2: Get all tables in the schema
    tables = spark.sql(f"SHOW TABLES IN {schema_name}").collect()
    
    for table in tables:
        table_name = table["tableName"]
        
        # Step 3: Get the fields and data types of the table
        fields = spark.sql(f"DESCRIBE {schema_name}.{table_name}").collect()
        
        for field in fields:
            schema_info.append(
                Row(
                    schema_name=schema_name,
                    table_name=table_name,
                    field_name=field["col_name"],
                    data_type=field["data_type"],
                )
            )

# Convert the list to a DataFrame
schema_df = spark.createDataFrame(schema_info)

# Save the DataFrame
schema_df.toPandas().to_csv("/dbfs/path/to/save/databricks_schema.csv", index=False)

Export Query History

Open a notebook in Databricks and paste this code:

from databricks.sdk import WorkspaceClient
from databricks.sdk.service import sql
import time
import json

# Initialize the Databricks Workspace client
w = WorkspaceClient()

# Calculate the current time in milliseconds
end_time_ms = int(time.time() * 1000)

# Calculate the time 29 days ago in milliseconds
start_time_ms = end_time_ms - (29 * 24 * 60 * 60 * 1000)

# Retrieve the list of queries within the specified time range
queries = w.query_history.list(
    filter_by=sql.QueryFilter(
        query_start_time_range=sql.TimeRange(
            start_time_ms=start_time_ms,
            end_time_ms=end_time_ms
        )
    )
)

# Convert the list of queries to JSON format and write to a file
with open('/dbfs/path/to/save/query_history.json', 'w') as f:
    json.dump([query.to_dict() for query in queries], f, indent=4)

Upload to Solid

1. Download the exported files from DBFS
2. Archive the files (ZIP or GZIP format)
3. Upload to the Solid Azure Storage container

Permissions Needed

Troubleshooting

Connection Issues

If you're having trouble connecting:

1. Verify the Server URL is correct and includes https:// — do not include any path after the domain
2. Verify the Workspace ID is the numeric ID from your workspace URL, not the SQL Warehouse ID or any other identifier
3. Check the token is valid and hasn't expired
4. Confirm the SQL Warehouse is running and accessible
5. Verify all grants completed successfully by running the verification queries in Step 4

Permission Issues

If Solid can't access certain catalogs or tables:

1. Check service principal permissions by running SHOW GRANTS ON CATALOG <catalog_name>
2. Verify system tables are enabled for your workspace
3. Ensure the SQL Warehouse has access to the Unity Catalog metastore
4. Confirm catalog and schema grants were applied correctly

SQL Warehouse Issues

If the SQL Warehouse isn't accessible:

1. Verify the service principal has "Can use" permission on the SQL Warehouse
2. Check the SQL Warehouse is running and not in a stopped state
3. Ensure the SQL Warehouse is connected to the correct metastore

Security Notes

• Store tokens securely and never commit them to version control
• Rotate tokens regularly according to your organization's security policies
• Grant minimum necessary permissions following the principle of least privilege
• Monitor token usage through Databricks audit logs
• Use a dedicated SQL Warehouse for integration workloads to isolate resource usage

Query text showing as <REDACTED>

If query text in system.query.history appears as <REDACTED>, Databricks is applying a blanket mask to the statement_text column. This is a workspace-wide default — only members of the databricks_pii_access group can see the actual SQL.

Quick fix: Ask your Databricks account admin to add the service account (or personal account) used by Solid to the databricks_pii_access group. Once added, Solid will receive real query text on the next sync.

Alternative — create a blanket column-mask policy:

If you prefer not to add the account to databricks_pii_access, you can define an explicit masking policy that grants the Solid service account access to unmasked query text:

1. Create a masking function

CREATE FUNCTION catalog.schema.mask_pii(val STRING)
RETURNS STRING
RETURN '***';

2. Attach a blanket column-mask policy

CREATE POLICY metastore_wide_pii_mask
ON CATALOG main
COLUMN MASK catalog.schema.mask_pii
TO `account users` EXCEPT `data_admins`
MATCH COLUMNS (has_tag_value('pii', 'yes'))
AS m ON COLUMN m;

Note: Replace catalog.schema with the catalog and schema where you created the function, and adjust data_admins to match the group that should always see unmasked data. Add the Solid service account to data_admins (or an equivalent group) so it receives real query text.