Data System Collector

The Data System Collector is an optional deployment component for organizations that cannot open inbound connections from Solid to their data warehouse or BI tools. Instead of Solid connecting into your network, you run a small worker service inside your own Kubernetes cluster. The collector makes only outbound connections — it polls Solid for work, runs the request locally against your database, and sends results back. No inbound ports are opened and no long-lived credentials are stored in your cluster.

When to use the collector

The standard SaaS deployment connects to your data warehouse outbound from Solid's network. If your security posture allows allowlisting Solid's static IPs, that's the simpler path — see Solid's Static IP Addresses.

Use the collector instead when:

• Your data warehouse sits behind a firewall that cannot accept any inbound connections from outside your network, even from an allowlisted IP
• Internal policy prohibits opening warehouse ports to any external host
• You need all database traffic to originate inside your own network perimeter

How it works

The collector authenticates to Solid's services using Azure Workload Identity Federation — short-lived tokens issued by your cluster's OIDC provider, trusted by an identity Solid creates in Solid's Azure tenant. No password or API key is ever stored in your cluster.

The end-to-end flow:

1. You provide Solid with your cluster's OIDC issuer URL and your cluster's egress IP
2. Solid registers a federated identity credential on its side, scoped to only your resources
3. Solid sends you the collector's container image, the Helm chart, and configuration values (client ID, customer ID, environment label)
4. You install the chart with a single helm install; the pod authenticates via federated workload identity and the collector begins operating

Prerequisites

• An AKS cluster with OIDC issuer and workload identity enabled — these are required for federated authentication and must be set at cluster creation time
• Helm and kubectl installed and configured for the cluster
• Docker (to load the image archive Solid supplies)
• Your cluster's static egress (outbound) IP

Step 1 — Get a cluster ready

If you don't have a cluster yet

Solid provides a self-contained Terraform stack (terraform.zip) that creates a private AKS cluster with workload identity preconfigured. Unzip it, fill in one configuration file, and run the included script:

./run-terraform.sh

The script signs you into Azure, confirms the target subscription, walks through terraform plan, and applies with a confirmation prompt — nothing is created without your review. It prints the OIDC issuer URL at the end.

Before running, check that your chosen VM size has quota in your region:

az vm list-usage --location "<your-region>" -o table
# New subscriptions often start with 0 quota on most VM families.
# If apply fails with ErrCode_InsufficientVCPUQuota, pick a different size
# or request a quota increase via Azure Portal → Quotas → Compute.

You can optionally attach the cluster to an existing VNet (create_vnet = false in terraform.tfvars) or create a container registry alongside the cluster (create_acr = true).

If you already have a cluster

Your cluster needs OIDC issuer and workload identity enabled. If they're not already on:

az aks update -g <resource-group> -n <cluster> \
  --enable-oidc-issuer \
  --enable-workload-identity

These must be enabled before the collector is deployed. They cannot be added after the collector depends on them without recreating the cluster.

Step 2 — Register with Solid

Send Solid two pieces of information — neither is a secret:

1. Your cluster's OIDC issuer URL

If you used Solid's Terraform stack:

terraform output aks_oidc_issuer_url

On an existing cluster:

az aks show \
  -g <resource-group> -n <cluster> \
  --query "oidcIssuerProfile.issuerUrl" -o tsv

2. Your cluster's static outbound (egress) IP

Solid's endpoints are not publicly open — your egress IP must be allowlisted before the collector can reach them.

az network public-ip show \
  --ids $(az aks show -g <resource-group> -n <cluster> \
    --query "networkProfile.loadBalancerProfile.effectiveOutboundIPs[].id" -o tsv) \
  --query ipAddress -o tsv

Solid uses the OIDC issuer URL to create the federated identity credential on its side. Once that's done, Solid sends you back:

• The collector container image archive (.tar file)
• The Helm chart
• A client ID, your customer ID, and an environment label for the values file

Step 3 — Load the container image

The image arrives as a .tar archive. Push it to a registry your cluster can pull from:

docker load -i data-system-collector-<version>.tar
docker tag <loaded-image>:<version> <your-registry>/<repository>:<version>
docker push <your-registry>/<repository>:<version>

Alternatively, if you have no registry, preload the archive onto every node directly into the container runtime (ctr -n k8s.io images import …). Note that new or replaced nodes won't have the image unless the preload is part of your node provisioning process.

Step 4 — Configure and install

Edit values-customer.yaml with the values Solid provided:

The collector is installed into the soliddata namespace with release name data-system-collector. Keep both as-is — the identity Solid created is bound to these exact names.

Before installing, render the chart locally to catch any configuration mistakes:

helm template data-system-collector . \
  -f values.yaml -f values-customer.yaml | less
# Confirm: exactly one Deployment, one ServiceAccount, one ConfigMap, and nothing else.

Install:

helm upgrade -i data-system-collector . \
  -f values.yaml -f values-customer.yaml \
  -n soliddata --create-namespace \
  --wait --timeout 5m

Step 5 — Verify

kubectl -n soliddata get pods -l app.kubernetes.io/name=data-system-collector

The pod should be Running. Once it is, the collector begins polling Solid for work and your data systems become accessible through it — no further configuration is needed on your side.

Uninstall

helm uninstall data-system-collector -n soliddata

Security properties