Sharing data across clouds, from AWS Databricks to Azure Databricks
In a pre-sales engagement, I proposed a framework to harden data sharing between an AWS Databricks producer and Azure Databricks consumers: a private network path, OAuth in place of bearer tokens, and monitoring of who reads what, and how much.
- Client
- An enterprise automotive data platform team
- Company
- Celebal Technologies
- Role
- Solution design, pre-sales
- Focus
- Solution architecture
The situation
The producer was an AWS Databricks workspace with Unity Catalog, holding the enterprise raw layer. The consumers were several Azure Databricks workspaces, mostly with no public IP, feeding downstream SAP, CRM and Power BI. Data moved through Delta Sharing with bearer tokens over the public internet, at about 20 TB a month and growing. The ask: harden it before scaling further.
Four distinct problems, not one
- Traffic over the public internet: encrypted, but not private, and short of the security an enterprise expects for data moving between clouds.
- Weak authentication: bearer tokens that never expired, sat in config files, could be copied, could not be traced to a person, and could not be revoked for a single recipient.
- Cost scaling badly: egress and NAT charges growing in line with volume.
- No observability: no way to answer who reads, how often, how much, full load or incremental, or whether retries were piling up.
The framework: one decision per layer
- Network transport: two options. A Site-to-Site VPN that stays fully native to AWS and Azure, or ExpressRoute plus Direct Connect through an interconnect partner for a fully private path at the lowest cost at scale. I recommended the private path, with the VPN as the alternative if no third party was allowed.
- Data pattern: Delta Sharing for bulk batch reads, and Lakehouse Federation for interactive analytics, so only filtered result sets cross clouds and data files stay in AWS.
- Authentication: OAuth tokens issued to Service Principals by the Unity Catalog OIDC endpoint. They expire within the hour, rotate automatically, can be revoked for each recipient, with real principals in the audit log.
- Observability: Databricks system tables plus a custom egress ledger, surfaced in a Power BI dashboard that also flags broken incremental loads.
Traceability and rollout
Every stated issue was mapped to a layer, a mechanism, and how that mechanism solves it, so the client could check the proposal against their own requirements line by line. Both network options were costed against the current baseline and across volume growth; savings widen as volume grows because the fixed circuit cost is spread over more data. Delivery was planned as a short discovery phase and three phases of layered, additive changes over roughly 14 to 16 weeks, with no workspace rebuilds.