Work / Solution architecture

Sharing data across clouds, from AWS Databricks to Azure Databricks

In a pre-sales engagement, I proposed a framework to harden data sharing between an AWS Databricks producer and Azure Databricks consumers: a private network path, OAuth in place of bearer tokens, and monitoring of who reads what, and how much.

Client
An enterprise automotive data platform team
Company
Celebal Technologies
Role
Solution design, pre-sales
Focus
Solution architecture
~20 TBshared between clouds every month, and growing
~50%lower estimated monthly cost with the recommended option
4problems, each mapped to a layer and a mechanism

The situation

The producer was an AWS Databricks workspace with Unity Catalog, holding the enterprise raw layer. The consumers were several Azure Databricks workspaces, mostly with no public IP, feeding downstream SAP, CRM and Power BI. Data moved through Delta Sharing with bearer tokens over the public internet, at about 20 TB a month and growing. The ask: harden it before scaling further.

Four distinct problems, not one

  • Traffic over the public internet: encrypted, but not private, and short of the security an enterprise expects for data moving between clouds.
  • Weak authentication: bearer tokens that never expired, sat in config files, could be copied, could not be traced to a person, and could not be revoked for a single recipient.
  • Cost scaling badly: egress and NAT charges growing in line with volume.
  • No observability: no way to answer who reads, how often, how much, full load or incremental, or whether retries were piling up.

The framework: one decision per layer

  • Network transport: two options. A Site-to-Site VPN that stays fully native to AWS and Azure, or ExpressRoute plus Direct Connect through an interconnect partner for a fully private path at the lowest cost at scale. I recommended the private path, with the VPN as the alternative if no third party was allowed.
  • Data pattern: Delta Sharing for bulk batch reads, and Lakehouse Federation for interactive analytics, so only filtered result sets cross clouds and data files stay in AWS.
  • Authentication: OAuth tokens issued to Service Principals by the Unity Catalog OIDC endpoint. They expire within the hour, rotate automatically, can be revoked for each recipient, with real principals in the audit log.
  • Observability: Databricks system tables plus a custom egress ledger, surfaced in a Power BI dashboard that also flags broken incremental loads.

Traceability and rollout

Every stated issue was mapped to a layer, a mechanism, and how that mechanism solves it, so the client could check the proposal against their own requirements line by line. Both network options were costed against the current baseline and across volume growth; savings widen as volume grows because the fixed circuit cost is spread over more data. Delivery was planned as a short discovery phase and three phases of layered, additive changes over roughly 14 to 16 weeks, with no workspace rebuilds.

AWS producer to Azure consumersAWSDatabricks, UnityCatalog, raw layerPRODUCERAzure workspaceAzure workspaceAzure workspacePRIVATEPATHOAuth, SPDelta Sharing for batch, Lakehouse Federation for queriesAudit, billing and egress ledger in Power BI
Next case studyExpenseOps: expense claims, automated from bill to submission