Forum Discussion
Fabric Data Ingestion from AWS
Good Day
I want to ingest data into Fabric where the source is AWS RDS
What would be the best data connectivity option here? Since the AWS databases would not be publically
shared I assume VNet Data Gateway could be an option??
Appreciate any help
Regards,
Zach
Hi Zach77 , If your AWS RDS databases aren’t publicly exposed, the best option is to ingest via Fabric Data Factory (Pipelines for orchestration) and a mix of several activities including Copy job activity using the native database connectors, combined with either Dataflow Gen2 or a Spark/Notebook job if you need transforms.
And yes, VNet Data Gateway could be an option BUT IT ONLY CONNECTS TO AZURE network, so you need to use Azure VPN Gateway as a "bridge" between Fabric and AWS.
In this article, Designing private network connectivity between AWS and Microsoft Azure | Microsoft Workloads on AWS explains how to connect to stablilsh a connection to Azure's Azure VPN Gateway, and from Fabric, you will need to use VNet Data Gateway, which operates at the data access layer, not the network layer: it allows Fabric workloads to use an existing Azure VNet’s connectivity to reach private data sources
Hope this works for your scenario 😅 there will be a LOT TO CONFIGURE on Azure (Route tables, Incoming/outgoing traffic, IP listing, etc. etc.) but is possible! I'd suggest get support from some people with expertise on networking 😊 for additional details.
But hope the overall idea makes sense, kudos if you find this information useful and mark as solution if appropiate... best of lucks mate!
5 Replies
- svenchio
Super User
Hi Zach77 , If your AWS RDS databases aren’t publicly exposed, the best option is to ingest via Fabric Data Factory (Pipelines for orchestration) and a mix of several activities including Copy job activity using the native database connectors, combined with either Dataflow Gen2 or a Spark/Notebook job if you need transforms.
And yes, VNet Data Gateway could be an option BUT IT ONLY CONNECTS TO AZURE network, so you need to use Azure VPN Gateway as a "bridge" between Fabric and AWS.
In this article, Designing private network connectivity between AWS and Microsoft Azure | Microsoft Workloads on AWS explains how to connect to stablilsh a connection to Azure's Azure VPN Gateway, and from Fabric, you will need to use VNet Data Gateway, which operates at the data access layer, not the network layer: it allows Fabric workloads to use an existing Azure VNet’s connectivity to reach private data sources
Hope this works for your scenario 😅 there will be a LOT TO CONFIGURE on Azure (Route tables, Incoming/outgoing traffic, IP listing, etc. etc.) but is possible! I'd suggest get support from some people with expertise on networking 😊 for additional details.
But hope the overall idea makes sense, kudos if you find this information useful and mark as solution if appropiate... best of lucks mate!
