What is the best extraction option for AWS ?

[JosueMolina]

Our operational source of information is spread among multiple RDS for SQL Server. We are looking to include a middle layer with AWS to consolidate and do some initial transformations. We have yet to decide, but this middle layer is probably going to be a simple RDS or Redshift. 

Currently, we extract multiple sources into S3 as parquet and then just start working with that in Fabric through the ABFS shortcut.

We would like to avoid using the S3 to minimize the steps in our pipeline. Would Fabric Data Pipeline be sufficient? 

Ideally, we would want a CDC into a Bronze Lakehouse. I know this is not supported, so what is the next best thing?

We are looking at Azure Data Pipelines as an option, but we fear the we are just swapping S3 for a ADLS2 without any real improvement but the added cost of the Azure service.

[JosueMolina]

@v-yilong-msft Thanks for the thorough explanation! I think this can help us choose a direction. Do you know if there is a resource where I can reach out to some Product Manager or Architect within Microsoft that can give a bit more of personalized help with this? Maybe a support channel or something like that.

[v-yilong-msft]

Hi  @JosueMolina ,

I am very glad that you are receptive to my comments, and I don't think I can directly connect you with the appropriate people at this time for your request.

You may wish to inquire through other channels. If it helps, can you mark the correct answer as a standard answer to help other members find it faster? Thank you very much for your kind cooperation!

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[v-yilong-msft]

Hi @JosueMolina ,

Given your current use of Microsoft Fabric, it may be worth exploring its data pipeline capabilities further. It can help you reduce complexity by managing your ETL processes within the same ecosystem.
For near real-time data ingestion, consider implementing incremental loading and watermarking in your data pipeline.
If more advanced features and better integration with other Azure services are required, Azure Data Pipelines may be a viable option, albeit at an additional cost.

1. Microsoft Fabric Data Pipeline can indeed help you manage ETL activities efficiently. It allows you to create data pipelines that can ingest, transform, and load data into your data warehouse or Lakehouse. However, there are some limitations, such as the lack of support for managed VNets and certain scheduling options. You can look at this document: Data Factory limitations overview - Microsoft Fabric | Microsoft Learn

2. While Fabric Data Pipeline doesn’t natively support CDC into a Bronze Lakehouse, you can still achieve near-real-time data ingestion using incremental loads and watermarking techniques. This approach can help you keep your data up-to-date without the need for full refreshes.

3. Azure Data Pipelines offer robust integration with various data sources, including ADLS2. While it might seem like you’re just swapping S3 for ADLS2, Azure Data Pipelines provide additional features like better integration with other Azure services, advanced monitoring, and CI/CD capabilities. These features can help you streamline your data workflows and potentially reduce operational overhead. Please read this document: Exploring CI/CD Capabilities in Microsoft Fabric: A Focus on Data Pipelines | Microsoft Fabric Blog ...

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.