Forum Discussion
Is it possible to insert data into fabric datawarehouse using a script that isnt in fabric notebook?
I have a small pipeline that gets data from mongo using lambda functions and glue and save these files in S3. Then i get the parquet files in S3 using fabric connectors. But I think It would be better to simply use the script to insert data into my fabric DW. Is there a way to do it from AWS?
Hi eduardo_schmitz ,
Thank you for reaching out in Microsoft Fabric Community.
Based on your scenario, there are multiple ways to ingest data from AWS into Microsoft Fabric Data Warehouse:1. Direct Data Ingestion from AWS
You can establish a connection from AWS services (such as AWS Lambda, AWS Glue) to Fabric Data Warehouse using ODBC/JDBC. This allows data to be inserted directly into Fabric DW using INSERT INTO or MERGE statements. However, setting up and maintaining the connection falls under AWS configurations.2. Using Fabric Data Pipelines.
Microsoft Fabric offers Data Pipelines to ingest data from AWS S3 using Fabric's S3 connector.
This method enables a no-code/low-code approach to move data into Fabric Data Warehouse efficiently.
Set up your Amazon S3 connection - Microsoft Fabric | Microsoft LearnSince setting up a direct connection from AWS to Fabric DW primarily involves AWS-side configurations, I recommend posting this in the AWS Community Forum for more insights from AWS experts:
AWS re:Post Community → https://repost.aws/Hope this helps. Please reach out for further assistance.
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and a kudos would be appreciated.
Best Regards,
Vinay.Hi eduardo_schmitz ,
Thanks for the update,
To help you get started with ODBC/JDBC connections to Fabric Data Warehouse, here are some useful resources:1. Connectivity to data warehousing - Microsoft Fabric | Microsoft Learn
2. Set up your Odbc connection - Microsoft Fabric | Microsoft Learn
3. Microsoft ODBC Driver for SQL Server - ODBC Driver for SQL Server | Microsoft Learn
4. Microsoft JDBC Driver for SQL Server - JDBC Driver for SQL Server | Microsoft Learn
If you're considering Fabric's REST API as an alternative, you can explore:
Microsoft Fabric REST API references - Microsoft Fabric REST APIs | Microsoft LearnHope these help. Please reach out for further assistance.
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and a kudos would be appreciated.
Thank you.
9 Replies
v-veshwara-msftCommunity Support
Hi eduardo_schmitz ,
Thank you for reaching out in Microsoft Fabric Community.
Based on your scenario, there are multiple ways to ingest data from AWS into Microsoft Fabric Data Warehouse:1. Direct Data Ingestion from AWS
You can establish a connection from AWS services (such as AWS Lambda, AWS Glue) to Fabric Data Warehouse using ODBC/JDBC. This allows data to be inserted directly into Fabric DW using INSERT INTO or MERGE statements. However, setting up and maintaining the connection falls under AWS configurations.2. Using Fabric Data Pipelines.
Microsoft Fabric offers Data Pipelines to ingest data from AWS S3 using Fabric's S3 connector.
This method enables a no-code/low-code approach to move data into Fabric Data Warehouse efficiently.
Set up your Amazon S3 connection - Microsoft Fabric | Microsoft LearnSince setting up a direct connection from AWS to Fabric DW primarily involves AWS-side configurations, I recommend posting this in the AWS Community Forum for more insights from AWS experts:
AWS re:Post Community → https://repost.aws/Hope this helps. Please reach out for further assistance.
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and a kudos would be appreciated.
Best Regards,
Vinay.Yes, it's possible to insert data into Microsoft Fabric Data Warehouse without using Fabric Notebooks. Since your pipeline already retrieves data from MongoDB using AWS Lambda and AWS Glue, you have a few options to push data directly into Fabric DW:
1. You can set up an ODBC/JDBC connection from AWS to Fabric DW and execute INSERT INTO or MERGE statements. This method requires configuring a secure connection on the AWS side.
2. If you’re using Azure services in combination with AWS, Synapse Link can streamline data movement from AWS storage to Fabric DW.
Another option is to use Microsoft Fabric’s REST APIs to push data into your DW without relying on Fabric notebooks.
If you prefer a script-based approach directly from AWS, using ODBC/JDBC or APIs would be the best choice. Have you considered trying a direct connection from Glue or Lambda via ODBC?
yes, a script-based approach would be ideal. My data requires little transformation, so I would much rather push it directly into the DW Fabric using Lambda or Glue.
However i'm not familiar with using ODBC/JDBC or fabric API just yet. Thank you very much for the suggestion. I will study this approach.
if you have an article about this, i'd appreciate, otherwise i'm alreayding accepting this as solution.- v-veshwara-msft
Hi eduardo_schmitz ,
Thanks for the update,
To help you get started with ODBC/JDBC connections to Fabric Data Warehouse, here are some useful resources:1. Connectivity to data warehousing - Microsoft Fabric | Microsoft Learn
2. Set up your Odbc connection - Microsoft Fabric | Microsoft Learn
3. Microsoft ODBC Driver for SQL Server - ODBC Driver for SQL Server | Microsoft Learn
4. Microsoft JDBC Driver for SQL Server - JDBC Driver for SQL Server | Microsoft Learn
If you're considering Fabric's REST API as an alternative, you can explore:
Microsoft Fabric REST API references - Microsoft Fabric REST APIs | Microsoft LearnHope these help. Please reach out for further assistance.
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and a kudos would be appreciated.
Thank you.
- v-veshwara-msft
Hi eduardo_schmitz ,
We’re following up once more regarding your query. If it has been resolved, please mark the helpful reply as the Accepted Solution to assist others facing similar challenges.
If you still need assistance, please let us know.
Thank you.
