Forum Discussion
Ingest parquet files in SQL Azure
Hi
We are exporting data from D365 to a Gen 2 datalake and from there I want to ingest it into a SQL azure database. I tried using the " Copy Dataverse data into Azure SQL using Synapse Link" template to acheive this, however the files in the storage account for the datalake are parquet files and this only seems to work for csv's.
What is the most efficient way to ingest the parquet files into SQL Azure? Is there an established pattern for the D365 data structure? Can I take advantage of the Common Data Model?
Cheers
Alex
Hello alexp01482
I would suggest this approach- Use Azure Synapse's serverless SQL pool to directly ( OPENROWSET) query Parquet files stored in Azure Data Lake Gen2.
- Transform the data using T-SQL and load it into Azure SQL Database using pipelines or external tables.
- This method is highly efficient for large datasets due to parallel processing and minimal data movement
When exporting data from Dynamics 365 to Azure Data Lake Gen2, it often adheres to the Common Data Model (CDM).
If this is helpful , please accept the answer and give kudos
1 Reply
Hello alexp01482
I would suggest this approach- Use Azure Synapse's serverless SQL pool to directly ( OPENROWSET) query Parquet files stored in Azure Data Lake Gen2.
- Transform the data using T-SQL and load it into Azure SQL Database using pipelines or external tables.
- This method is highly efficient for large datasets due to parallel processing and minimal data movement
When exporting data from Dynamics 365 to Azure Data Lake Gen2, it often adheres to the Common Data Model (CDM).
If this is helpful , please accept the answer and give kudos