Ingest parquet files in SQL Azure

[alexp01482]

Hi

We are exporting data from D365 to a Gen 2 datalake and from there I want to ingest it into a SQL azure database. I tried using the " Copy Dataverse data into Azure SQL using Synapse Link"  template to acheive this, however the files in the storage account for the datalake are parquet files and this only seems to work for csv's.

What is the most efficient way to ingest the parquet files into SQL Azure? Is there an established pattern for the D365 data structure? Can I take advantage of the Common Data Model?

Cheers

Alex

[nilendraFabric · Accepted solution]

Hello @alexp01482 

I would suggest this approach

• Use Azure Synapse's serverless SQL pool to directly ( OPENROWSET) query Parquet files stored in Azure Data Lake Gen2.
• Transform the data using T-SQL and load it into Azure SQL Database using pipelines or external tables.
• This method is highly efficient for large datasets due to parallel processing and minimal data movement

When exporting data from Dynamics 365 to Azure Data Lake Gen2, it often adheres to the Common Data Model (CDM).

If this is helpful , please accept the answer and give kudos