Forum Discussion
Microsoft EmployeeCan't setup copyjob to read partitioned delta table
Solved ! Thank you for the help 🙂
Community SupportHi nicolasbejar
Thank you for reaching out to the Microsoft Fabric community forum.
The problem occurs because the Copy Activity source is set to ADLS Gen2 with the file format as Parquet, but the path actually leads to a Delta Lake table with a _delta_log directory and partitioned folders. Reading a Delta table as Parquet skips the transaction log, which prevents correct schema reading, partition detection, and version control, leading to load failures.
To resolve this, update the source to point to the Delta table’s root folder the directory that contains the _delta_log rather than an individual partition path such as year=…/month=…, and change the file format from Parquet to Delta (Delta Lake). When configured correctly, the Delta reader handles partition pruning automatically without the need for wildcards or recursive searches.
Also, check that the pipeline’s Managed Identity (either Fabric workspace MI or ADF MI) has the required permissions, such as Storage Blob Data Contributor RBAC and suitable read/traverse ACLs on the container and table root. If the storage account uses a Private Endpoint or VNet, make sure the pipeline runtime can access the DFS endpoint. After these changes, both Preview and full Copy should work with the partitioned Delta table.
I hope this information is helpful. If you have any further questions, please let us know. we can assist you further.
Regards,
Microsoft Fabric Community Support Team.
nicolasbejarHi !
Thank you for engaging
I'm still unable to read it .
1. My original path was already pointing to the folder that contains both the _delta_log and also the partitioned subfolders containning the parquet data.
2. Should the connection change from ADLS gen2 to something else ?
3. I can't see a format that says Delta.- v-karpurapud
Hi nicolasbejar
I want to assure you that there is nothing wrong with the way you originally configured your source. Pointing to the folder that contains _delta_log is exactly what you’re supposed to do for a Delta table, and based on what you described, you were already following the correct Delta Lake conventions. My earlier explanation focused on how Delta tables are normally read from the table root and through the Delta transaction log which is why I highlighted those details.
The key insight, as the Ugk161610 pointed out, is that the limitation isn’t with your setup at all, but with the current capabilities of the Copy Job UI. Even if the path and structure are correct, the Copy Job simply doesn’t recognize Delta Lake tables in ADLS right now, which is why the “Delta” format isn’t appearing for you and why the load continues to fail. Their clarification complements my explanation by making it clear that the issue lies with the tool rather than anything you misconfigured.
So while your approach is right, the Copy Job currently can’t process ADLS-backed Delta tables. Once Delta support is added to the Copy Job, your configuration should work exactly as expected.
I would also like to thank you Ugk161610 for your active participation and for sharing solutions within the community forum.Regards,
Microsoft Fabric Community Support Team.