Forum Discussion
Copy data from lakehouse to ADLSg2
I'm trying to COPY a delta table hosted in my lakheouse into ADLSg2 container in parquet format. I'm trying with some tables, some of them I'm able to copy, but with one of them I get this error:
Failure happened on 'Source' side. ErrorCode=PartitionDiscoveryWithInconsistentFolderPath,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Source file pathes are not in the same pattern when processing partition discovery. Please check the folder pathes.,Source=Microsoft.DataTransfer.ClientLibrary,'
I can't get any info about this kind of error in the documentation.
Any ideas of what is happening?
7 Replies
Hi amaaiia
Thank for using Microsoft Fabric Community.
The error message "PartitionDiscoveryWithInconsistentFolderPath" indicates an issue with how Fabric is discovering partitions in your Delta table during the copy process to ADLSg2.
Partition discovery: Fabric uses partition folders to identify and organize data within your Delta table.
Inconsistent folder paths: The error suggests that the folder structure used for partitioning in your Delta table isn't consistent across all partitions. This inconsistency throws off Fabric's ability to properly discover the data.
Review the folder structure of the delta table in the Microsoft Fabric Lakehouse. Ensure that the partitioned data is organized consistently within the folders. Manually inspect the folder structure within the table location and look for any inconsistencies in how partition folders are named or organized.
Ensure that the partitioning scheme used in the delta table is correctly configured and consistent across all partitions. Any variation in the partitioning scheme can lead to inconsistencies in folder paths.
I hope this information helps.
Could you please provide the repro steps that you have followed here. In order to understand your query and guide you in better way.
Thank you.
Hi amaaiia
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.Thanks.
I've checked the folders bellow my table in View files, and I have the same partitions: YEAR > MONTH > DAY. Bellow DAY folder there are just parquet files. So I guess is not partition schema issue...
What I've been doing is to run a COPY activity that reads from SOURCE a table in lakehouse and writes it into ADLSg2 container as DESTINATION in parquet format. Most of the tables performs well, but one of them doesn't, showing me that error.
Hi amaaiia
Apologies for the inconvenience.
Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.
Please go ahead and raise a support ticket to reach our support team:
https://support.fabric.microsoft.com/support
After creating a Support ticket please provide the ticket number as it would help us to track for more information.
Thank you.
Hello @amaaiia
This is a nice use case and I would like to learn from you how are you able to copy data from Fabric Lakehouse to ADLS Gen 2.
I would appreciate if you may please share the link or any documentation on how to perform this operation.
Thanks,
P
Hello Pawansingh
There are a few ways to do that.
1. You can use a Data Pipeline component, with a COPY activity. You can select a lakehouse table as Source and a ADLSg2 container table as Destination.
2. You can also create a shortcut in your lakehouse to a table in yout ADLSg2 container, and you can write data through that shortcut.
