Forum Discussion
Skilled SharerConvert partitioned Parquet table into Delta table dinamically
Hi.
I have a list of Parquet tables under Files directory in my lakehouse. Each table is partitioned by different columns, some of them are partitioned by 1 column, others by 2, others by 3... and the name of each column is different depending on the table.
I want to create a Data Pipeline so I pass a table_name parameter and it COPIES the table_name table under Files directory into Tables directory as a Delta table, keeping the same partitions as in Files.
I need this proccess to be dynamic, that is, I'm not telling the pipeline which are the partitions for the table, I only pass the table_name and de pipeline should be able to COPY the table with SAME partitions but as delta table.
If possible, I want to do it with COPY activity instead of using notebook.
Any ideas?
Hi amaaiia
Here are some suggestions I have for the questions you asked:
Create a pipeline parameter called table_name.
Use Get Metadata activity to retrieve the partition information for the specified table. This activity will help you determine the partition column dynamically.
Use ForEach activity to iterate over the partition columns retrieved from the Get Metadata activity.
In the ForEach activity, use Copy activity to copy the data from the Parquet table in the Files directory to the Delta table in the Tables directory. Make sure to set the partition column dynamically based on the metadata.
Regards,
Nono Chen
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
1 Reply
Hi amaaiia
Here are some suggestions I have for the questions you asked:
Create a pipeline parameter called table_name.
Use Get Metadata activity to retrieve the partition information for the specified table. This activity will help you determine the partition column dynamically.
Use ForEach activity to iterate over the partition columns retrieved from the Get Metadata activity.
In the ForEach activity, use Copy activity to copy the data from the Parquet table in the Files directory to the Delta table in the Tables directory. Make sure to set the partition column dynamically based on the metadata.
Regards,
Nono Chen
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.