Forum Discussion
Does Fabric duplicate data when creating a table from a Parquet file?
I'm working on a POC for a simple data ingestion pipeline in Microsoft Fabric. My process is as follows:
- Read a CSV from a shortcut to ADLS2
- Convert it to Delta Parquet
- Save it in the FILES path of the Lakehouse
- Create a table from the stored Parquet in the Fabric portal
However, after creating the table, I noticed that the Parquet files also appear in the TABLES path, even though I initially saved them in FILES. Is Fabric duplicating the Parquet files when creating the table?
Would it be better to save the Parquet files directly in the TABLES path instead of FILES to avoid redundancy?
Hi sawilliams
Thank you for reaching out microsoft fabric community forum.Yes, Microsoft Fabric does duplicate the data when creating a table from a Parquet file. This happens because the data is initially stored in the FILES path and then copied to the TABLES path when creating the table.
To avoid redundancy, it would indeed be better to save the Parquet files directly in the TABLES path from the start. This way, you won't have duplicate copies of your data
If you need any further assistance or have any questions, please feel free to reach us.
If this solution helps, please consider giving us Kudos and accepting it as the solution so that it may assist other members in the community.
Thank you.
1 Reply
Hi sawilliams
Thank you for reaching out microsoft fabric community forum.Yes, Microsoft Fabric does duplicate the data when creating a table from a Parquet file. This happens because the data is initially stored in the FILES path and then copied to the TABLES path when creating the table.
To avoid redundancy, it would indeed be better to save the Parquet files directly in the TABLES path from the start. This way, you won't have duplicate copies of your data
If you need any further assistance or have any questions, please feel free to reach us.
If this solution helps, please consider giving us Kudos and accepting it as the solution so that it may assist other members in the community.
Thank you.