topic Re: Does Fabric duplicate data when creating a table from a Parquet file? in Data Engineering

Does Fabric duplicate data when creating a table from a Parquet file?

sawilliams — Wed, 05 Feb 2025 12:40:09 GMT

Hi Community,

I'm working on a POC for a simple data ingestion pipeline in Microsoft Fabric. My process is as follows:

Read a CSV from a shortcut to ADLS2
Convert it to Delta Parquet
Save it in the FILES path of the Lakehouse
Create a table from the stored Parquet in the Fabric portal

However, after creating the table, I noticed that the Parquet files also appear in the TABLES path, even though I initially saved them in FILES. Is Fabric duplicating the Parquet files when creating the table?

Would it be better to save the Parquet files directly in the TABLES path instead of FILES to avoid redundancy?

Re: Does Fabric duplicate data when creating a table from a Parquet file?

Anonymous — Thu, 06 Feb 2025 07:12:28 GMT

Hi sawilliams
Thank you for reaching out microsoft fabric community forum.

Yes, Microsoft Fabric does duplicate the data when creating a table from a Parquet file. This happens because the data is initially stored in the FILES path and then copied to the TABLES path when creating the table.

To avoid redundancy, it would indeed be better to save the Parquet files directly in the TABLES path from the start. This way, you won't have duplicate copies of your data

If you need any further assistance or have any questions, please feel free to reach us.

If this solution helps, please consider giving us Kudos and accepting it as the solution so that it may assist other members in the community.

Thank you.