topic Re: Inquiry about file size difference between Widow and Fabric in Pipelines

Inquiry about file size difference between Widow and Fabric

leesukjun — Thu, 16 Jan 2025 07:14:12 GMT

The file size shown in Widow and the file size of the data upload are different.

If I look at the file size of test2.csv on Windows it's 1.81GB, but if I try to upload it from Fabric it's 2.28GB.

Why is that?

Re: Inquiry about file size difference between Widow and Fabric

nilendraFabric — Thu, 16 Jan 2025 13:10:17 GMT

The difference in size (1.81 GB on Windows vs. 2.28 GB in Fabric) is primarily due to how Microsoft Fabric processes and stores files after upload. It involves transformations such as adding metadata, changing storage formats, or applying encoding adjustments that make the uploaded version larger than its original representation on Windows.

Re: Inquiry about file size difference between Widow and Fabric

leesukjun — Fri, 17 Jan 2025 00:30:14 GMT

Thank you for your reply.

Do you have any more detailed information?

If I don't know the information about how to save the file, I can't convince my client.
Should I contact MS, not the community, for this?
Do you have any detailed information about this?

Thank you.

Re: Inquiry about file size difference between Widow and Fabric

Anonymous — Fri, 17 Jan 2025 01:46:26 GMT

Hi leesukjun

nilendraFabric provides a very meaningful explanation.

Windows usually uses NTFS, which has its own way of storing and compressing files. Fabric mainly relies on OneLake, a unified data lake storage system. OneLake uses the open Delta Lake format to store data. This format allows efficient data storage and processing, and supports a variety of data analysis and machine learning workloads.

Therefore, these are two different ways, which may cause changes in file size when uploading.

When files are uploaded to Fabric, additional encoding and metadata may be added to ensure compatibility and integrity within the Fabric environment. This may include information about the file structure, permissions, and other properties.

Fabric may transform the data to optimize its processing needs. This transformation may include changes in the data format, structure, or encoding, which can increase the file size.

This information is not visible in the Windows file size.

Knowing these details can help you explain the difference to your clients and that the increased file size is a normal part of the upload process to Fabric.

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Inquiry about file size difference between Widow and Fabric

nilendraFabric — Tue, 21 Jan 2025 09:42:40 GMT

Hi leesukjun

could you please elaborate what exactly you are looking for ?

thanks