Onelake File Explorer Excessive Disk Space

[zzthatcher]

Hello,

I am having an issue where my whole team is seeing excessive disk space usage by onelake file explorer on their computers. This is without syncing anything locally. 

I have a feeling the space is entirely metadata. I have a few processes that ran deletes and updates on lakehouses without frequently running optimizes or vacuums and that this likely lead to the initial issue (I started seeing large lakehouse BCDR costs which is how I knew). But it has never gone away even though I now run daily optimizes and vacuums.

It is to the point where it is slowing down our PCs, sometimes significantly. 

We try to "free up space" or even uninstall the app, but it seems like there is so much metadata that it is incapable of completing either of these actions. 

At this point, it seems like this software can effectively brick a computer if it continues to go unchecked, which is a pretty big bug if you ask me. 

Any ideas?

[zzthatcher]

I want to reiterate that this program is still bricking my team's computers. Every now and again someone new joins and downloads it, or someone starts using fabric for the first time and downloads it. And they go hours at critical times without being able to use their computer. I have suggested to the product manager at fabcon to give us a default option to not sync tables in hopes that would solve the issue, but I haven't seen any updates about that. 

In my opinion this is the biggest issue with microsoft fabric and it seems to not be getting any attention. Which is outstandlingly odd to me since so many of my team members consistently run into this problem. 

[v-kpoloju-msft]

Hi @zzthatcher,

Thanks for reaching out to the Microsoft fabric community forum.

I would also take a moment to personally thanks @nilendraFabric, for actively participating in the community forum and his inputs.

After thoroughly reviewing the details you provided, here are few alternative workarounds that might help resolve the issue. Please follow the steps below:

• Navigate to %USERPROFILE%\AppData\Local\Microsoft\OneLake and delete the cache files to free up space used by temporary files and metadata. Adjust the sync settings in OneLake File Explorer to limit the amount of metadata being synced, reducing overall disk space usage.
• Use built-in disk cleanup tools, to identify and remove unnecessary files and metadata. Limiting access to large datasets or lake houses that are not frequently used can also help reduce metadata sync to local machines.
• Schedule regular maintenance tasks, including running OPTIMIZE and VACUUM commands with appropriate retention periods, to clean up and optimize the data lake. Implement monitoring tools to track disk usage and identify any unusual spikes in metadata size, allowing proactive issue resolution.

Please refer to the following link for more information:
Access Fabric data locally with OneLake file explorer - Microsoft Fabric | Microsoft Learn

If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.

Best Regards.

[v-kpoloju-msft]

Hi @zzthatcher,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[v-kpoloju-msft]

Hi @zzthatcher,

We have not yet received a response regarding our previous reply and would like to check if your issue has been resolved. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.

[v-kpoloju-msft]

Hi @zzthatcher,

I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.

Thank you.

[nilendraFabric · Accepted solution]

Hello @zzthatcher 

this is known behaviour of onelake explorer. 

OneLake File Explorer automatically syncs metadata for all files and folders that a user has access to in OneLake. This includes creating placeholders for files instead of downloading the actual data. While this approach minimizes bandwidth usage and prevents unnecessary file downloads, the cumulative size of metadata can become significant, especially when dealing with large datasets or numerous files and folders

https://learn.microsoft.com/en-us/fabric/onelake/onelake-file-explorer

Placeholder are lightweight representations of files that allow users to see the structure of their data without downloading it. However, frequent updates to metadata or placeholders (e.g., due to changes in the data lake structure or user access) can increase disk space usage over time.
3. Impact of Large Datasets: For organizations managing extensive datasets or lakehouses, the volume of metadata synced locally can grow substantially. This issue may be exacerbated if proper cleanup processes are not implemented for outdated or redundant data, leading to unnecessary accumulation of metadata.
4. No Automatic Data Download: It is important to note that OneLake File Explorer does not download actual file content unless explicitly requested by the user (e.g., by opening a file). This design ensures that disk space consumption is primarily due to metadata rather than full file downloads

Residual Metadata from Lakehouse Operations: If `OPTIMIZE` and `VACUUM` commands were not run consistently in the past, residual metadata from deleted or updated files might still exist in the lakehouse storage. Even after implementing daily cleanup routines, older metadata may persist unless explicitly purged with lower retention periods

Some workarounds 

Navigate to `%USERPROFILE%\OneLake - Microsoft` and manually delete unnecessary placeholders or temporary files if safe to do so.

Optimize Lakehouse Data
• Run `VACUUM` commands with a reduced retention period (e.g., 0 hours) to permanently remove residual metadata from deleted files.
• Continue running `OPTIMIZE` commands regularly to compact Delta tables and reduce metadata overhead

if this is helpful please give kudos and accept the solution 

[zzthatcher]

Won't manually deleting files from `%USERPROFILE%\OneLake - Microsoft` delete them in the lakehouse as well? I do not want to delete files that are in the lakehouse. 

I have gotten my computer to the point where these placeholders are only taking up 6 GB, which is relatively insignificant compared to the actual size of the data being stored. Even still, this seems like a lot for what is essentially metadata and I can see how this could still slow down a computer. 

How can I make onelake sync the metadata less frequently or even just on demand?

[nilendraFabric]

Manually deleting files from `%USERPROFILE%\OneLake - Microsoft` will not delete data from your lakehouse, as these local files are merely metadata placeholders representing OneLake structure. The OneLake File Explorer syncs metadata (file/folder names, timestamps, etc.) but doesn’t store actual data unless explicitly downloaded