topic Re: File directory not synchronized - synfs after mssparkutils.fs.mount in Data Engineering

File directory not synchronized - synfs after mssparkutils.fs.mount

dbeavon3 — Fri, 03 Oct 2025 14:10:37 GMT

I had been struggling for about 10 minutes to read a file from the lakehouse.

I mounted like so:

fspath = mssparkutils.fs.mount(my_files_path, "/LakehouseFiles")

... then I added new directories and a new file in to the lakehouse "Files".

The file was called "Partition.xml".

Then I tried to read the Partition.xml file (below) and it kept failing:

FileNotFoundError: [Errno 2] No such file or directory: '/synfs/notebook/xxxxxxx-cf8e-45a8-9097-15f0db4f3883/LakehouseFiles/InventoryManagement/InventoryAgings/Partition.xml'

Finally after about ten minutes it worked. What a nightmare.... Can someone tell me if this is expected behavior for the "synfs" mounts? This crappy behavior sounds like the par for the course, based on my other experiences on this platform, but ten minutes is way too long. Is there anyway to force the mssparkutils.fs mounts to behave better? Eg. should I synch on demand?

Re: File directory not synchronized - synfs after mssparkutils.fs.mount

tayloramy — Fri, 03 Oct 2025 14:28:29 GMT

Hi @dbeavon3,

You’re not imagining it - the /synfs path you get back from mssparkutils.fs.getMountPath(...) is an eventually-consistent projection of OneLake into the notebook’s file system. When you add or rename files outside the running Spark session (e.g., via the Lakehouse UI, another job, or upload), it can take a while before the mount’s directory listing catches up. That’s why your open("/synfs/notebook/.../Partition.xml") failed for several minutes and then “magically” started working.

Two workarounds:

Refresh the mounts before you read
```
import mssparkutils
mssparkutils.fs.refreshMounts()
```
This forces the workspace/job mount metadata to resync so new files show up faster. See: Microsoft Spark Utilities docs and the refreshMounts reference notes here: Synapse mount API (same APIs).
(Better) Skip /synfs for reads and use an ABFSS path
Read the file directly from OneLake instead of the mounted filesystem. With Python, use fsspec so you can call open() on ABFSS:
```
import fsspec

abfss_path = "abfss://<workspaceId>@onelake.dfs.fabric.microsoft.com/<LakehouseName>/Files/InventoryManagement/InventoryAgings/Partition.xml"
with fsspec.open(abfss_path, "rb") as f:
    full_doc = f.read()
```
This avoids the /synfs cache entirely. Nice walkthrough: Using fsspec with OneLake.

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

Re: File directory not synchronized - synfs after mssparkutils.fs.mount

dbeavon3 — Fri, 03 Oct 2025 14:51:07 GMT

Thanks. I wanted to use more conventional python libraries (os). Sometimes that makes it easier for AI assistance, and for migrating solutions to other platforms as needed (or for simply copy/pasting sample pyspark scripts to other standard python solutions).

I like the tip about refreshMounts. I will do that more frequently. Only about 1% of my operations read directly from files. Most of the file operations are for reading/writing blob data from abfss via pyspark.