Forum Discussion
Memorable MemberRandom Lakehouse 403 Forbidden (Assume no metadata directory.)
I'm trying to use Fabric notebooks, with a default lakehouse.
My notebooks will succeed most of the time. Then they will randomly start failing for no particularly good reason. IMO, everything should work more consistently from one notebook execution to the next.
After digging in to the "stderr" of the driver, my notebooks appear to initialize with the correct default lakehouse, and a few util-mounts that I introduce myself. But later if I use certain lakehouse commands (like "saveAsTable") the stderr will start generating some really weird messages and the whole notebook will crash. One example is shown below. Notice there is an innocent warning for FileStreamSink:
.. WARN FileStreamSink [Thread-60]: Assume no metadata directory. Error while looking for metadata directory in the path:
..but it points to an adls location that it pulled out of thin air. It is VERY worrisome, in my case, given I'm doing work in my DEV workspace environment, but the notebook somehow starts trying to get metadata from some random PROD workspace.
Can someone please explain what this part of the log (WARN FileStreamSink) is trying to say to us? Here is the full stack:
I suspect these bugs are related to "lakehouse schemas" which are in preview, but I never opted into them, so I'm not sure why they are having an effect on my notebooks.
Hi ABD128
Thanks for the reply. I think I will need to use the API's you provided to do a certain amount of validation before running notebook code. There appears to be a high risk that any Fabric user who has access to both DEV and PROD will potentially contaminate one environment with the data from the other. It is easy to do, and very worrisome.
I took another look and my "external table" in DEV is pointed at PROD. I think this is the type of problem you were hinting at before.I suspect what happened is that someone (probably me) must have opened the notebook in the production environment after a failure and stepped thru some portion of it, so that we could determine what was going wrong.
... The problem is that the first two steps of the notebook set up the default lakehouse to be the "DEV" environment (!) and this configuration will stick in place for the rest of the notebook when it is used interactively. (it is only different if/when the notebook is executed by way of the REST API or pipelines.)
The first two steps are like so:
... notice how those cells will point at DEV by default. It was built this way since the vast majority of development work happens in the DEV environment. But if/when the notebook is ever executed in this way in production then it will behave in unexpected ways, and will cause problems!
6 Replies
ABD128Resolver II
Hi dbeavon3
The error occurs because Microsoft Fabric notebooks interact with Spark and Lakehouse storage in a way that can cause issues during runtime. The warning “Assume no metadata directory” means Spark couldn’t find metadata like _delta_log for a Delta Lake table, so it defaulted to using a standard Parquet format. The unusual part is that Spark is trying to access a path in the PROD Lakehouse environment, even though your notebook is running in a DEV workspace. This results in a 403 Forbidden error from Azure Data Lake Storage (ADLS), since your session doesn’t have permission for the PROD location. This usually happens due to leftover session metadata or cached references from previous runs, such as temporary views or mount points that Spark retains between notebook executions. Features like Lakehouse schema inference may also try to reuse table definitions across environments. To prevent this, clear the Spark session state at the start of each run, avoid using .saveAsTable() without specifying the output path, and ensure all file operations are directed to your DEV workspace. This will help keep your environments separate and prevent access errors.
Thanks.
dbeavon3Please be specific about the steps that should be used to "clear session state". That should not be a concern for a notebook developer. The session is brand new, and I'm not using "high concurrency" features or anything like that.
Are there any API's where I can run operations to reflectively inspect the session prior to using commands like "saveAsTable"? Obviously no developer wants to be sharing metadata between a DEV and PROD environment. I think this is a VERY serious bug if it is working the way we believe it is working. My DEV and PROD environments should not even know about each other, let alone share data back and forth. Thankfully the distinct permissions in each environment prevented the data-sharing.
Your response definitely seems to be generated by AI or something. It is not providing specific or actionable information, and I'm not really that much better off than I was when I first posted my question. It is also a lot more alarming that intended, and shares no mitigating details about why the bug may arise in some cases and not others. If Microsoft themselves had the information you shared, they would probably never have released any of this stuff to GA:
- ABD128
Hi,
To clarify, my previous explanation was a structured summary based on observed behavior in Microsoft Fabric notebooks and Spark runtime characteristics, not auto-generated or AI content. The goal was to highlight a possible root cause unexpected path resolution during Spark execution and to suggest best practices for maintaining workspace isolation. While notebook sessions are expected to be clean and isolated, experience and community feedback show that Spark sessions in managed environments like Fabric can sometimes retain cached references or environment-specific artifacts, especially when session reuse or implicit schema inference occurs. This is due to how Fabric manages Spark sessions, not necessarily because high-concurrency mode is enabled. I agree that developers should not need to manage session state manually, and cross-environment issues should not happen. If PROD paths are being accessed during a DEV session without explicit reference, it may point to a deeper issue with Fabric’s runtime or metadata resolution. Currently, Fabric offers limited APIs for checking session state, but Spark session methods like spark.catalog.listTables() and spark.catalog.listDatabases() can help identify lingering references before execution. Given the potential impact on environment boundaries, I recommend contacting Microsoft Support with your workspace and notebook details. They can investigate internal logs and provide more information on session management and workspace isolation, especially if preview features like Lakehouse schemas are involved.
Regards,
ABD128.
v-karpurapudCommunity Support
Hi dbeavon3
We have not yet received a response from you regarding your query. If the previous response was helpful, please let us know. If not, feel free to provide more details so we can assist you further.
Thank you.