Forum Discussion
Spark SQL ABFS Path Not Working from Schema Enabled Lakehouses
Hi v-sathmakuri ,
I'm fairly sure it's related to the OneLake Security Private Preview. Received the activation email a couple days after I posted this so didn't realize it was activated at the time. But it would explain why the issue is specific to schema enabled lakehouses and those five workspaces for our dev environment. Emailed the OneLake Security Private Preview team and am waiting to hear back.
Checked the points you mentioned and resources and settings look good. The notebook also fails after only 12 seconds when can handle the same and a lot more just by changing the default connection to a non-schema enabled lakehouse. Thanks.
There is a part of our dev ETL pipeline that requires it to be connected to a schema enabled lakehouse when collecting info and creating Materialized Lake Views from notebooks so did a pipeline deployment from dev to test and it's working normally in test; just need to rebuild dev now.
Hi PANDAmonium ,
Thank you for reaching out to Microsoft Fabric Community.
I have tried reproducing your scenario, I have created a Lakehouse one with Schema enabled and other without schema enabling. Uploaded the same file to both the Lakehouse and created a delta table, tried accessing the table from the Lakehouse using ABFSS Path and I was able to execute it successfully. PFBS for reference.
Please execute the same from your end again and let me know if there is any change in the repro steps I have followed.
Thank you!!
Hi @v-shr ,
It looks like the error is occuring only with notebooks created in our Dev pipeline which includes 5 workspaces. So if I create a notebook in any of those workspaces and repeat those steps, it fails. Create a notebook in any other workspace, it works as expected even using the same exact steps, same connected lakehouse, same environment, same capacity, just within a different workspace. Best guess is I was doing some work with Materialized Lake Views between the time it last worked, but it's also failing in a dev workspace that doesn't have any lakehouses or MLVs so I have no idea. Included a screenshot of the full error.
We have our workaround though, connecting it to a non-schema lakehouse. If the issue doesn't fix itself in time, after we deploy to our Test pipeline, we'll just delete and redeploy the Dev pipeline.
Thanks for reproducing the steps and following up. Understanding it's an isolated issue helped me figure out what to do next. If the screenshot/error helps identify the cause of the issue, please let me know.
Hi PANDAmonium ,
Please check the troubleshooting steps below to help determine the root cause of the issue.
- Try running a query on a different table in LakehouseA to see if the problem is specific to one table.
- Check workspace-level policies or capacity settings that could impact long-running sessions.
- Verify that the workspace has sufficient compute resources, such as memory, cores, and concurrency.
- Some workspaces have stricter execution time limits, refer to the screenshot below for details.
Thanks!!
Hi v-sathmakuri ,
I'm fairly sure it's related to the OneLake Security Private Preview. Received the activation email a couple days after I posted this so didn't realize it was activated at the time. But it would explain why the issue is specific to schema enabled lakehouses and those five workspaces for our dev environment. Emailed the OneLake Security Private Preview team and am waiting to hear back.
Checked the points you mentioned and resources and settings look good. The notebook also fails after only 12 seconds when can handle the same and a lot more just by changing the default connection to a non-schema enabled lakehouse. Thanks.
There is a part of our dev ETL pipeline that requires it to be connected to a schema enabled lakehouse when collecting info and creating Materialized Lake Views from notebooks so did a pipeline deployment from dev to test and it's working normally in test; just need to rebuild dev now.
Hi PANDAmonium ,
Thanks for sharing the information.
Good to hear it’s working as expected just by changing the default connection to a non-schema enabled lakehouse your test environment.
Please let us know if you face any issues while rebuilding the dev environment or any other help needed from our end.
Thanks!!