Forum Discussion
Spark SQL ABFS Path Not Working from Schema Enabled Lakehouses
Hi v-sathmakuri ,
I'm fairly sure it's related to the OneLake Security Private Preview. Received the activation email a couple days after I posted this so didn't realize it was activated at the time. But it would explain why the issue is specific to schema enabled lakehouses and those five workspaces for our dev environment. Emailed the OneLake Security Private Preview team and am waiting to hear back.
Checked the points you mentioned and resources and settings look good. The notebook also fails after only 12 seconds when can handle the same and a lot more just by changing the default connection to a non-schema enabled lakehouse. Thanks.
There is a part of our dev ETL pipeline that requires it to be connected to a schema enabled lakehouse when collecting info and creating Materialized Lake Views from notebooks so did a pipeline deployment from dev to test and it's working normally in test; just need to rebuild dev now.
Hi PANDAmonium ,
Please check the troubleshooting steps below to help determine the root cause of the issue.
- Try running a query on a different table in LakehouseA to see if the problem is specific to one table.
- Check workspace-level policies or capacity settings that could impact long-running sessions.
- Verify that the workspace has sufficient compute resources, such as memory, cores, and concurrency.
- Some workspaces have stricter execution time limits, refer to the screenshot below for details.
Thanks!!
Hi v-sathmakuri ,
I'm fairly sure it's related to the OneLake Security Private Preview. Received the activation email a couple days after I posted this so didn't realize it was activated at the time. But it would explain why the issue is specific to schema enabled lakehouses and those five workspaces for our dev environment. Emailed the OneLake Security Private Preview team and am waiting to hear back.
Checked the points you mentioned and resources and settings look good. The notebook also fails after only 12 seconds when can handle the same and a lot more just by changing the default connection to a non-schema enabled lakehouse. Thanks.
There is a part of our dev ETL pipeline that requires it to be connected to a schema enabled lakehouse when collecting info and creating Materialized Lake Views from notebooks so did a pipeline deployment from dev to test and it's working normally in test; just need to rebuild dev now.
Hi PANDAmonium ,
Thanks for sharing the information.
Good to hear it’s working as expected just by changing the default connection to a non-schema enabled lakehouse your test environment.
Please let us know if you face any issues while rebuilding the dev environment or any other help needed from our end.
Thanks!!