Spark SQL ABFS Path Not Working from Schema Enabled Lakehouses

[PANDAmonium]

Hi All,

I'm unable to use the ABFS Path in a Spark SQL query when using a schema enabled lakehouse as the default lakehouse. Are there any known issues around this or is anyone having the same issue?

I have LakehouseA (schema) and LakehouseB (non schema) and Notebook1. I have two Spark SQL Queries in Notebook 1:

%%sql

SELECT

    *

FROM

    delta.`abfss://[Workspace]@onelake.dfs.fabric.microsoft.com/[LakehouseA].Lakehouse/Tables/dbo/[Table]`

LIMIT 1000

%%sql

SELECT

    *

FROM

    delta.`abfss://[Workspace]@onelake.dfs.fabric.microsoft.com/[LakehouseB].Lakehouse/Tables/[Table]`

LIMIT 1000

The ABFS paths were copied using 'Copy Path' so they're correct. When I use the non schema lakehouse, LakehouseB, as the default lakehouse both work. When I use the schema enabled lakehouse, LakehouseA, they fail with a long error message starting with this.

Exception thrown in awaitResult:

Tried swapping out everything I could think of, notebooks, target tables, workspaces. Just wanted to see if anyone had more info to figure out what my next steps should be. Didn't see this in the Fabric Known Issues page.

Thanks!

[v-sathmakuri]

Hi @PANDAmonium ,

Could you please let us know whether the provided response helped in resolving the issue. 

Thank you!!

[v-sathmakuri]

Hi @PANDAmonium ,

Thank you for reaching out to Microsoft Fabric Community.

I have tried reproducing your scenario, I have created a Lakehouse one with Schema enabled and other without schema enabling. Uploaded the same file to both the Lakehouse and created a delta table, tried accessing the table from the Lakehouse using ABFSS Path and I was able to execute it successfully. PFBS for reference.

Please execute the same from your end again and let me know if there is any change in the repro steps I have followed.

Thank you!!

[PANDAmonium]

Hi @v-shr ,

It looks like the error is occuring only with notebooks created in our Dev pipeline which includes 5 workspaces. So if I create a notebook in any of those workspaces and repeat those steps, it fails. Create a notebook in any other workspace, it works as expected even using the same exact steps, same connected lakehouse, same environment, same capacity, just within a different workspace. Best guess is I was doing some work with Materialized Lake Views between the time it last worked, but it's also failing in a dev workspace that doesn't have any lakehouses or MLVs so I have no idea. Included a screenshot of the full error.

We have our workaround though, connecting it to a non-schema lakehouse. If the issue doesn't fix itself in time, after we deploy to our Test pipeline, we'll just delete and redeploy the Dev pipeline.

Thanks for reproducing the steps and following up. Understanding it's an isolated issue helped me figure out what to do next. If the screenshot/error helps identify the cause of the issue, please let me know.

[v-sathmakuri]

Hi @PANDAmonium ,

Please check the troubleshooting steps below to help determine the root cause of the issue.

• Try running a query on a different table in LakehouseA to see if the problem is specific to one table.
• Check workspace-level policies or capacity settings that could impact long-running sessions.
• Verify that the workspace has sufficient compute resources, such as memory, cores, and concurrency.
• Some workspaces have stricter execution time limits, refer to the screenshot below for details.

Thanks!!

[PANDAmonium · Accepted solution]

Hi @v-sathmakuri ,

I'm fairly sure it's related to the OneLake Security Private Preview. Received the activation email a couple days after I posted this so didn't realize it was activated at the time. But it would explain why the issue is specific to schema enabled lakehouses and those five workspaces for our dev environment. Emailed the OneLake Security Private Preview team and am waiting to hear back.

Checked the points you mentioned and resources and settings look good. The notebook also fails after only 12 seconds when can handle the same and a lot more just by changing the default connection to a non-schema enabled lakehouse. Thanks.

There is a part of our dev ETL pipeline that requires it to be connected to a schema enabled lakehouse when collecting info and creating Materialized Lake Views from notebooks so did a pipeline deployment from dev to test and it's working normally in test; just need to rebuild dev now.

[v-sathmakuri]

Hi @PANDAmonium ,

Please let us know if you need any further assistance from our end.

Thank you!!

[v-sathmakuri]

Hi @PANDAmonium ,

Thanks for sharing the information. 

Good to hear it’s working as expected just by changing the default connection to a non-schema enabled lakehouse your test environment.

Please let us know if you face any issues while rebuilding the dev environment or any other help needed from our end.

Thanks!!