Issue with SparkSQL Schema Behavior in Fabric

Hi SKONA,

Thank you for contacting the Microsoft Fabric Community Forum and for the detailed explanation.

Based on my understanding, the behavior you are observing is related to how Spark SQL resolves catalog and schema context in Fabric, particularly when using Lakehouse Schemas (Preview). Fabric has recently aligned with the standard Spark/Unity Catalog namespace resolution. As a result, your Spark session is no longer automatically scoped to your lakehouse’s catalog. When you execute USE SCHEMA RawStore; Spark searches for RawStore in the current catalog, which in your session is not the lakehouse catalog, causing the error: [SCHEMA_NOT_FOUND] The schema 'rawstore' cannot be found. When you run USE SCHEMA workspace.lakehouse.rawstore; you explicitly specify the correct catalog and schema, so the query succeeds. This issue may occur after runtime updates or when switching between notebook and pipeline sessions.

Please follow the steps below, which may help resolve the issue:

1. Check your current Spark session context by running:
   SELECT current_catalog(), current_schema();
2. Set your lakehouse catalog and schema explicitly:
   USE CATALOG <workspace>;
   USE SCHEMA <lakehouse>.<RawStore>;
3. Re-run your original script.

Alternatively, use fully qualified table paths as shown below. This approach works for pipelines and production workloads:
FROM workspace.lakehouse.rawstore.mvst

For further reference, please refer the link below:
Lakehouse schemas (Preview) - Microsoft Fabric | Microsoft Learn

We hope the information helps resolve the issue. If you have any further queries, please feel free to contact the Microsoft Fabric community.

Thank you.

Hi SKONA,

We would like to follow up and see whether the details we shared have resolved your problem. If you need any more assistance, please feel free to connect with the Microsoft Fabric community.

Thank you.

Thank you for your kind words, I'm glad the information was helpful!