Lakehouse Tutorial issues preparing and transforming data in the lakehouse

[mlukicic]

Hey Everyone, 

I am following the End-to-end Lake House Tutorial under Lakehouse tutorial - Prepare and transform lakehouse data - Microsoft Fabric | Microsoft Learn .

When running the second Notebook `02 - Data Transformation - Business Aggregates.ipynb` (step 20 from the https://learn.microsoft.com/en-us/fabric/data-engineering/tutorial-lakehouse-data-preparation) : 

df_fact_sale = spark.read.table("wwilakehouse.fact_sale") 
df_dimension_date = spark.read.table("wwilakehouse.dimension_date")
df_dimension_city = spark.read.table("wwilakehouse.dimension_city")

...I get an AnalysisException:

---------------------------------------------------------------------------
AnalysisException                         Traceback (most recent call last)
Cell In[11], line 1
----> 1 df_fact_sale = spark.read.table("wwilakehouse.fact_sale") 
      2 df_dimension_date = spark.read.table("wwilakehouse.dimension_date")
      3 df_dimension_city = spark.read.table("wwilakehouse.dimension_city")

File /opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py:484, in DataFrameReader.table(self, tableName)
    450 def table(self, tableName: str) -> "DataFrame":
    451     """Returns the specified table as a :class:`DataFrame`.
    452 
    453     .. versionadded:: 1.4.0
   (...)
    482     >>> _ = spark.sql("DROP TABLE tblA")
    483     """
--> 484     return self._df(self._jreader.table(tableName))

File ~/cluster-env/trident_env/lib/python3.11/site-packages/py4j/java_gateway.py:1322, in JavaMember.__call__(self, *args)
   1316 command = proto.CALL_COMMAND_NAME +\
   1317     self.command_header +\
   1318     args_command +\
   1319     proto.END_COMMAND_PART
   1321 answer = self.gateway_client.send_command(command)
-> 1322 return_value = get_return_value(
   1323     answer, self.gateway_client, self.target_id, self.name)
   1325 for temp_arg in temp_args:
   1326     if hasattr(temp_arg, "_detach"):

File /opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py:185, in capture_sql_exception.<locals>.deco(*a, **kw)
    181 converted = convert_exception(e.java_exception)
    182 if not isinstance(converted, UnknownException):
    183     # Hide where the exception came from that shows a non-Pythonic
    184     # JVM exception message.
--> 185     raise converted from None
    186 else:
    187     raise

AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view `wwilakehouse`.`fact_sale` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS.;
'UnresolvedRelation [wwilakehouse, fact_sale], [], false

How can I solve this issue?

Thanks for your help.

[Thomaslleblanc]

try database.schema.table names like wwilakehouse.dbo.fact_sale

or reading Tables\fact_sales

[mlukicic]

Hi!

Thank you for your suggestion.

wwilakehouse.dbo.fact_sale raises the same AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] Exception.

I can read the parquet file with:

df = spark.read.format("parquet").load('Files/wwi-raw-data/full/fact_sale')

However, I would like to read the Delta files (like tutorial does).

I noticed two things. First, in Explorer, under the wwilakehouse / Tables, I have Undefined folders under each dimension_ table.

And second, running:

spark.sql("SELECT current_catalog()").show(truncate=False)
spark.sql("SELECT current_database()").show(truncate=False)

will return:

+-----------------+
|current_catalog()|
+-----------------+
|spark_catalog |
+-----------------+

+----------------------------------------------------------------------------------------+
|current_database() |
+----------------------------------------------------------------------------------------+
|chimcobldhq2ahj1c9p6iop0dhgmmpb8dtqn6p90ahqn8rrid5gmo81d410k29bnetkmoobbclk6utbjckim8ojf|
+----------------------------------------------------------------------------------------+

Maybe I am interpreting it wrong, but it seems to me like spark is not attached to the lakehouse (even the Notebook is).

Thanks,
Marko

[v-aatheeque]

Hi @mlukicic 

It looks like your Spark session isn’t properly connected to the Lakehouse, which is why the table reference fails. The output showing spark_catalog and a long default database name means Spark is running in a default workspace context, not inside the Lakehouse.

Try this suggestion points as below :

• Go to the notebook toolbar.
• Look for the “Lakehouse” icon or “Attached items.”
• Click “Attach to” and select your Lakehouse :  wwilakehouse.
• Wait until it shows up under “Attached items” in the Explorer panel on the left.

Once attached, re-run your queries and it should be able to access the Lakehouse tables correctly.

Hope this helps !!
Thank You.

[mlukicic]

Hi @v-aatheeque!

Thank you for your reply. Can you help me locate the "Lakehouse" icon or "Attached items"?

Am I missing something? 

Thank you very much for your help,

Marko

[v-aatheeque]

Hi @mlukicic 

Based on the screenshot you shared, I followed the steps from the provided document (Lakehouse tutorial - Prepare and transform lakehouse data - Microsoft Fabric | Microsoft Learn) and was able to get the expected output as shown below.

Can you once cross check this below steps :

1. Refresh your Lakehouse view to make sure the tables (fact_sale, dimension_date, and dimension_city) are actually there.

2. Check that your Spark code is using the right schema.

3. Make sure your notebook is connected to the right Lakehouse.

4. Double-check the table names, even a small spelling mistake can cause this error.

To attach the lakehouse as per the newFabric UI :

• In the Explorer panel, click + Add data items.
• Select New Lakehouse/Existing data sources.
• Choose your Lakehouse (e.g., wwilakehouse) and click Add.
• Ensure it appears under Items with a database icon.

Hope this helps !!
Thank You.

[v-aatheeque]

Hi @mlukicic 

Have you had a chance to look through the responses shared earlier? If anything is still unclear, we’ll be happy to provide additional support.

[mlukicic]

Hi @v-aatheeque,

Thank you for your help and time.

Unfortunately, I couldn’t get this to work. I deleted the entire Lakehouse and started over, following the tutorial step by step. However, I noticed some differences between my UI and the tutorial.

When I opened the Notebooks from the tutorial, I saw that some resources were attached in the Explorer, all marked with red icons and a white cross, indicating an error. This happened on both Notebooks. I deleted all these resources, added the wwilakehouse Lakehouse, and set it as the default resource.

The first Notebook ran without any issues. However, in the second Notebook, the files were not auto-registered as Delta tables as described in the tutorial.

Unless you have any suggestions on where to look next, I’m closing this and moving forward. From my perspective as a machine learning engineer, Fabric seems workable, but it still needs improvement.

Thanks,

Marko

[v-aatheeque]

Hi @mlukicic 

Thank you for the detailed update ,It’s great to see the thorough steps you’ve taken to validate and troubleshoot the tutorial process.

The differences you noticed in the UI and the auto-registration behavior are valuable observations. Since Fabric is still evolving rapidly, your feedback could help improve future experiences.

If you’d like, you can share this suggestion directly through the Microsoft Fabric Ideas portal (https://ideas.fabric.microsoft.com/). The product team actively reviews user inputs there for feature improvements.

Appreciate your time exploring this and sharing your insights!

[mlukicic]

Dear @v-aatheeque ,

Many thanks. And thank you for the link. In fact, I would like to share a few suggestions, as I believe the strategy is great and could become an excellent tool for machine learning engineers working with graphs as well.

Kind regards,

Marko