Forum Discussion

mlukicic's avatar
mlukicic
Regular Visitor
10 months ago

Lakehouse Tutorial issues preparing and transforming data in the lakehouse

Hey Everyone, 

 

I am following the End-to-end Lake House Tutorial under Lakehouse tutorial - Prepare and transform lakehouse data - Microsoft Fabric | Microsoft Learn .

 

When running the second Notebook `02 - Data Transformation - Business Aggregates.ipynb` (step 20 from the https://learn.microsoft.com/en-us/fabric/data-engineering/tutorial-lakehouse-data-preparation) : 

df_fact_sale = spark.read.table("wwilakehouse.fact_sale") 
df_dimension_date = spark.read.table("wwilakehouse.dimension_date")
df_dimension_city = spark.read.table("wwilakehouse.dimension_city")


...I get an AnalysisException:

---------------------------------------------------------------------------
AnalysisException                         Traceback (most recent call last)
Cell In[11], line 1
----> 1 df_fact_sale = spark.read.table("wwilakehouse.fact_sale") 
      2 df_dimension_date = spark.read.table("wwilakehouse.dimension_date")
      3 df_dimension_city = spark.read.table("wwilakehouse.dimension_city")

File /opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py:484, in DataFrameReader.table(self, tableName)
    450 def table(self, tableName: str) -> "DataFrame":
    451     """Returns the specified table as a :class:`DataFrame`.
    452 
    453     .. versionadded:: 1.4.0
   (...)
    482     >>> _ = spark.sql("DROP TABLE tblA")
    483     """
--> 484     return self._df(self._jreader.table(tableName))

File ~/cluster-env/trident_env/lib/python3.11/site-packages/py4j/java_gateway.py:1322, in JavaMember.__call__(self, *args)
   1316 command = proto.CALL_COMMAND_NAME +\
   1317     self.command_header +\
   1318     args_command +\
   1319     proto.END_COMMAND_PART
   1321 answer = self.gateway_client.send_command(command)
-> 1322 return_value = get_return_value(
   1323     answer, self.gateway_client, self.target_id, self.name)
   1325 for temp_arg in temp_args:
   1326     if hasattr(temp_arg, "_detach"):

File /opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py:185, in capture_sql_exception.<locals>.deco(*a, **kw)
    181 converted = convert_exception(e.java_exception)
    182 if not isinstance(converted, UnknownException):
    183     # Hide where the exception came from that shows a non-Pythonic
    184     # JVM exception message.
--> 185     raise converted from None
    186 else:
    187     raise

AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view `wwilakehouse`.`fact_sale` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS.;
'UnresolvedRelation [wwilakehouse, fact_sale], [], false


How can I solve this issue?

 

Thanks for your help.

9 Replies

  • try database.schema.table names like wwilakehouse.dbo.fact_sale

    or reading Tables\fact_sales

    • mlukicic's avatar
      mlukicic
      Regular Visitor

      Hi!

      Thank you for your suggestion.

      wwilakehouse.dbo.fact_sale raises the same AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] Exception.

       

      I can read the parquet file with:

      df = spark.read.format("parquet").load('Files/wwi-raw-data/full/fact_sale')

      However, I would like to read the Delta files (like tutorial does).

       

      I noticed two things. First, in Explorer, under the wwilakehouse / Tables, I have Undefined folders under each dimension_ table.

       

      And second, running:

      spark.sql("SELECT current_catalog()").show(truncate=False)
      spark.sql("SELECT current_database()").show(truncate=False)

      will return:

      +-----------------+
      |current_catalog()|
      +-----------------+
      |spark_catalog |
      +-----------------+
      
      +----------------------------------------------------------------------------------------+
      |current_database() |
      +----------------------------------------------------------------------------------------+
      |chimcobldhq2ahj1c9p6iop0dhgmmpb8dtqn6p90ahqn8rrid5gmo81d410k29bnetkmoobbclk6utbjckim8ojf|
      +----------------------------------------------------------------------------------------+

      Maybe I am interpreting it wrong, but it seems to me like spark is not attached to the lakehouse (even the Notebook is).

       

      Thanks,
      Marko

      • v-aatheeque's avatar
        v-aatheeque
        Icon for Community Support rankCommunity Support

        Hi mlukicic 

        It looks like your Spark session isn’t properly connected to the Lakehouse, which is why the table reference fails. The output showing spark_catalog and a long default database name means Spark is running in a default workspace context, not inside the Lakehouse.

         

        Try this suggestion points as below :

        • Go to the notebook toolbar.
        • Look for the “Lakehouse” icon or “Attached items.”
        • Click “Attach to” and select your Lakehouse :  wwilakehouse.
        • Wait until it shows up under “Attached items” in the Explorer panel on the left.

        Once attached, re-run your queries and it should be able to access the Lakehouse tables correctly.

        Hope this helps !!
        Thank You.