Forum Discussion

mlukicic's avatar
mlukicic
Regular Visitor
10 months ago

Lakehouse Tutorial issues preparing and transforming data in the lakehouse

Hey Everyone, 

 

I am following the End-to-end Lake House Tutorial under Lakehouse tutorial - Prepare and transform lakehouse data - Microsoft Fabric | Microsoft Learn .

 

When running the second Notebook `02 - Data Transformation - Business Aggregates.ipynb` (step 20 from the https://learn.microsoft.com/en-us/fabric/data-engineering/tutorial-lakehouse-data-preparation) : 

df_fact_sale = spark.read.table("wwilakehouse.fact_sale") 
df_dimension_date = spark.read.table("wwilakehouse.dimension_date")
df_dimension_city = spark.read.table("wwilakehouse.dimension_city")


...I get an AnalysisException:

---------------------------------------------------------------------------
AnalysisException                         Traceback (most recent call last)
Cell In[11], line 1
----> 1 df_fact_sale = spark.read.table("wwilakehouse.fact_sale") 
      2 df_dimension_date = spark.read.table("wwilakehouse.dimension_date")
      3 df_dimension_city = spark.read.table("wwilakehouse.dimension_city")

File /opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py:484, in DataFrameReader.table(self, tableName)
    450 def table(self, tableName: str) -> "DataFrame":
    451     """Returns the specified table as a :class:`DataFrame`.
    452 
    453     .. versionadded:: 1.4.0
   (...)
    482     >>> _ = spark.sql("DROP TABLE tblA")
    483     """
--> 484     return self._df(self._jreader.table(tableName))

File ~/cluster-env/trident_env/lib/python3.11/site-packages/py4j/java_gateway.py:1322, in JavaMember.__call__(self, *args)
   1316 command = proto.CALL_COMMAND_NAME +\
   1317     self.command_header +\
   1318     args_command +\
   1319     proto.END_COMMAND_PART
   1321 answer = self.gateway_client.send_command(command)
-> 1322 return_value = get_return_value(
   1323     answer, self.gateway_client, self.target_id, self.name)
   1325 for temp_arg in temp_args:
   1326     if hasattr(temp_arg, "_detach"):

File /opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py:185, in capture_sql_exception.<locals>.deco(*a, **kw)
    181 converted = convert_exception(e.java_exception)
    182 if not isinstance(converted, UnknownException):
    183     # Hide where the exception came from that shows a non-Pythonic
    184     # JVM exception message.
--> 185     raise converted from None
    186 else:
    187     raise

AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view `wwilakehouse`.`fact_sale` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS.;
'UnresolvedRelation [wwilakehouse, fact_sale], [], false


How can I solve this issue?

 

Thanks for your help.

9 Replies

  • try database.schema.table names like wwilakehouse.dbo.fact_sale

    or reading Tables\fact_sales

    • mlukicic's avatar
      mlukicic
      Regular Visitor

      Hi!

      Thank you for your suggestion.

      wwilakehouse.dbo.fact_sale raises the same AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] Exception.

       

      I can read the parquet file with:

      df = spark.read.format("parquet").load('Files/wwi-raw-data/full/fact_sale')

      However, I would like to read the Delta files (like tutorial does).

       

      I noticed two things. First, in Explorer, under the wwilakehouse / Tables, I have Undefined folders under each dimension_ table.

       

      And second, running:

      spark.sql("SELECT current_catalog()").show(truncate=False)
      spark.sql("SELECT current_database()").show(truncate=False)

      will return:

      +-----------------+
      |current_catalog()|
      +-----------------+
      |spark_catalog |
      +-----------------+
      
      +----------------------------------------------------------------------------------------+
      |current_database() |
      +----------------------------------------------------------------------------------------+
      |chimcobldhq2ahj1c9p6iop0dhgmmpb8dtqn6p90ahqn8rrid5gmo81d410k29bnetkmoobbclk6utbjckim8ojf|
      +----------------------------------------------------------------------------------------+

      Maybe I am interpreting it wrong, but it seems to me like spark is not attached to the lakehouse (even the Notebook is).

       

      Thanks,
      Marko

      • v-aatheeque's avatar
        v-aatheeque
        Community Support

        Hi mlukicic 

        It looks like your Spark session isn’t properly connected to the Lakehouse, which is why the table reference fails. The output showing spark_catalog and a long default database name means Spark is running in a default workspace context, not inside the Lakehouse.

         

        Try this suggestion points as below :

        • Go to the notebook toolbar.
        • Look for the “Lakehouse” icon or “Attached items.”
        • Click “Attach to” and select your Lakehouse :  wwilakehouse.
        • Wait until it shows up under “Attached items” in the Explorer panel on the left.

        Once attached, re-run your queries and it should be able to access the Lakehouse tables correctly.

        Hope this helps !!
        Thank You.