Forum Discussion
Lakehouse Tutorial issues preparing and transforming data in the lakehouse
Hey Everyone,
I am following the End-to-end Lake House Tutorial under Lakehouse tutorial - Prepare and transform lakehouse data - Microsoft Fabric | Microsoft Learn .
When running the second Notebook `02 - Data Transformation - Business Aggregates.ipynb` (step 20 from the https://learn.microsoft.com/en-us/fabric/data-engineering/tutorial-lakehouse-data-preparation) :
df_fact_sale = spark.read.table("wwilakehouse.fact_sale")
df_dimension_date = spark.read.table("wwilakehouse.dimension_date")
df_dimension_city = spark.read.table("wwilakehouse.dimension_city")
...I get an AnalysisException:
---------------------------------------------------------------------------
AnalysisException Traceback (most recent call last)
Cell In[11], line 1
----> 1 df_fact_sale = spark.read.table("wwilakehouse.fact_sale")
2 df_dimension_date = spark.read.table("wwilakehouse.dimension_date")
3 df_dimension_city = spark.read.table("wwilakehouse.dimension_city")
File /opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py:484, in DataFrameReader.table(self, tableName)
450 def table(self, tableName: str) -> "DataFrame":
451 """Returns the specified table as a :class:`DataFrame`.
452
453 .. versionadded:: 1.4.0
(...)
482 >>> _ = spark.sql("DROP TABLE tblA")
483 """
--> 484 return self._df(self._jreader.table(tableName))
File ~/cluster-env/trident_env/lib/python3.11/site-packages/py4j/java_gateway.py:1322, in JavaMember.__call__(self, *args)
1316 command = proto.CALL_COMMAND_NAME +\
1317 self.command_header +\
1318 args_command +\
1319 proto.END_COMMAND_PART
1321 answer = self.gateway_client.send_command(command)
-> 1322 return_value = get_return_value(
1323 answer, self.gateway_client, self.target_id, self.name)
1325 for temp_arg in temp_args:
1326 if hasattr(temp_arg, "_detach"):
File /opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py:185, in capture_sql_exception.<locals>.deco(*a, **kw)
181 converted = convert_exception(e.java_exception)
182 if not isinstance(converted, UnknownException):
183 # Hide where the exception came from that shows a non-Pythonic
184 # JVM exception message.
--> 185 raise converted from None
186 else:
187 raise
AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view `wwilakehouse`.`fact_sale` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS.;
'UnresolvedRelation [wwilakehouse, fact_sale], [], false
How can I solve this issue?
Thanks for your help.
9 Replies
- ThomaslleblancSuper User
try database.schema.table names like wwilakehouse.dbo.fact_sale
or reading Tables\fact_sales
- mlukicicRegular Visitor
Hi!
Thank you for your suggestion.
wwilakehouse.dbo.fact_sale raises the same AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] Exception.
I can read the parquet file with:
df = spark.read.format("parquet").load('Files/wwi-raw-data/full/fact_sale')However, I would like to read the Delta files (like tutorial does).
I noticed two things. First, in Explorer, under the wwilakehouse / Tables, I have Undefined folders under each dimension_ table.
And second, running:
spark.sql("SELECT current_catalog()").show(truncate=False) spark.sql("SELECT current_database()").show(truncate=False)will return:
+-----------------+ |current_catalog()| +-----------------+ |spark_catalog | +-----------------+ +----------------------------------------------------------------------------------------+ |current_database() | +----------------------------------------------------------------------------------------+ |chimcobldhq2ahj1c9p6iop0dhgmmpb8dtqn6p90ahqn8rrid5gmo81d410k29bnetkmoobbclk6utbjckim8ojf| +----------------------------------------------------------------------------------------+Maybe I am interpreting it wrong, but it seems to me like spark is not attached to the lakehouse (even the Notebook is).
Thanks,
Marko- v-aatheequeCommunity Support
Hi mlukicic
It looks like your Spark session isn’t properly connected to the Lakehouse, which is why the table reference fails. The output showing spark_catalog and a long default database name means Spark is running in a default workspace context, not inside the Lakehouse.
Try this suggestion points as below :
- Go to the notebook toolbar.
- Look for the “Lakehouse” icon or “Attached items.”
- Click “Attach to” and select your Lakehouse : wwilakehouse.
- Wait until it shows up under “Attached items” in the Explorer panel on the left.
Once attached, re-run your queries and it should be able to access the Lakehouse tables correctly.
Hope this helps !!
Thank You.