topic Re: Cannot use spark.catalog.listTables in notebook in Data Engineering

Cannot use spark.catalog.listTables in notebook

dbeavon3 — Fri, 01 Aug 2025 20:03:55 GMT

I have errors when calling spark.catalog.listTables() in fabic, with a default lakehouse connected.

The error is like so:

[PARSE_SYNTAX_ERROR] Syntax error at or near end of input.(line 1, pos 0) == SQL == ^^^

runtime 1.3
spark 3.5
delta 3.2

I did not enroll in an preview features as far as I'm aware. Here is the full screenshot.

According to the docs this stuff is supposed to be GA.
Why is there such a basic and incoherent error when listing tables???

https://learn.microsoft.com/en-us/fabric/data-engineering/runtime

Re: Cannot use spark.catalog.listTables in notebook

BhaveshPatel — Sun, 03 Aug 2025 09:34:21 GMT

There is definitely permissions issue in your spark environment. I can see the tables in spark.catalog.listTables(). It was available in Delta Lake 2.2.0 and it is part of data lake of spark.

Re: Cannot use spark.catalog.listTables in notebook

dbeavon3 — Mon, 04 Aug 2025 02:09:14 GMT

The error message does not say anything about permissions. It says :

PARSE_SYNTAX_ERROR

This bug was reported before in the community. The last time it was reported, the users were told to upgrade to a GA version of delta. I'm on GA for all versions.

If you have a command to prove or disprove your theory about permissions, please let me know. There are no clue's on my end to support that theory. I can use my default lakehouse fine, and can run almost every type of spark operation except I get failures on certain "catalog" operations from pyspark.

Re: Cannot use spark.catalog.listTables in notebook

dbeavon3 — Mon, 04 Aug 2025 02:12:27 GMT

This may be a problem in the delta project. I found the following:

https://github.com/delta-io/delta/issues/2610

Re: Cannot use spark.catalog.listTables in notebook

BhaveshPatel — Mon, 04 Aug 2025 07:57:42 GMT

This is still visible in my spark environment. There is a permissions issue.

use pyspark and notebooks altogether.

Re: Cannot use spark.catalog.listTables in notebook

v-pnaroju-msft — Tue, 05 Aug 2025 19:15:07 GMT

Thankyou, BhaveshPatel, for your response.

Hi dbeavon3,

We appreciate your inquiry submitted through the Microsoft Fabric Community Forum.

As per my understanding, the error “[PARSE_SYNTAX_ERROR] Syntax error at or near end of input” may occur if the default Lakehouse is not attached, if the Lakehouse is schema enabled which is a known limitation with certain Spark catalog APIs or due to the parsing behavior of Python 3.11 and Delta 3.2 generating invalid SQL for spark.catalog.listTables().

Kindly follow the steps below to resolve the issue:

Check the default database using spark.catalog.currentDatabase(). If this returns an error or shows default with no tables, you will need to specify your Lakehouse name explicitly.
List the available databases using spark.catalog.listDatabases(). Confirm that your Lakehouse name is present in the list.
Explicitly list tables from the Lakehouse by using spark.catalog.listTables("YourLakehouseName"). Please replace YourLakehouseName with the actual name of your Lakehouse.
Ensure that the notebook has the correct default Lakehouse attached.

Please find the attached screenshot regarding the schema enabled Lakehouse limitation:

Additionally, kindly refer to the following link for more information:
Lakehouse schemas (Preview) - Microsoft Fabric | Microsoft Learn

We hope that the above information will assist in resolving the issue. Should you have any further queries, please feel free to contact the Microsoft Fabric Community.

Thank you.

Re: Cannot use spark.catalog.listTables in notebook

v-pnaroju-msft — Sat, 09 Aug 2025 17:08:42 GMT

Hi dbeavon3,

We would like to follow up and see whether the details we shared have resolved your problem.
If you need any more assistance, please feel free to connect with the Microsoft Fabric community.

Thank you.

Re: Cannot use spark.catalog.listTables in notebook

v-pnaroju-msft — Tue, 12 Aug 2025 11:25:02 GMT

Hi dbeavon3,

We are following up to see if what we shared solved your issue. If you need more support, please reach out to the Microsoft Fabric community.

Thank you.

Re: Cannot use spark.catalog.listTables in notebook

v-pnaroju-msft — Fri, 15 Aug 2025 14:07:09 GMT

Hi dbeavon3,

We wanted to see if the information we gave helped fix your problem. If you need more help, please feel free to contact the Microsoft Fabric community.

Thank you.