NotImplementedError in OnelakeExternalCatalog.functionExists / listFunctions in Fabric Runtime 1.3

[Padam_Prakash]

When using Fabric Runtime 1.3, calls to spark.catalog.functionExists(...) and spark.catalog.listFunctions() (or equivalents via Java/Scala sparkSession.catalog().functionExists(...)) throw a scala.NotImplementedError: an implementation is missing. in earlier runtimes (one month back) these worked (or at least did not crash). 
Our code include logic like:

if (!sparkSession.catalog().functionExists(functionName)) {
    sparkSession.udf().register(functionName, udfImpl);
}

that logic now fails at the functionExists() call (i.e. the NotImplementedError is thrown before registering).
we tried to check by running 

spark.catalog.listFunctions()

Py4JJavaError: An error occurred while calling oYYYY.listFunctions.
: scala.NotImplementedError: an implementation is missing
at scala.Predef$.$qmark$qmark$qmark(Predef.scala:288)
at com.microsoft.fabric.spark.catalog.OnelakeExternalCatalog.listFunctions(OnelakeExternalCatalog.scala:404)
at com.microsoft.fabric.spark.catalog.InstrumentedExternalCatalog.$anonfun$listFunctions$1(OnelakeExternalCatalog.scala:627)
…

Expected Behavior

spark.catalog.functionExists("someName") should return true or false depending on whether a function with that name is registered, without throwing an error.
Environment / Configuration Details
Fabric runtime version: 1.3

[tayloramy · Accepted solution]

Hi @Padam_Prakash,

This error means you’re calling a “functions API” on the external catalog (OneLake) that doesn’t implement function metadata in Fabric Runtime 1.3. In 1.3, OnelakeExternalCatalog.listFunctions and functionExists throw scala.NotImplementedError, so checks like spark.catalog.functionExists(...) now fail even before you try to (re)register your UDF. It’s expected in this runtime, though the change feels like a regression from earlier behavior.

 

Why it happens

• spark.catalog.functionExists and spark.catalog.listFunctions() route to the active catalog. In Fabric Lakehouses that’s the OneLake external catalog, whose function APIs aren’t implemented in Runtime 1.3. Hence:

  com.microsoft.fabric.spark.catalog.OnelakeExternalCatalog.listFunctions → NotImplementedError

• Your UDFs are typically session-scoped (temporary) and live in Spark’s function registry, not in the external catalog. So even conceptually, the external catalog isn’t the right place to ask about them.

Practical workarounds

1. Skip the existence check and (re)register

   In both Scala and PySpark, registering the same name overwrites the previous temp function. So you can safely register every run.

   Scala

   // Replace “functionExists + conditional” with unconditional register
   spark.udf.register(functionName, udfImpl)  // acts like “create or replace” for temp UDFs

   Python

   spark.udf.register(function_name, udf_impl)  # overwrites same-named temp UDF

2. If you must check first, query the session registry via SQL

   Use SHOW TEMPORARY FUNCTIONS LIKE (or SHOW USER FUNCTIONS LIKE) which resolves via the session’s function registry instead of the external catalog:

   Scala

   val exists =
     spark.sql(s"SHOW TEMPORARY FUNCTIONS LIKE `${functionName}`").count() > 0
   
   if (!exists) {
     spark.udf.register(functionName, udfImpl)
   }

   Python

   exists = spark.sql(f"SHOW TEMPORARY FUNCTIONS LIKE `{function_name}`").count() > 0
   if not exists:
       spark.udf.register(function_name, udf_impl)

3. SQL-only “create or replace”

   If your UDF is SQL-expressible, you can lean on SQL’s replace semantics:

   CREATE OR REPLACE TEMPORARY FUNCTION my_func AS 'com.example.udfs.MyFunc';

Notes and recommendations

• Persistent/catalog functions: Not supported in the OneLake external catalog in Runtime 1.3. Treat UDFs as session-scoped and register them at job/notebook start.
• Avoid spark.catalog.listFunctions() and .functionExists() until Fabric exposes a supported implementation for external catalogs.

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

[Padam_Prakash]

Hi @tayloramy ,
Thanks for the response. It aligns perfectly with what I was seeing.

What ultimately worked for me: 

I created a new Lakehouse, but this time left the “Lakehouse Schemas (Public Preview)” box unchecked.

This forced Fabric to use the classic OnelakeExternalCatalog path, where the function APIs are still implemented.
After that, my spark.catalog.functionExists() and UDF registration logic worked exactly as before.
Here is what i found written issue in their official docs
https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-schemas?#public-preview-limitati...

[v-tejrama]

Hi  @Padam_Prakash ,

Thank you for the update. As your issue has been resolved, please mark it as the solution so that other community members with the same problem can easily find the answer.

Best Regards,

Tejaswi.

Community Support

[tayloramy]

Hi @Padam_Prakash, 

Glad to hear the problem is resolved. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.