NotImplementedError in OnelakeExternalCatalog.functionExists / listFunctions in Fabric Runtime 1.3

Hi Padam_Prakash,

This error means you’re calling a “functions API” on the external catalog (OneLake) that doesn’t implement function metadata in Fabric Runtime 1.3. In 1.3, OnelakeExternalCatalog.listFunctions and functionExists throw scala.NotImplementedError, so checks like spark.catalog.functionExists(...) now fail even before you try to (re)register your UDF. It’s expected in this runtime, though the change feels like a regression from earlier behavior.

Why it happens

• spark.catalog.functionExists and spark.catalog.listFunctions() route to the active catalog. In Fabric Lakehouses that’s the OneLake external catalog, whose function APIs aren’t implemented in Runtime 1.3. Hence:

  com.microsoft.fabric.spark.catalog.OnelakeExternalCatalog.listFunctions → NotImplementedError

• Your UDFs are typically session-scoped (temporary) and live in Spark’s function registry, not in the external catalog. So even conceptually, the external catalog isn’t the right place to ask about them.

Practical workarounds

1. Skip the existence check and (re)register

   In both Scala and PySpark, registering the same name overwrites the previous temp function. So you can safely register every run.

   Scala

   // Replace “functionExists + conditional” with unconditional register
   spark.udf.register(functionName, udfImpl)  // acts like “create or replace” for temp UDFs

   Python

   spark.udf.register(function_name, udf_impl)  # overwrites same-named temp UDF

2. If you must check first, query the session registry via SQL

   Use SHOW TEMPORARY FUNCTIONS LIKE (or SHOW USER FUNCTIONS LIKE) which resolves via the session’s function registry instead of the external catalog:

   Scala

   val exists =
     spark.sql(s"SHOW TEMPORARY FUNCTIONS LIKE `${functionName}`").count() > 0
   
   if (!exists) {
     spark.udf.register(functionName, udfImpl)
   }

   Python

   exists = spark.sql(f"SHOW TEMPORARY FUNCTIONS LIKE `{function_name}`").count() > 0
   if not exists:
       spark.udf.register(function_name, udf_impl)

3. SQL-only “create or replace”

   If your UDF is SQL-expressible, you can lean on SQL’s replace semantics:

   CREATE OR REPLACE TEMPORARY FUNCTION my_func AS 'com.example.udfs.MyFunc';

Notes and recommendations

• Persistent/catalog functions: Not supported in the OneLake external catalog in Runtime 1.3. Treat UDFs as session-scoped and register them at job/notebook start.
• Avoid spark.catalog.listFunctions() and .functionExists() until Fabric exposes a supported implementation for external catalogs.

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.