Incompatibility between synapseutils_2.12:1.6.2 (Scala) and Fabric notebookutils.lakehouse.list

Hi s_hampe ,

Great investigation, your reflection-based workaround is solid, and I can confirm this is a known limitation rather than a simple bug.

Root Cause: NotebookUtils is not supported in Spark Job Definitions

The official Microsoft documentation for NotebookUtils explicitly states:

"Notebook utilities aren't applicable for Apache Spark job definitions (SJD)."

This means the notebookutils.lakehouse.list() method (and other notebookutils APIs) are designed to run inside Fabric Notebooks only, not in compiled Spark Job Definitions. The synapseutils_2.12:1.6.2 library provides compilation stubs, but these stubs are intentionally incomplete, they were never meant to fully replicate the runtime API available inside Notebooks.

That's why the method signature mismatch exists:

• Runtime (Notebook): list(workspaceId: String = "", maxResults: Int = 1000)
• Stubs (synapseutils JAR): list(workspaceId: String) missing the maxResults parameter
  Your Workaround is the Right Approach

Your Java reflection approach is actually the correct pattern for this scenario. Since the stubs don't match the runtime, reflection bypasses the compilation check and calls the method directly at runtime.

Alternative: Use the Fabric REST API

For Spark Job Definitions, the recommended approach is to use the Fabric REST API instead of notebookutils:

The Lakehouse Management REST API provides full CRUD operations and works reliably in both Notebooks and Spark Job Definitions.

Suggestion to Microsoft

It would be helpful if the synapseutils library either:

1. Matched the full runtime signatures (even for SJD stubs)
2. Or threw a clear UnsupportedOperationException with a message like "notebookutils.lakehouse is not supported in Spark Job Definitions"

This would save developers significant debugging time.

Hope this helps! Feel free to ask if you have follow-up questions.