Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified as a Fabric Data Engineer: Check your eligibility for a 50% exam voucher offer and join us for free live learning sessions to get prepared for Exam DP-700. Get started

Reply
MichBDO
Frequent Visitor

Error message when trying to run mssparkutils.lakehouse.get("Lakehouse name")

We are using a CICD pipeline to deploy our notebooks in workspaces.

In the DEV-TEST-ACC-PRD workspaces the same lakehouse can be found with the same name 'FA_BRONZE'. We now need to write the notebooks dynamically so that in every workspace, that correct lakehouse is used when writing tables away. For that we need to get the lakehouse ID so that we can use it with a variable in the abfss:// path. 

This post: https://fabric.guru/programmatically-creating-managing-lakehouses-in-fabric. (screenshot) references a function mssparkutils.lakehouse.get() to get the ID of a lakehouse when you provide the name of that lakehouse. But for us unfortunatly this gives an error (see below).

Does any of you have any experience with using this function or another solution to our problem?

MichBDO_0-1702544736011.png

 

Py4JJavaError: An error occurred while calling z:mssparkutils.lakehouse.get. : org.json4s.MappingException: Do not know how to convert JArray(List(JString(Lakehouse), JString(Lakehouse), JString(Lakehouse), JString(Lakehouse), JString(Lakehouse), JString(Lakehouse), JString(Lakehouse), JString(Lakehouse))) into class java.lang.String at org.json4s.reflect.package$.fail(package.scala:53) at org.json4s.Extraction$.convert(Extraction.scala:888) at org.json4s.Extraction$.$anonfun$extract$10(Extraction.scala:456) at org.json4s.Extraction$.$anonfun$customOrElse$1(Extraction.scala:780) at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) at scala.PartialFunction$$anon$1.applyOrElse(PartialFunction.scala:257) at org.json4s.Extraction$.customOrElse(Extraction.scala:780) at org.json4s.Extraction$.extract(Extraction.scala:454) at org.json4s.Extraction$.extract(Extraction.scala:56) at org.json4s.ExtractableJsonAstNode.extract(ExtractableJsonAstNode.scala:22) at com.microsoft.spark.notebook.msutils.impl.artifact.Artifact$.parseItem(Artifact.scala:43) at com.microsoft.spark.notebook.msutils.impl.artifact.Artifact$.$anonfun$parseJsons$1(Artifact.scala:56) at scala.collection.immutable.List.flatMap(List.scala:366) at com.microsoft.spark.notebook.msutils.impl.artifact.Artifact$.parseJsons(Artifact.scala:55) at com.microsoft.spark.notebook.msutils.impl.artifact.MSArtifactUtilsImpl.list(MSArtifactUtilsImpl.scala:173) at com.microsoft.spark.notebook.msutils.impl.artifact.MSArtifactUtilsImpl.get(MSArtifactUtilsImpl.scala:83) at mssparkutils.lakehouse$.get(lakehouse.scala:16) at mssparkutils.lakehouse.get(lakehouse.scala) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.base/java.lang.Thread.run(Thread.java:829)

4 REPLIES 4
MichBDO
Frequent Visitor

Hi, the workspace was already in runtime 1.2.

Hi @MichBDO 
Apologies for the delay in response. Are you still facing this issue?
If yes we will revert back and try to help you.

v-nikhilan-msft
Community Support
Community Support

Hi @MichBDO 
Thanks for using Fabric Community.
The mssparkutils.lakehouse feature is not documented yet officially. Hence we do not know when the feature will be deployed.
As a workaround you can leverage the Fabric REST Item APIs to get the lakehouse ID. Please refer to this link for more information: Link1

Hope this helps. Please let me know if you have any further questions.

Hi @MichBDO 
There is a known issue in runtime1.1. So could you please change to runtime1.2 (spark3.4) version? 
Go to workspace settings and change the spark environment.

vnikhilanmsft_0-1702902114772.png

Now try to run the code in the notebook.

vnikhilanmsft_0-1702902530831.png

 

Hope this helps. Please let me know if you have any further questions.

Helpful resources

Announcements
Feb2025 Sticker Challenge

Join our Community Sticker Challenge 2025

If you love stickers, then you will definitely want to check out our Community Sticker Challenge!

JanFabricDE_carousel

Fabric Monthly Update - January 2025

Explore the power of Python Notebooks in Fabric!

JanFabricDW_carousel

Fabric Monthly Update - January 2025

Unlock the latest Fabric Data Warehouse upgrades!