topic Spark Dataset API error in Data Engineering

Spark Dataset API error

smpa01 — Mon, 20 Nov 2023 19:33:55 GMT

I am getting the following error with dataset API but not with dataframe API in notebook.

// works val df= Seq( ("Boston", "USA", 0.67), ("Dubai", "UAE", 3.1), ("Cordoba", "Argentina", 1.39) ).toDF("city", "country", "population") df.show() //does not work case class City(englishName: String, continent: String) val cities = Seq( City("bejing", "asia"), City("new york", "north america"), City("paris", "europe") ).toDS() cities.show() // does not work case class City(englishName: String, continent: String) val cities2 = spark.createDataset( Seq( City("bejing", "asia"), City("new york", "north america"), City("paris", "europe") ) )

Re: Spark Dataset API error

v-cboorla-msft — Tue, 21 Nov 2023 12:12:46 GMT

Hi @smpa01

Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this.
We will update you once we hear back from them.
Thanks

Re: Spark Dataset API error

v-cboorla-msft — Mon, 18 Dec 2023 16:43:57 GMT

Hi @smpa01

Apologies for the delay in response.

Define case class in a separate jar. Example:

1) create a jar file from .scala (ensure same scala version while compiling, see Fabric Spark runtimes (Apache Spark runtime in Fabric - Microsoft Fabric | Microsoft Learn)

package com.example

case class City(englishName: String, continent: String)

2) e.g. you can upload the jar from notebook resources (How to use notebooks - Microsoft Fabric | Microsoft Learn) and then load it: %load_new_custom_jar {mssparkutils.nbResPath}/builtin/City.jar

3) import: import com.example.City

I hope this helps. If you have any further questions please do let us know.

Re: Spark Dataset API error

smpa01 — Wed, 20 Dec 2023 16:40:52 GMT

@v-cboorla-msft i will try and let you know.

Re: Spark Dataset API error

v-cboorla-msft — Tue, 26 Dec 2023 07:16:13 GMT

Hi @smpa01

We haven’t heard from you on the last response and was just checking back to see, did you got a chance to look into this?

In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks

Re: Spark Dataset API error

v-cboorla-msft — Tue, 02 Jan 2024 07:17:05 GMT

Hi @smpa01

We haven’t heard from you on the last response and was just checking back to see, did you got a chance to look into this?

In case if you have any resolution please do share that same with the community as it can be helpful to others.
If you have any question relating to the current thread, please do let us know and we will try out best to help you.
In case if you have any other question on a different issue, we request you to open a new thread.

Thanks