Spark Dataset API error

[smpa01]

I am getting the following error with dataset API but not with dataframe API in notebook.

// works
val df= Seq(
    ("Boston", "USA", 0.67),
    ("Dubai", "UAE", 3.1),
    ("Cordoba", "Argentina", 1.39)    
).toDF("city", "country", "population")

df.show()

//does not work
case class City(englishName: String, continent: String)

val cities = Seq(
  City("bejing", "asia"),
  City("new york", "north america"),
  City("paris", "europe")
).toDS()

cities.show()

// does not work
case class City(englishName: String, continent: String)
val cities2 = spark.createDataset(
  Seq(
    City("bejing", "asia"),
    City("new york", "north america"),
    City("paris", "europe")
  )
)

[v-cboorla-msft]

Hi @smpa01 

Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this.
We will update you once we hear back from them.
Thanks

[v-cboorla-msft]

Hi @smpa01 

Apologies for the delay in response.

Define case class in a separate jar. Example:

1) create a jar file from .scala (ensure same scala version while compiling, see Fabric Spark runtimes (Apache Spark runtime in Fabric - Microsoft Fabric | Microsoft Learn)

package com.example

case class City(englishName: String, continent: String)

2) e.g. you can upload the jar from notebook resources (How to use notebooks - Microsoft Fabric | Microsoft Learn) and then load it: %load_new_custom_jar {mssparkutils.nbResPath}/builtin/City.jar

3) import: import com.example.City

I hope this helps. If you have any further questions please do let us know.

[smpa01]

@v-cboorla-msft  i will try and let you know.

[v-cboorla-msft]

Hi @smpa01 

We haven’t heard from you on the last response and was just checking back to see, did you got a chance to look into this?

In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks

[v-cboorla-msft]

Hi @smpa01 

We haven’t heard from you on the last response and was just checking back to see, did you got a chance to look into this?

In case if you have any resolution please do share that same with the community as it can be helpful to others.
If you have any question relating to the current thread, please do let us know and we will try out best to help you.
In case if you have any other question on a different issue, we request you to open a new thread.

Thanks