Which is Spark version supported by Fabric

[pmscorca]

Hi,

I'd like to know which Spark version is supported by Fabric.

Any suggests to me? Thanks

[RohtashvGehllot]

If you try to create a table in Lakehouse, you may encounter an error 'Please ensure that the workspace is configured with Spark version <= 3.4 (i.e. Fabric runtime version <= 1.2)'

The Spark version compatible with Fabric should be <=3.4 (Fabric runtime version <= 1.2).

To check and modify the Spark version, follow the below steps:

- Go to your Workspace

- Workspace Settings (top right)

- Expand 'Data Engineering/Science' from left pane

- Select 'Spark Settings'

- Go To 'Environments'

- Check 'Runtime Version'

- From the dropdown, select the Runtime Version 1.2

- Save

- Try creating the table again and it should work

[v-huijiey-msft]

Hi @pmscorca ,

Thanks for the reply from frithjof_v .

By default, all new workspaces use the latest runtime version, currently Runtime 1.2.

For more information on Runtime 1.2, please see:

Runtime 1.2 in Fabric - Microsoft Fabric | Microsoft Learn

Deep Dive into Apache Spark-based Runtime 1.2 Available in the Fabric.

Take a deeper dive into Apache Spark-based Runtime 1.2 in the fabric and learn about its unique features, functionality, and best practices.

Microsoft Fabric Runtime 1.2 is the latest version of the GA runtime.The key components of Runtime 1.2 include

• Apache Spark 3.4.1
• Operating System: Mariner 2.0
• Java: 11
• Scala: 2.12.17
• Python 3.10
• Delta Lake: 2.4.0
• R: 4.2.2
  
  

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

[pmscorca]

Hi, I'm new with Spark and I've appreciated your help, thanks.

I'd like to know if it exists a set of pyspark templates for Fabric to have a look. In particular, I'm searching a template to create or define a table using pyspark, a table with column having a specific data type and a specific dimension (f.e. char/varchar(20)) as possible with a T-SQL create table statement.

It should be useful to have a set of pyspark templates that function in Fabric successfully.

Thanks

[frithjof_v · Accepted solution]

I never tried the template snippets, but maybe they can be helpful:

https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#code-snippets

Here are some learning materials and exercise for working with PySpark, Dataframes, Tables:

https://learn.microsoft.com/en-us/training/modules/use-apache-spark-work-files-lakehouse/

https://learn.microsoft.com/en-us/training/modules/work-delta-lake-tables-fabric/

Personally, ChatGPT has helped me solve some code questions regarding PySpark, SparkSQL, etc.

And also Googling for PySpark / Delta Lake questions can provide answers. Many Databricks related PySpark / Delta Lake questions and answers can be relevant also for Fabric.

[v-huijiey-msft]

Hi @pmscorca ,

Thanks a lot @frithjof_v for your attention to this case, the links provided were very helpful.

I would like to add that you can view the current Spark version by opening the notebook and selecting "Workspace default", in my case you can see that Spark version 3.4 is the supported version.

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

[frithjof_v]

https://learn.microsoft.com/en-us/fabric/data-engineering/runtime