topic Re: Which is Spark version supported by Fabric in Data Engineering

Which is Spark version supported by Fabric

pmscorca — Tue, 02 Jul 2024 19:56:59 GMT

Hi,

I'd like to know which Spark version is supported by Fabric.

Any suggests to me? Thanks

Re: Which is Spark version supported by Fabric

frithjof_v — Tue, 02 Jul 2024 22:52:07 GMT

https://learn.microsoft.com/en-us/fabric/data-engineering/runtime

Re: Which is Spark version supported by Fabric

Anonymous — Wed, 03 Jul 2024 01:26:44 GMT

Hi @pmscorca ,

Thanks a lot @frithjof_v for your attention to this case, the links provided were very helpful.

I would like to add that you can view the current Spark version by opening the notebook and selecting "Workspace default", in my case you can see that Spark version 3.4 is the supported version.

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

Re: Which is Spark version supported by Fabric

pmscorca — Wed, 03 Jul 2024 05:22:09 GMT

Hi, I'm new with Spark and I've appreciated your help, thanks.

I'd like to know if it exists a set of pyspark templates for Fabric to have a look. In particular, I'm searching a template to create or define a table using pyspark, a table with column having a specific data type and a specific dimension (f.e. char/varchar(20)) as possible with a T-SQL create table statement.

It should be useful to have a set of pyspark templates that function in Fabric successfully.

Thanks

Re: Which is Spark version supported by Fabric

frithjof_v — Wed, 03 Jul 2024 16:30:28 GMT

I never tried the template snippets, but maybe they can be helpful:

https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#code-snippets

Here are some learning materials and exercise for working with PySpark, Dataframes, Tables:

https://learn.microsoft.com/en-us/training/modules/use-apache-spark-work-files-lakehouse/

https://learn.microsoft.com/en-us/training/modules/work-delta-lake-tables-fabric/

Personally, ChatGPT has helped me solve some code questions regarding PySpark, SparkSQL, etc.

And also Googling for PySpark / Delta Lake questions can provide answers. Many Databricks related PySpark / Delta Lake questions and answers can be relevant also for Fabric.

Re: Which is Spark version supported by Fabric

Anonymous — Thu, 04 Jul 2024 02:48:43 GMT

Hi @pmscorca ,

Thanks for the reply from frithjof_v .

By default, all new workspaces use the latest runtime version, currently Runtime 1.2.

For more information on Runtime 1.2, please see:

Runtime 1.2 in Fabric - Microsoft Fabric | Microsoft Learn

Deep Dive into Apache Spark-based Runtime 1.2 Available in the Fabric.

Take a deeper dive into Apache Spark-based Runtime 1.2 in the fabric and learn about its unique features, functionality, and best practices.

Microsoft Fabric Runtime 1.2 is the latest version of the GA runtime.The key components of Runtime 1.2 include

Apache Spark 3.4.1
Operating System: Mariner 2.0
Java: 11
Scala: 2.12.17
Python 3.10
Delta Lake: 2.4.0
R: 4.2.2

Best Regards,
Yang
Community Support Team

Re: Which is Spark version supported by Fabric

RohtashvGehllot — Tue, 24 Sep 2024 11:30:54 GMT

If you try to create a table in Lakehouse, you may encounter an error 'Please ensure that the workspace is configured with Spark version <= 3.4 (i.e. Fabric runtime version <= 1.2)'

The Spark version compatible with Fabric should be <=3.4 (Fabric runtime version <= 1.2).

To check and modify the Spark version, follow the below steps:

- Go to your Workspace

- Workspace Settings (top right)

- Expand 'Data Engineering/Science' from left pane

- Select 'Spark Settings'

- Go To 'Environments'

- Check 'Runtime Version'

- From the dropdown, select the Runtime Version 1.2

- Save

- Try creating the table again and it should work