topic Re: Fabric spark job definition parallel runs in Data Engineering

Fabric spark job definition parallel runs

yongshao — Fri, 24 May 2024 19:35:13 GMT

With Fabric notebooks, mssparkutils.notebook.runMultiple() could run them in parallel way

How to run the spark jobs defined by spark job defintion in a similar parallel way?

Re: Fabric spark job definition parallel runs

Anonymous — Mon, 27 May 2024 07:29:53 GMT

Hi @yongshao ,

Thanks for using Fabric Community.
As mentioned in docs - " Notebook utilities aren't applicable for Apache Spark job definitions (SJD). "

Unfortunately we cannot run spark jobs defined by spark job defintion in a parallel way.
At present the only way to run the Spark Job Definitions is -

1. Run a Spark job definition manually by selecting Run from the Spark job definition item in the job list.
2. Schedule a Spark job definition by setting up a schedule plan on the Settings tab. Select Settings on the toolbar, then select Schedule.

Docs to refer -
Run an Apache Spark job definition - Microsoft Fabric | Microsoft Learn
Microsoft Spark Utilities (MSSparkUtils) for Fabric - Microsoft Fabric | Microsoft Learn

We use customer feedback like yours to prioritize future features. The more users who request the ability to customize backgrounds, the higher it moves on our list.

Appreciate if you could share the feedback on our Microsoft Fabric Ideas. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

I hope this information helps. If you have any further queries please do let us know.

Thanks

Re: Fabric spark job definition parallel runs

Anonymous — Wed, 29 May 2024 05:48:59 GMT

Hi @yongshao ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .

Thanks

Re: Fabric spark job definition parallel runs

Anonymous — Thu, 30 May 2024 11:13:49 GMT

Re: Fabric spark job definition parallel runs

yongshao — Fri, 31 May 2024 14:53:55 GMT

Thank you for the answer, we'll stick with notebook

another question: from notebook, use PYODBC to connect a lakehouse's SQL endpoint, could a view be created or updated?

I tried it, the select query ran fine, but create view query ran a long time without any result

Re: Fabric spark job definition parallel runs

Anonymous — Mon, 03 Jun 2024 10:19:48 GMT

Hi @yongshao ,

Glad to know that you got some insights. Please continue using Fabric Community on your further queries.

Coming to your another question, I would like to understand why you are using pyodbc inorder to connect lakehouse?
It would be great if you can raise a new thread as it is a different question. After raising the query please tag me and share the url here, so I can look into it closely.

Re: Fabric spark job definition parallel runs

yongshao — Wed, 05 Jun 2024 19:57:04 GMT

We tried to connect lakehouse SQL endpoint from a notebook. By the way, we've no problem to use spark (dataFrame and SQL) with the lakehouse. But we want to create Views with the lakehouse SQL endpoint.

The views could be created manually from SQL endpoint query console, but we want to automate View creation/update from a Fabric notebook

Appreciated for your help

Re: Fabric spark job definition parallel runs

Anonymous — Thu, 06 Jun 2024 11:15:51 GMT

Hi @yongshao ,

Please refer this thread - Solved: Re: Creating views from spark notebook into SQL an... - Microsoft Fabric Community

Incase if you have further queries, please raise a new thread as it is a different question. After raising the query please tag me and share the url here, so I can look into it closely.

Thank you

Re: Fabric spark job definition parallel runs

yongshao — Thu, 06 Jun 2024 13:27:10 GMT

Regarding of spark job definition, its parallel runs would boost ETL performance and scalability, hope such feature would be supported in Fabric future release