topic Re: Triggering multiple pipelines at the same time in Data Engineering

Triggering multiple pipelines at the same time

Anonymous — Mon, 03 Jun 2024 12:41:47 GMT

Hi,

In Azure synapse analytics I can trigger a pipeline by just pressing trigger now.

But in fabric pressing trigger opens set alert box.

Is this something not supported in fabric at the moment or it is located somewhere I don't know ? 😁
My usecase is running the pipelines simulatenously for couple of times to see how spark notebooks perform. I'm interested to see if they get queued or it can run them in paralell just fine

Thanks

Re: Triggering multiple pipelines at the same time

AndyDDC — Mon, 03 Jun 2024 13:17:15 GMT

Hi @Anonymous you should just be able to press the "Run" button to run the pipeline adhoc.

Re: Triggering multiple pipelines at the same time

Anonymous — Mon, 03 Jun 2024 13:24:13 GMT

I can run the pipeline with run button, But I need to run same pipeline (differnet parameters) while the one I triggered is running
after pressing Run I only have the option to cancel the run

Run seems to be similar to debug runs of synapse. Hence I was looking to press trigger to run the same pipeline with differnet parameters

Re: Triggering multiple pipelines at the same time

Expiscornovus — Mon, 03 Jun 2024 14:45:42 GMT

Hi @Anonymous,

You might be able to solve this via HTTP requests and the API. Have a look at this blog which demonstrates a POST request to trigger a pipeline run, https://www.thatbluecloud.com/triggering-synapse-pipelines-from-microsoft-fabric-via-rest-apis/

You could for example trigger these requests at the same time in a Power Automate cloud flow, that product supports parallel branches.

Re: Triggering multiple pipelines at the same time

Anonymous — Tue, 04 Jun 2024 17:33:45 GMT

Hi @Anonymous
We haven’t heard from you on the last response and was just checking back to see if your query has been resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks

Re: Triggering multiple pipelines at the same time

jwinchell40 — Wed, 05 Jun 2024 02:17:19 GMT

@Anonymous - So you just want to be able to manually run the same pipeline but with different parameters manually; not automated.

If you open the dataflow in 2 different browser sessions, does it let you start the 2nd run in the other browser session?

Re: Triggering multiple pipelines at the same time

Anonymous — Wed, 05 Jun 2024 10:30:35 GMT

Actually this worked 😁. Two different tabs made it possible to run same pipeline twice