topic Fabric Executor Running Time too long in Pipelines

Fabric Executor Running Time too long

D4zk — Fri, 08 Nov 2024 07:55:26 GMT

Hi all,

I'm using F4 and run a pipeline that calls some child Fabric notebooks to load data. I'm facing with the execution is too long 7-8 mins. While In my notebook, all cells take total execution time only 15s for 6000 rows (read dataframe, write to next storage).

And when I checked the monitor screen, It took over 7 mins with Running stage from 5:57 to 5:59 - 2 mins only.

I already configureed high concurrency and try to test with some spark configurations such as spark.ms.autotune.enabled.

How can I resolve this?

Re: Fabric Executor Running Time too long

Anonymous — Mon, 11 Nov 2024 02:25:08 GMT

Hi D4zk ,

It looks like your pipeline is taking much longer to execute than a single notebook unit.

Are there any other concurrent interactions during the time you are running the pipeline? Make sure these dependencies are not causing delays. Network latency or slow response from external systems can affect the overall execution time.

Alternatively, refer to the following document for a description of the detailed metrics of pipeline operation.

How to monitor pipeline runs - Microsoft Fabric | Microsoft Learn

Best Regards,
Adamk Kong

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Fabric Executor Running Time too long

D4zk — Wed, 20 Nov 2024 06:44:19 GMT

Hi, I already find an approach to help reduce the time to start, queue and idle of node, that is using DAG to call all notebooks from 1 main notebook. It can help me reduce from 2 hours to 37 minutes for 37 tables.

Thanks for you response.

Re: Fabric Executor Running Time too long

Srisakthi — Wed, 20 Nov 2024 12:57:47 GMT

Hi D4zk ,

Are you using parallelisation in DAG. I guess if you do parallel execution of all child notebook , this can further save execution time.

Regards,

Srisakthi