Forum Discussion
Data pipeline is remaning into InProgress state
Hello ankitmori91,
The F4 capacity alone does not tell us why the pipeline is taking 15–25 minutes.First check the pipeline run and open the Notebook activity details. Fabric shows the activity duration and performance details in the pipeline run. Check those first to see where the time is being spent.
Try running the Notebook on its own and see how long it takes compared with the pipeline run.
If the Notebook itself takes 15–25 minutes, investigate the Notebook execution. If it completes much faster but the pipeline remains InProgress, the pipeline activity details should help narrow down the cause.
Microsoft Learn: Monitor pipeline runs in Fabric Data Factory
Solution SageHi ankitmori91,
I would also check the Spark side of the run before assuming the F4 itself should be sufficient.
Fabric Spark concurrency depends on the Spark resources available on the capacity at that moment, and notebook jobs submitted through pipelines can queue when those resources are already in use. The newer job concurrency and queue monitoring view under Workspace settings > Data Engineering/Science > Spark settings > Jobs can help confirm whether the notebook is actually waiting for compute.
If it isn't queued, I would also check Session start insights. That can show whether the delay is happening while Fabric acquires the Spark session, for example because it couldn't use the starter pool, rather than inside your notebook code itself.
Between that and the pipeline activity details Olufemi7 mentioned, you should be able to separate pipeline overhead, Spark startup/queueing and actual notebook execution before changing the capacity size.