Performance issues running pipelines and notebooks

I have the same problem, too. Running the notebook directly to read and transform some data takes, including the startup of the session, around 30 seconds.

Executing the same notebook from a data pipeline takes between 22 and 26 minutes. I have no explanation so far. We only have a small fabric capacity, but running the notebook stand alone or from within the pipeline should take about the same amount of time, imo.

EDIT: This may not be for every use case that has this problem. In my pipeline that took so long, I executed the Notebooks from a ForEach loop. When I reduced the Batch Count parameter to 2, it went much faster. Still not as fast as running the Notebook directly, but each execution of the notebook took between 30 seconds and 7 minutes. Now it is getting to the usable range. Before I had no value for the batch count, thus 3 Notebooks ran in parallel.