Overhead time / performance: what to expect

Hi nullabee

The notebook gap is most likely Spark session start, which the per-cell times don't include. A starter pool with no libraries or custom Spark properties normally starts in 5 to 10 seconds. Libraries from an environment add 30 seconds to 5 minutes. If the workspace uses private links or managed VNets, starter pools aren't supported and a session takes 2 to 5 minutes to start. Half a minute for 10 quick cells is the kind of gap that explains.

If one pipeline runs several notebooks, turn on high concurrency for pipelines under Workspace settings › Data Engineering/Science › Spark settings › High concurrency. Notebooks then share one session when they have the same user, default lakehouse, Spark settings and libraries, with up to 5 per session tag.

I can't see your setup, so I can't say anything on the pipeline to pipeline and SQL lookup gaps. Do your notebooks use an environment with custom libraries, or does the workspace have a managed VNet or private link?