Faster Way to Copy Dataframes to Lakehouse?

Hi russellhq 

Not sure if this is the best approach but one idea that comes to my mind since I used the function in a different context:

With Microsoft's mssparkutils package you can run notebooks out of another notebook (like an orchestrator) with the runMultiple function (https://learn.microsoft.com/en-us/fabric/data-engineering/microsoft-spark-utilities#reference-run-multiple-notebooks-in-parallel). The advantage from my point of view is that this function executes the called notebooks in parallel (with respect to available resources) and I assume that this might help you with your problem since the sequential execution via a for loop might be the problem here. 

So what would you have to do: 

• Rewrite your notebook to facilitate parameters (https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#designate-a-parameters-cell) so that it copies just one table that is defined as a parameter in the notebook.
• Write another "orchstrator" notebook in which you automatically generate a DAG as dictionary with your 1700 tables each represented by a call to the first notebook. And the run runMultiple with this DAG. 

As I said, not sure if this is the easiest possibility but I think this at least might be one since the compolexity seems to come from sequential execution and not the size of copied tables itself. Keep in mind that this solution uses the spark session from the "outer" (orchestration) notebook so you should also get a point to scale the performance via the settings of the used spark cluster. 

Since I used runMultiple() in a different context I have not tested this or had the problem myself and so I would be happy to hear from you if you tested the solution and if it worked for you. 🙂

BR 
Martin