Forum Discussion
Helper ISaving small delta table via notebook extremely slow
Hi ntimmerman,
In fact, I think this cost of processing almost upper to limit and changing to use other functions may not obviously improve the performance when you operate with small tables.
For optimized on the fabric notebook performance, you can take a look at the following document if it helps:Delta Lake table optimization and V-Order - Microsoft Fabric | Microsoft Learn
Regards,
Xiaoxin Sheng
Helper IHi Anonymous , thanks for your quick reply! That's greatly appreciated! Can you elaborate a bit on what other functions are available which would have better performance for those small tables?
Thanks
Hi ntimmerman,
In fact, I think this cost of processing almost upper to limit and changing to use other functions may not obviously improve the performance when you operate with small tables.
For optimized on the fabric notebook performance, you can take a look at the following document if it helps:
Delta Lake table optimization and V-Order - Microsoft Fabric | Microsoft Learn
Regards,
Xiaoxin Sheng
- ntimmerman
Thank you!
I found out that there is hardly any difference between writing 10 rows or writing 10.000 rows.
It seems like there's just a lot of overhead in the start of the writing process.
So, I've optimized the process now to combine tables before dumping them in the bronze layer.