Audit logs in Notebooks

Hi Poojitha_P,

Thank you Mauro89, for your insights,

Frequent small DML operations from Spark notebooks, such as audit-log inserts or updates, can result in many small files within Delta tables. This leads to increased task and metadata overhead, which can impact performance. According to Microsoft documentation, enabling Optimize Write helps by performing pre-write compaction, making it useful for tables with frequent small inserts or UPDATE and MERGE operations. Additionally, Auto-compaction can automatically compact fragmented files after writes, which helps maintain table performance when there are frequent small writes.

Tune File Size - Microsoft Fabric | Microsoft Learn

Cross-Workload Table Maintenance and Optimization - Microsoft Fabric | Microsoft Learn

Thank you.