For low code users, it would be awesome if there was a Data Pipeline activity that can be used to run Optimize on a table in a Lakehouse (or all tables in a Lakehouse).
For example, when using ...
frithjof_v
Community Champion
Community ChampionIt could also be beneficial if this mechanism can have AutoCompaction features, so that we can set a threshold to only execute the compaction if the number of small files > x, where x is a threshold value we can adjust, similar to the minNumFiles setting in AutoCompaction https://docs.delta.io/3.1.0/optimizations-oss.html#auto-compaction
