Forum Discussion
Storage increasing daily
You should use Python notebooks. It is so easy top use bronze --> silver --> gold layer. You should use OPTIMIZE commands for indexing and Vaccum commands as well. That way You can apply VACUUM and you can save money a lot.
Super UserHi shirishmathanka,
To add to what BhaveshPatel said, when you change a table in a lakehouse/warehouse, the old data is not removed, it is just unreferenced.
So if you only change 3 rows, the old parquet file that had all the rows still exists, and a new parquet file with the modifications is created.
This is useful for the time travel feature.
using the VACUUM command will delete unreferenced files, which means that you can no longer time travel back to them, but will free up storage space.
hi tayloramy and BhaveshPatel thanks for the response,
i am running vaccum once a day with 0 rettention and my pipeline is running 12 time a day in every 2 hours