Forum Discussion
QUESTION::Databricks delta files Spark SQL DML not supported in Fabric
- 1 year ago
Ok, but then, if this default behavior you mention is active, why do I see NO snappy file show up in the folders, and the total number of small files is not reduced if I do not run the following commands?
And lately, also small files is not reduced after I run the following commands from a Notebook? (the below sequence is the same cell sequence as in the Notebook):
spark.conf.set("spark.databricks.delta.optimize.maxFileSize", "10GB") spark.conf.set("spark.databricks.delta.optimizeWrite.enabled", "true") spark.conf.set("spark.databricks.delta.optimizeWrite.binSize", "5GB")%%sql OPTIMIZE RAW_SUMMARY ZORDER BY ([YEAR], [MONTH]); OPTIMIZE RAW_TRANSACTION ZORDER BY ([YEAR], [MONTH], Direction, VehicleClass, Lane); OPTIMIZE FACT_SUMMARY ZORDER BY ([YEAR], [MONTH], [DATE]); OPTIMIZE FACT_TRANSACTION ZORDER BY ([YEAR], [MONTH], [DATE], [HOUR]);spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false")%%sql VACUUM RAW_SUMMARY RETAIN 0 HOURS; VACUUM RAW_TRANSACTION RETAIN 0 HOURS; VACUUM FACT_SUMMARY RETAIN 0 HOURS; VACUUM FACT_TRANSACTION RETAIN 0 HOURS;
Upon plunging more deeply into the documentation, and with the help of AI, I adjusted my max and bin sizes down close to the upper bound for the size of the source files. And relaunched the process at EOD, once a day. Now it works. There was a UI glitch whereby it would not auto-refresh the page showing the content of folders and as a result it was showing the previous folder content without the snappy files. Force refresh, et voilà! All snappy files are there.
Hi Element115,
Thank you for your detailed follow-up and for sharing your findings. We're glad to hear the issue was resolved and that the optimization steps worked as expected.
Please go ahead and mark this as the accepted solution if it addressed your query.
It helps other users who are searching for this same information and find the information.
Thank you.