Forum Discussion
QUESTION::Databricks delta files Spark SQL DML not supported in Fabric
- 1 year ago
Ok, but then, if this default behavior you mention is active, why do I see NO snappy file show up in the folders, and the total number of small files is not reduced if I do not run the following commands?
And lately, also small files is not reduced after I run the following commands from a Notebook? (the below sequence is the same cell sequence as in the Notebook):
spark.conf.set("spark.databricks.delta.optimize.maxFileSize", "10GB") spark.conf.set("spark.databricks.delta.optimizeWrite.enabled", "true") spark.conf.set("spark.databricks.delta.optimizeWrite.binSize", "5GB")%%sql OPTIMIZE RAW_SUMMARY ZORDER BY ([YEAR], [MONTH]); OPTIMIZE RAW_TRANSACTION ZORDER BY ([YEAR], [MONTH], Direction, VehicleClass, Lane); OPTIMIZE FACT_SUMMARY ZORDER BY ([YEAR], [MONTH], [DATE]); OPTIMIZE FACT_TRANSACTION ZORDER BY ([YEAR], [MONTH], [DATE], [HOUR]);spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false")%%sql VACUUM RAW_SUMMARY RETAIN 0 HOURS; VACUUM RAW_TRANSACTION RETAIN 0 HOURS; VACUUM FACT_SUMMARY RETAIN 0 HOURS; VACUUM FACT_TRANSACTION RETAIN 0 HOURS;
Hi Element115
no official documentation or roadmap around when it will be supported.
as you have mentioned spark.conf.set is the only workaround as of now.
But if we look deeper ,Fabric uses its own configurations for similar optimizations:
• Optimized Write: Enabled by default to consolidate files during writes.
• V-Order: Automatically applied for compression and read performance.