Forum Discussion

Element115's avatar
Element115
Memorable Member
1 year ago
Solved

QUESTION::Databricks delta files Spark SQL DML not supported in Fabric

When will the following DML be supported in a Fabric Spark SQL Notebook?   ALTER TABLE TABLE_NAME SET TBLPROPERTIES ( delta.targetFileSize = '128MB', delta.tuneFileSizesForRewrites = t...
notebook
  • Element115's avatar
    Element115
    1 year ago

    Ok, but then, if this default behavior you mention is active,  why do I see NO snappy file show up in the folders, and the total number of small files is not reduced if I do not run the following commands? 

     

    And lately, also small files is not reduced after I run the following commands from a Notebook? (the below sequence is the same cell sequence as in the Notebook):

     

     

    spark.conf.set("spark.databricks.delta.optimize.maxFileSize", "10GB")
spark.conf.set("spark.databricks.delta.optimizeWrite.enabled", "true")
spark.conf.set("spark.databricks.delta.optimizeWrite.binSize", "5GB")

     

    %%sql

OPTIMIZE RAW_SUMMARY ZORDER BY ([YEAR], [MONTH]);
OPTIMIZE RAW_TRANSACTION ZORDER BY ([YEAR], [MONTH], Direction, VehicleClass, Lane);

OPTIMIZE FACT_SUMMARY ZORDER BY ([YEAR], [MONTH], [DATE]);
OPTIMIZE FACT_TRANSACTION ZORDER BY ([YEAR], [MONTH], [DATE], [HOUR]);

     

    spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false")

     

    %%sql

VACUUM RAW_SUMMARY RETAIN 0 HOURS;
VACUUM RAW_TRANSACTION RETAIN 0 HOURS;

VACUUM FACT_SUMMARY RETAIN 0 HOURS;
VACUUM FACT_TRANSACTION RETAIN 0 HOURS;