Forum Discussion

smpa01's avatar
smpa01
Community Champion
1 year ago
Solved

clusterBy does not work in dataframe API?

The following works in databricks but not in fabric. It works through DeltaTable API though. But why does it not work through dataframeAPI? # write load at t - Create the table with clustering ena...
data engineering
Lakehouse
Notebook
  • v-prasare's avatar
    v-prasare
    1 year ago

    Hi smpa01 ,

    the .clusterBy() method on DataFrameWriter is not supported because Fabric uses a customized Spark runtime that limits certain APIs to ensure simplicity and compatibility within its managed environment. Unlike Databricks, which offers extended Delta Lake features directly through the PySpark DataFrameWriter, Fabric restricts clustering capabilities to SQL DDL and the DeltaTable API

     

     

    Thanks,

    Prashanth Are

    MS Fabric community support

v-prasare's avatar
v-prasare
Community Support
1 year ago

Hi smpa01 ,

the .clusterBy() method on DataFrameWriter is not supported because Fabric uses a customized Spark runtime that limits certain APIs to ensure simplicity and compatibility within its managed environment. Unlike Databricks, which offers extended Delta Lake features directly through the PySpark DataFrameWriter, Fabric restricts clustering capabilities to SQL DDL and the DeltaTable API

 

 

Thanks,

Prashanth Are

MS Fabric community support

  • smpa01's avatar
    smpa01
    Community Champion
    1 year ago

    v-prasare  without clusterBy in dataframe writer, I am guessing the clusterd files can't be written, if one intends to write only the raw files with the intention to create an external table.

     

    df.write\
    .format("delta")\
    .mode("append")\
    .clustrBy (cluster by fields)\
    .save(file_path)