topic Re: clusterBy does not work in dataframe API? in Data Engineering

clusterBy does not work in dataframe API?

smpa01 — Thu, 10 Jul 2025 20:07:27 GMT

The following works in databricks but not in fabric. It works through DeltaTable API though. But why does it not work through dataframeAPI?

# write load at t - Create the table with clustering enabled from the start (df.write.format("delta") .mode("overwrite") .clusterBy("id") # <--- ADD THIS LINE HERE to enable clustering at creation .saveAsTable(table_name) # Use the fully qualified name here for consistency ) # AttributeError: 'DataFrameWriter' object has no attribute 'clusterBy'

DataFrameWriter Doc

Re: clusterBy does not work in dataframe API?

v-prasare — Fri, 11 Jul 2025 07:49:05 GMT

Hi smpa01 ,

the .clusterBy() method on DataFrameWriter is not supported because Fabric uses a customized Spark runtime that limits certain APIs to ensure simplicity and compatibility within its managed environment. Unlike Databricks, which offers extended Delta Lake features directly through the PySpark DataFrameWriter, Fabric restricts clustering capabilities to SQL DDL and the DeltaTable API

Thanks,

Prashanth Are

MS Fabric community support

Re: clusterBy does not work in dataframe API?

smpa01 — Wed, 16 Jul 2025 13:11:15 GMT

v-prasare without clusterBy in dataframe writer, I am guessing the clusterd files can't be written, if one intends to write only the raw files with the intention to create an external table.

df.write\ .format("delta")\ .mode("append")\ .clustrBy (cluster by fields)\ .save(file_path)