Forum Discussion
clusterBy does not work in dataframe API?
The following works in databricks but not in fabric. It works through DeltaTable API though. But why does it not work through dataframeAPI?
# write load at t - Create the table with clustering enabled from the start
(df.write.format("delta")
.mode("overwrite")
.clusterBy("id") # <--- ADD THIS LINE HERE to enable clustering at creation
.saveAsTable(table_name) # Use the fully qualified name here for consistency
)
# AttributeError: 'DataFrameWriter' object has no attribute 'clusterBy'DataFrameWriter Doc
Hi smpa01 ,
the .clusterBy() method on DataFrameWriter is not supported because Fabric uses a customized Spark runtime that limits certain APIs to ensure simplicity and compatibility within its managed environment. Unlike Databricks, which offers extended Delta Lake features directly through the PySpark DataFrameWriter, Fabric restricts clustering capabilities to SQL DDL and the DeltaTable API
Thanks,
Prashanth Are
MS Fabric community support
2 Replies
Hi smpa01 ,
the .clusterBy() method on DataFrameWriter is not supported because Fabric uses a customized Spark runtime that limits certain APIs to ensure simplicity and compatibility within its managed environment. Unlike Databricks, which offers extended Delta Lake features directly through the PySpark DataFrameWriter, Fabric restricts clustering capabilities to SQL DDL and the DeltaTable API
Thanks,
Prashanth Are
MS Fabric community support
