topic Converting Fabric dataframe to spark dataframe in Data Engineering

Converting Fabric dataframe to spark dataframe

smpa01 — Sun, 15 Sep 2024 01:22:29 GMT

How can I convert a sempy.fabric.fabricdataframe to spark df?

The following does not work

dataset = (fabric .evaluate_dax(workspace= server, dataset=db, dax_string=query_string) ).to_pandas() AttributeError: 'FabricDataFrame' object has no attribute 'to_pandas()'

Re: Converting Fabric dataframe to spark dataframe

frithjof_v — Sun, 15 Sep 2024 19:03:29 GMT

dataset = (fabric .evaluate_dax(workspace= server, dataset=db, dax_string=query_string) ) spark_df = spark.createDataFrame(dataset)

This works for me.

I think you can treat the Fabric dataframe as a Pandas dataframe, and convert it to a Spark dataframe the same way you would convert a Pandas dataframe to a Spark dataframe.

https://learn.microsoft.com/en-us/fabric/data-science/semantic-link-overview#fabricdataframe-data-structure

"
The FabricDataFrame class:

Supports all pandas operations.
Subclasses the pandas DataFrame and adds metadata, such as semantic information and lineage.
(...)

Re: Converting Fabric dataframe to spark dataframe

smpa01 — Sun, 15 Sep 2024 18:59:42 GMT

Liked the 1-liner; took a longer route which I can happily discard

# data_as_dict = dataset.to_dict() # columns = dataset.columns # flattened_rows = [Row(**{col: data_as_dict[col][i] for col in columns}) for i in range(len(data_as_dict[columns[0]]))]