Forum Discussion
Solved
Converting Fabric dataframe to spark dataframe
How can I convert a sempy.fabric.fabricdataframe to spark df?
The following does not work
dataset = (fabric
.evaluate_dax(workspace= server,
dataset=db,
d...
dataset = (fabric .evaluate_dax(workspace= server, dataset=db, dax_string=query_string) ) spark_df = spark.createDataFrame(dataset)This works for me.
I think you can treat the Fabric dataframe as a Pandas dataframe, and convert it to a Spark dataframe the same way you would convert a Pandas dataframe to a Spark dataframe.
"
The FabricDataFrame class:- Supports all pandas operations.
- Subclasses the pandas DataFrame and adds metadata, such as semantic information and lineage.
- (...)
"
dataset = (fabric
.evaluate_dax(workspace= server,
dataset=db,
dax_string=query_string)
)
spark_df = spark.createDataFrame(dataset)
This works for me.
I think you can treat the Fabric dataframe as a Pandas dataframe, and convert it to a Spark dataframe the same way you would convert a Pandas dataframe to a Spark dataframe.
"
The FabricDataFrame class:
- Supports all pandas operations.
- Subclasses the pandas DataFrame and adds metadata, such as semantic information and lineage.
- (...)
"
Liked the 1-liner; took a longer route which I can happily discard
# data_as_dict = dataset.to_dict()
# columns = dataset.columns
# flattened_rows = [Row(**{col: data_as_dict[col][i] for col in columns}) for i in range(len(data_as_dict[columns[0]]))]