Forum Discussion
Cannot add object to new semantic model
- 2 years ago
Ok. After several testing, it appears that a delta table created from a sempy dataframe cannot be added to a sql endpoint either because it is not supported or it is a bug.
To give you a brief background, I have a very large pbix file imported from a data source that's been deprecated. The data is still in the pbix but it needs to be further transformed and connecting XMLA endpoint has proven to be very slow to the point of a capacity error. The data can be exported to CSV but will require a lot of computing power to materialize within Power BI so exporting it at once using DAX Studio is out of question and doing it in chunks can be very tedious.
So I loaded the delta table to a notebook using spark sql and then saved it back as a delta table and voila, it now appears as a sql endpoint object and can be queried.
To those who has the same use case, here's the cleaned notebook code:
#convert from a semantic model to a sempy data frame from sempy import fabric as FabricDataFrame sempy_dataframe_name = FabricDataFrame.read_table(workspace="workspace name, not id", dataset="semantic model or dataset name, not id", table="table name in the model")# rename columns so there are no spaces column_mappings = {'colum name': 'column_name'} # Rename columns using the mapping dictionary sempy_dataframe_name.rename(columns=column_mappings, inplace=True)from pyspark.sql import SparkSession # Create a SparkSession spark = SparkSession.builder \ .appName("Convert DataFrame to PySpark DataFrame") \ .getOrCreate() #convert sempy dataframe to spark df df_spark = spark.createDataFrame(sempy_dataframe_name)# Specify the delta table name and path delta_table_name = "delta table name" delta_table_path = "Tables/" + delta_table_name # Write Spark DataFrame to Delta Table df_spark.write.format("delta").save(delta_table_path)
Ok. After several testing, it appears that a delta table created from a sempy dataframe cannot be added to a sql endpoint either because it is not supported or it is a bug.
To give you a brief background, I have a very large pbix file imported from a data source that's been deprecated. The data is still in the pbix but it needs to be further transformed and connecting XMLA endpoint has proven to be very slow to the point of a capacity error. The data can be exported to CSV but will require a lot of computing power to materialize within Power BI so exporting it at once using DAX Studio is out of question and doing it in chunks can be very tedious.
So I loaded the delta table to a notebook using spark sql and then saved it back as a delta table and voila, it now appears as a sql endpoint object and can be queried.
To those who has the same use case, here's the cleaned notebook code:
#convert from a semantic model to a sempy data frame
from sempy import fabric as FabricDataFrame
sempy_dataframe_name = FabricDataFrame.read_table(workspace="workspace name, not id", dataset="semantic model or dataset name, not id", table="table name in the model")# rename columns so there are no spaces
column_mappings = {'colum name': 'column_name'}
# Rename columns using the mapping dictionary
sempy_dataframe_name.rename(columns=column_mappings, inplace=True)from pyspark.sql import SparkSession
# Create a SparkSession
spark = SparkSession.builder \
.appName("Convert DataFrame to PySpark DataFrame") \
.getOrCreate()
#convert sempy dataframe to spark df
df_spark = spark.createDataFrame(sempy_dataframe_name)# Specify the delta table name and path
delta_table_name = "delta table name"
delta_table_path = "Tables/" + delta_table_name
# Write Spark DataFrame to Delta Table
df_spark.write.format("delta").save(delta_table_path)
Hi danextian ,
It was great to know that you were able to get to a resolution . We expect you to keep using this forum and also motivate others to do that same . You can always help other community members by answering to their queries