Forum Discussion

danextian's avatar
danextian
Icon for Super User rankSuper User
2 years ago
Solved

Cannot add object to new semantic model

Hello,   I created a  delta table but I cannot add it to a new semantic model. There are no objects to select from in the New semantic model dialogue box. I also used the SQL connection string bu...
  • danextian's avatar
    2 years ago

    Ok. After several testing, it appears that a delta table created from a sempy dataframe cannot be added to a sql endpoint either because it is not supported or it is a bug.

    To give you a brief background, I have a very large pbix file imported  from a data source that's been deprecated.  The data is still in the pbix but it needs to be further transformed and connecting XMLA endpoint has proven to be very slow to the point of a capacity error.  The data can be exported to CSV but will require a lot of computing power to materialize within Power BI so exporting it at once using DAX Studio is out of question  and doing it in chunks can be very tedious.

    So I loaded the delta table to a notebook using spark sql and then saved it back as a delta table and voila, it now appears as a sql endpoint object and can be queried.

    To those who has the same use case, here's the cleaned notebook code:

     

    #convert from a semantic model to a sempy data frame
    from sempy import fabric as FabricDataFrame
    sempy_dataframe_name = FabricDataFrame.read_table(workspace="workspace name, not id", dataset="semantic model or dataset name, not id", table="table name in the model")
    # rename columns so there are no spaces
    column_mappings = {'colum name': 'column_name'}
    
    # Rename columns using the mapping dictionary
    sempy_dataframe_name.rename(columns=column_mappings, inplace=True)
    from pyspark.sql import SparkSession
    
    # Create a SparkSession
    spark = SparkSession.builder \
        .appName("Convert DataFrame to PySpark DataFrame") \
        .getOrCreate()
    
    #convert sempy dataframe to spark df
    df_spark = spark.createDataFrame(sempy_dataframe_name)
    # Specify the delta table name and path
    delta_table_name = "delta table name"
    delta_table_path = "Tables/" + delta_table_name
    
    # Write Spark DataFrame to Delta Table
    df_spark.write.format("delta").save(delta_table_path)