Forum Discussion

AyusmanBasu0604's avatar
AyusmanBasu0604
Icon for Helper II rankHelper II
1 year ago
Solved

How to create a Materialized Lake View out of a Data Frame?

I am trying to create a Materialized Lake View in Fabric based on a dataFrame. Since, it seem impossible directly (unless anobody can suggest) I tried to create both a temporary view/ Global tempo...
lakehouse
  • BhaveshPatel's avatar
    BhaveshPatel
    1 year ago

    For that, you have to understand bronze, silver and gold tables. 

     

    So, this is how it works:

     

    Python --> Apache Spark ( spark ) --> delta lake ( parquet with _delta_log transaction log) ( bronze, silver and gold tables )  .

     

    In Delta Lake, we are implementing bronze tables. Once the initial table is created in bronze tables, then we will proceed to materialised lake view ( silver tables ), and finally we will implement the final gold tables.

     

    so here First, create a apache spark ( dataframe  ) using pyspark which is correct in your instance ( df ) . then convert it into bronze delta tables . For that, 

     
    df.write.mode("overwrite").option("overwriteSchema", "true").format("delta").saveAsTable("Tables")  ( # creating bronze delta tables ) 
     
    Now, once bronze delta tables are created, then you are going to implement silver tables, aka Materialised Silver Tables. 
    createView = f"""
            CREATE MATERIALIZED LAKE VIEW IF NOT EXISTS vw_Tables AS
            SELECT * FROM Tables
            """
    result = spark.sql(createView)
    display(result)
     
    Here you are going to implement ETL using Delta Lake silver tables and finally you are preparing Final Gold Tables
     
BhaveshPatel's avatar
BhaveshPatel
Icon for Super User rankSuper User
1 year ago

For that, you have to understand bronze, silver and gold tables. 

 

So, this is how it works:

 

Python --> Apache Spark ( spark ) --> delta lake ( parquet with _delta_log transaction log) ( bronze, silver and gold tables )  .

 

In Delta Lake, we are implementing bronze tables. Once the initial table is created in bronze tables, then we will proceed to materialised lake view ( silver tables ), and finally we will implement the final gold tables.

 

so here First, create a apache spark ( dataframe  ) using pyspark which is correct in your instance ( df ) . then convert it into bronze delta tables . For that, 

 
df.write.mode("overwrite").option("overwriteSchema", "true").format("delta").saveAsTable("Tables")  ( # creating bronze delta tables ) 
 
Now, once bronze delta tables are created, then you are going to implement silver tables, aka Materialised Silver Tables. 
createView = f"""
        CREATE MATERIALIZED LAKE VIEW IF NOT EXISTS vw_Tables AS
        SELECT * FROM Tables
        """
result = spark.sql(createView)
display(result)
 
Here you are going to implement ETL using Delta Lake silver tables and finally you are preparing Final Gold Tables