Forum Discussion
Poweraegg
Advocate IV
Advocate IVFabric - Moving Data in Medallion Architecure between Workspaces
Hi, I am trying to load and transform data from the Lakehouse in the Silver Layer Workspace to a Warehouse in the Gold Layer Workspace. However, I only seem to be able to use a Gen2 Dataflow for ...
PoweraeggAdvocate IV
Advocate IVfrom delta.tables import *
from pyspark.sql.types import *
from pyspark.sql.functions import *
# Define the path
silver_path = "abfss://[email protected]/Contoso_Silver_LH.Lakehouse/Files/Product_Silver_External"
gold_path = "abfss://[email protected]/WH_Contoso_Gold.Datawarehouse/Tables/dbo/Products_Gold"
# Read the Delta table
deltaTable = DeltaTable.forPath(spark, silver_path)
# Convert the Delta table to a DataFrame
df_silver = deltaTable.toDF()
# Cast the 'Unit_Cost' and 'Unit_Price' columns to match the types in the Gold Warehouse table
df_silver = df_silver.withColumn("Unit_Cost", col("Unit_Cost").cast(DecimalType(18,0)))
df_silver = df_silver.withColumn("Unit_Price", col("Unit_Price").cast(DecimalType(18,0)))
# Write the data to the Gold Warehouse
df_silver.write.format("delta").mode("overwrite").save(gold_path)
Output -
ERROR: Caused by: Operation failed: "Bad Request", 400, PUT, http://onelake.dfs.fabric.microsoft.com/Contoso_Gold/WH_Contoso_Gold.Datawarehouse/Tables/dbo/Products_Gold/part-00000-3c5b3f4d-a440-46f3-9649-eea4c5e22693-c000.snappy.parquet?resource=file&timeout=90, OperationNotAllowedOnThePath, "PUT call is not allowed on Tables path"
- bcdobbs
Community Champion
I think the issue is you can only write to warehouse tables through the sql end point. I'll have a play and get back to you.
