Forum Discussion
Snowflake Iceburg Table
i have done a project in which i transfer the data from sql server to power bi report and the step by step process is explained below
1) first i create the pipeline in fabric data factory though which i transfer the data from sql server to fabric data lake which layer i called bronze layer . i created two different pipeline first is full and second is delta . if i want to transfer the all data then i need to execute the full otherwise i execute delta in which the change tracking is updated in the table which is stored in sql server .
2) then whatever the full and delta data are stored in the bronze layer i transfer this into silver by writing the pyspark notebook and i create only one table for one data set like in this i didnot create separate full or delta table if i run the notebook by passing the in_mode para as full then it will truncate the whole data of the table from silver and add the wholle data from full table whihc is in bronze similarly for delta para only updated values are updataed , inserted , deleted
3) After that i want to transsfer the data from fabric silver to blob though pipline which i created in fabric |
4) then to transfer the data from blob storage to snowflake i use the concept of iceburge .
i have done till blob storage but my doubt is how i can and why i create the iceberge to transfer the data from blob to snowflake
OTHERWISE IS IT POSSIBLE TO LOAD DATA FROM FABRIC SILVER TO DIRECT SNOWFLAKE SILVER LAYER