Forum Discussion
Solved
Best practice for loading data into snowflake
Hello Everyone, Can you please suggest best practice for loading data from SQL/Oracle to snowflake using Microsoft Fabric data pipeline. Case 1 : Multiple tables at a time with huge volume of da...
Hi Srisakthi
Thank you for reaching out Microsoft Fabric Community Forum.- For high-volume data loads : We should focus on parallel loading using bulk loading techniques (COPY INTO), leveraging larger compute warehouses for performance, and batch processing to minimize compute and storage costs.
- For low-volume data loads : we aim to reduce compute overhead with smaller virtual warehouses, sequential loads, and incremental loading (Streams and Tasks). Also, consider using Snowpipe for real-time or near-real-time data loading to optimize cost and compute resource usage.
If this solution helps, please consider giving us Kudos and accepting it as the solution so that it may assist other members in the community.
Thank you.
Hi Srisakthi
Thank you for reaching out Microsoft Fabric Community Forum.
- For high-volume data loads : We should focus on parallel loading using bulk loading techniques (COPY INTO), leveraging larger compute warehouses for performance, and batch processing to minimize compute and storage costs.
- For low-volume data loads : we aim to reduce compute overhead with smaller virtual warehouses, sequential loads, and incremental loading (Streams and Tasks). Also, consider using Snowpipe for real-time or near-real-time data loading to optimize cost and compute resource usage.
If this solution helps, please consider giving us Kudos and accepting it as the solution so that it may assist other members in the community.
Thank you.