Forum Discussion
Best orchestration choice to reduce Fabric capacity usage when transformations run in Snowflake
Hi,
I’m looking for advice on reducing Microsoft Fabric capacity usage. The same Fabric capacity is used for BI, so I want to avoid unnecessary data integration workloads affecting BI.
Our current flow is:
- Fabric Data Factory pipelines ingest data from external sources into ADLS.
- Fabric Data Factory pipelines load data from ADLS into Snowflake Silver.
- Fabric Data Factory pipelines trigger Snowflake Silver-to-Gold processing.
Most transformations run inside Snowflake. Fabric Data Factory is mainly handling ingestion, loading, and orchestration. These pipelines currently run on Fabric’s native schedules.
Would moving orchestration to a Fabric Apache Airflow Job reduce Fabric capacity consumption, or would it add more capacity usage while the Data Factory pipeline activities still run in Fabric? For this type of setup, what approach would you recommend to reduce Fabric capacity impact while keeping reliable scheduling and monitoring?
4 Replies
- v-abhinavmu
Community Support
Hi girishtharwani2,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.
Thank you. - jubinsoni
Skilled Sharer
Hi girishtharwani2,
Agree that changing the scheduler alone won't help. The bigger lever in your flow is step 2: ADLS → Snowflake doesn't need to move bytes through Fabric at all. If that load is a Copy activity today, Fabric is paying for data movement that Snowflake can do with its own compute.
today: Fabric Copy (ADLS -> Snowflake) <- Fabric CU for data movement instead: Fabric Script activity -> COPY INTO <- Snowflake compute does the load, Fabric only sends one SQL statement or: Snowpipe auto-ingest on the ADLS container <- no Fabric involvement for step 2-- one-time setup in Snowflake CREATE STORAGE INTEGRATION adls_int TYPE = EXTERNAL_STAGE STORAGE_PROVIDER = 'AZURE' ENABLED = TRUE AZURE_TENANT_ID = '<tenant-id>' STORAGE_ALLOWED_LOCATIONS = ('azure://<account>.blob.core.windows.net/landing/'); CREATE STAGE silver.landing_stage URL = 'azure://<account>.blob.core.windows.net/landing/' STORAGE_INTEGRATION = adls_int FILE_FORMAT = (TYPE = PARQUET); -- what the Fabric Script activity runs per load COPY INTO silver.orders FROM @silver.landing_stage/orders/ MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE;Step 3 can also stay entirely in Snowflake: a Snowflake Task (or a stream + task) that runs Silver → Gold when new rows arrive, instead of Fabric triggering it.
What stays in Fabric is step 1 (sources → ADLS), and that's where you'll see the real CU cost. Two ways to keep it away from BI:
- split capacities: a small F SKU for the ingestion workspace, BI stays on its own capacity (no more background CU competing with reports) - if that's not possible, schedule heavy copies outside BI peak hours AND check Capacity Metrics -> Background % per item; smoothing spreads cost but doesn't remove itSo my overall approach would be:
step 1: keep in Fabric (or ADF) -> ideally its own small capacity step 2: Script activity + COPY INTO / Snowpipe -> Snowflake compute, not Fabric CU step 3: Snowflake Tasks/streams for Silver -> Gold keep native pipeline schedules (Airflow adds CU, doesn't remove it)AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.
- tayloramy
Super User
Hi girishtharwani2,
The first thing to check is what is actually consuming all your capacity.
Take a look at the capacity metrics app to see what is consuming all your capacity, and then look at optimizing that.
- Kagiyama_yutaka
Skilled Sharer
I think moving just the scheduler to Fabric Airflow won't reduce CU usage. Pipeline charges remain, plus 5/10 CUs while the Airflow pool runs. Keep native schedules. Check DataMovement, ActivityRun and BI throttling in Capacity Metrics. Background CU usage is smoothed over 24 hours, so shifting run times alone won't isolate BI.
If Copy dominates, move ingestion/loading to ADF or a separate Fabric capacity. Costs may shift or rise. Snowflake Tasks can handle Silver-to-Gold once loads complete. Monitor failures across systems.