Forum Discussion

girishtharwani2's avatar
girishtharwani2
Icon for Helper III rankHelper III
7 days ago

Best orchestration choice to reduce Fabric capacity usage when transformations run in Snowflake

Hi,

I’m looking for advice on reducing Microsoft Fabric capacity usage. The same Fabric capacity is used for BI, so I want to avoid unnecessary data integration workloads affecting BI.

Our current flow is:

  1. Fabric Data Factory pipelines ingest data from external sources into ADLS.
  2. Fabric Data Factory pipelines load data from ADLS into Snowflake Silver.
  3. Fabric Data Factory pipelines trigger Snowflake Silver-to-Gold processing.

Most transformations run inside Snowflake. Fabric Data Factory is mainly handling ingestion, loading, and orchestration. These pipelines currently run on Fabric’s native schedules.

Would moving orchestration to a Fabric Apache Airflow Job reduce Fabric capacity consumption, or would it add more capacity usage while the Data Factory pipeline activities still run in Fabric? For this type of setup, what approach would you recommend to reduce Fabric capacity impact while keeping reliable scheduling and monitoring?

4 Replies

  • v-abhinavmu's avatar
    v-abhinavmu
    Icon for Community Support rankCommunity Support

    Hi girishtharwani2​,

    I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. 


    Thank you.

  • jubinsoni's avatar
    jubinsoni
    Icon for Skilled Sharer rankSkilled Sharer

    Hi girishtharwani2​,

    Agree that changing the scheduler alone won't help. The bigger lever in your flow is step 2: ADLS → Snowflake doesn't need to move bytes through Fabric at all. If that load is a Copy activity today, Fabric is paying for data movement that Snowflake can do with its own compute.

    today:   Fabric Copy (ADLS -> Snowflake)          <- Fabric CU for data movement
    instead: Fabric Script activity -> COPY INTO       <- Snowflake compute does the load,
                                                         Fabric only sends one SQL statement
    or:      Snowpipe auto-ingest on the ADLS container <- no Fabric involvement for step 2
    -- one-time setup in Snowflake
    CREATE STORAGE INTEGRATION adls_int
      TYPE = EXTERNAL_STAGE STORAGE_PROVIDER = 'AZURE' ENABLED = TRUE
      AZURE_TENANT_ID = '<tenant-id>'
      STORAGE_ALLOWED_LOCATIONS = ('azure://<account>.blob.core.windows.net/landing/');
    
    CREATE STAGE silver.landing_stage
      URL = 'azure://<account>.blob.core.windows.net/landing/'
      STORAGE_INTEGRATION = adls_int FILE_FORMAT = (TYPE = PARQUET);
    
    -- what the Fabric Script activity runs per load
    COPY INTO silver.orders
    FROM @silver.landing_stage/orders/
    MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE;

    Step 3 can also stay entirely in Snowflake: a Snowflake Task (or a stream + task) that runs Silver → Gold when new rows arrive, instead of Fabric triggering it.

    What stays in Fabric is step 1 (sources → ADLS), and that's where you'll see the real CU cost. Two ways to keep it away from BI:

    - split capacities: a small F SKU for the ingestion workspace,
      BI stays on its own capacity (no more background CU competing with reports)
    - if that's not possible, schedule heavy copies outside BI peak hours AND check
      Capacity Metrics -> Background % per item; smoothing spreads cost but doesn't remove it

    So my overall approach would be:

    step 1: keep in Fabric (or ADF) -> ideally its own small capacity
    step 2: Script activity + COPY INTO / Snowpipe -> Snowflake compute, not Fabric CU
    step 3: Snowflake Tasks/streams for Silver -> Gold
    keep native pipeline schedules (Airflow adds CU, doesn't remove it)

    AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

  • Hi girishtharwani2​,

    The first thing to check is what is actually consuming all your capacity. 

    Take a look at the capacity metrics app to see what is consuming all your capacity, and then look at optimizing that. 

  • I think moving just the scheduler to Fabric Airflow won't reduce CU usage. Pipeline charges remain, plus 5/10 CUs while the Airflow pool runs. Keep native schedules. Check DataMovement, ActivityRun and BI throttling in Capacity Metrics. Background CU usage is smoothed over 24 hours, so shifting run times alone won't isolate BI.

    If Copy dominates, move ingestion/loading to ADF or a separate Fabric capacity. Costs may shift or rise. Snowflake Tasks can handle Silver-to-Gold once loads complete. Monitor failures across systems.