Forum Discussion
Ingest data from S3 into Snowflake
Hi, I am currently working around to ingest data from S3 into Snowflake, I use Pipeline in Fabric to do it. There are 2 options that I found:
- First is create a shortcuts from S3 into Fabric Onelake, then I will process one Onelake into Snowflake.
- Second is create a pipeline with source is S3 and sink the data directly into Snowflake.
Could you recommend me which approach is better (for Fabric, it would acting as orchestrator, since all transformation will be done later in Snowflake). Also if i use shortcuts, will it cost more than using S3 connector in pipeline item ?
Hello bao_phan
Both the options you mentioned will work technically, however, given your specific use case, and yuor consideration on cost, there's one that fits better.
Option 1: Shortcut via OneLake (S3 > OneLake > Snowflake)
When you use OneLake shortcuts, Fabric gives you a logical view over S3 rather than copying the data. Every time the pipeline reads the shortcut, Fabric is effectively streaming data out of S3 and into your process. This works well when Fabric engines also need to analyse the same data, but less so when Fabric is only acting as a coordinator.
The main upside here is architectural flexibility. You get a shared, cloud‑agnostic data layer that Spark, SQL and Power BI can all reuse without duplication. It also provides a clean abstraction if you expect Fabric to become a consumer later on.
The downside is cost and indirection. You incur AWS S3 egress on reads, Fabric capacity consumption for OneLake access, and then Snowflake load compute on top. Even with shortcut caching, this tends to be more expensive for batch ingestion, because each run still has to materialise data through Fabric.
Option 2: Direct pipeline (S3 > Snowflake)
With a direct pipeline, Fabric stays firmly in the orchestration role. Data flows straight from S3 into Snowflake, and Fabric simply triggers and monitors the COPY operation. There’s no intermediate lake layer and no additional data hop.
This approach is simpler to reason about operationally. You only pay unavoidable AWS egress and Snowflake warehouse compute, while Fabric consumption stays minimal because it’s not reading or materialising the data itself. Cost attribution and performance tuning remain Snowflake‑centric.
The trade‑off is that you’re not building a shared lake for Fabric users. If, later on, Fabric workloads need to query the raw data, you’d need to reintroduce OneLake or point Fabric directly at Snowflake instead. For ingestion‑only scenarios, though, this is usually acceptable.
Recommended approachGiven that you’re using Fabric purely as an orchestrator and all transformations happen downstream, the direct S3 > Snowflake pipeline is the better fit. It avoids unnecessary Fabric compute, removes an extra data hop, and keeps the cost model clean and predictable. OneLake shortcuts simply don’t add enough value in this scenario to justify the overhead.
3 Replies
Hello bao_phan
Both the options you mentioned will work technically, however, given your specific use case, and yuor consideration on cost, there's one that fits better.
Option 1: Shortcut via OneLake (S3 > OneLake > Snowflake)
When you use OneLake shortcuts, Fabric gives you a logical view over S3 rather than copying the data. Every time the pipeline reads the shortcut, Fabric is effectively streaming data out of S3 and into your process. This works well when Fabric engines also need to analyse the same data, but less so when Fabric is only acting as a coordinator.
The main upside here is architectural flexibility. You get a shared, cloud‑agnostic data layer that Spark, SQL and Power BI can all reuse without duplication. It also provides a clean abstraction if you expect Fabric to become a consumer later on.
The downside is cost and indirection. You incur AWS S3 egress on reads, Fabric capacity consumption for OneLake access, and then Snowflake load compute on top. Even with shortcut caching, this tends to be more expensive for batch ingestion, because each run still has to materialise data through Fabric.
Option 2: Direct pipeline (S3 > Snowflake)
With a direct pipeline, Fabric stays firmly in the orchestration role. Data flows straight from S3 into Snowflake, and Fabric simply triggers and monitors the COPY operation. There’s no intermediate lake layer and no additional data hop.
This approach is simpler to reason about operationally. You only pay unavoidable AWS egress and Snowflake warehouse compute, while Fabric consumption stays minimal because it’s not reading or materialising the data itself. Cost attribution and performance tuning remain Snowflake‑centric.
The trade‑off is that you’re not building a shared lake for Fabric users. If, later on, Fabric workloads need to query the raw data, you’d need to reintroduce OneLake or point Fabric directly at Snowflake instead. For ingestion‑only scenarios, though, this is usually acceptable.
Recommended approachGiven that you’re using Fabric purely as an orchestrator and all transformations happen downstream, the direct S3 > Snowflake pipeline is the better fit. It avoids unnecessary Fabric compute, removes an extra data hop, and keeps the cost model clean and predictable. OneLake shortcuts simply don’t add enough value in this scenario to justify the overhead.Hi bao_phan ,
Before answering which option is better, there's a prior question that changes everything: are you already using Fabric for other things, or would this be your first use case?
If you already have an active Fabric capacity, the decision is fairly straightforward. You only bring data into OneLake if you're going to use it within Fabric, for analytics, Power BI, Spark transformations, or anything else. If the only destination is Snowflake, that intermediate layer adds complexity and compute cost without adding any value. In that case, a direct Copy Data from S3 to Snowflake inside the pipeline is the cleanest approach, with Fabric acting as a pure orchestrator.
If you're not on Fabric yet and this would be your first use case, I'd go straight to Azure Data Factory. It has everything you need for this scenario and its cost model is pay-as-you-go under an Azure subscription, which makes it more cost-effective for a standalone use case with no capacity management involved. Fabric is also pay-as-you-go, but it runs on a capacity that you need to manually pause and resume to keep costs under control, which adds an operational overhead that doesn't make sense to take on if this is your only workload. If you're already paying for that capacity or have more Fabric workloads planned, then use Fabric pipelines directly and skip ADF altogether.
The general rule: bring data into OneLake only if it has a life inside Fabric. Otherwise, go with a direct Copy Data. And if Fabric isn't in the picture yet, ADF is the simpler and more cost-effective choice for this scenario.
Hope this was helpful.
If my comment helped solve your question, it would be great if you could mark it as the accepted solution and give me like. It helps others with the same issue and it also motivates me to keep contributing.
Thanks a lot. I really appreciate it
Hi bao_phan
since Fabric is just the orchestrator here and all transformation happens in Snowflake, the decision is mostly about cost, simplicity, and maintenance.
For your use case, option 2 (S3 source → Snowflake sink via pipeline) is the better fit. Here’s why:
You skip an intermediate layer entirely. No shortcuts to manage, no OneLake storage to maintain for data that’s just passing through. Fewer moving parts means less to troubleshoot.
Shortcuts themselves are free (they’re just metadata pointers), but when Fabric reads data through a shortcut from S3, it still incurs compute and egress.
