Forum Discussion

MelisandeRicou's avatar
MelisandeRicou
Regular Visitor
1 year ago
Solved

Shortcut to AWS S3 - AWS Data Transfer Costs

Hi, I am investigating the option to create a Fabric shortcut to a AWS S3 standard bucket. The bucket contains about 200 GB of data in parquet files, that are refreshed daily. In Fabric, this da...
data engineering
lakehouse
  • Rufyda's avatar
    Rufyda
    1 year ago

    Hi MelisandeRicou 

    Copy the data once to OneLake / Azure.

    After that, use incremental loads to transfer only daily changes instead of the full 200 GB.

    Partition your data so Fabric reads only the required portion.

    For permanent, heavy workloads, move the dataset entirely to Azure to avoid ongoing egress costs.

    If this helps, consider giving some Kudos. If I answered your question, please mark this as the solution.



  • v-prasare's avatar
    v-prasare
    1 year ago

    Hi MelisandeRicou,

     

    • A shortcut does not copy data, it streams it from S3.

    • On-demand reads trigger AWS S3 egress (billed as “DataTransfer-Out-Bytes”).

    • Caching in Fabric can reduce repeated egress

     

     

     

    Thank you for your patience and look forward to hearing from you.
    Best Regards,
    Prashanth Are
    MS Fabric community support

v-prasare's avatar
v-prasare
Icon for Community Support rankCommunity Support
1 year ago

May I ask if you have resolved this issue? If so, Can you please share the resolution steps here. This will be helpful for other community members who have similar problems to solve it faster.
If we don’t hear back, we’ll go ahead and close this thread. For any further discussions or questions, please start a new thread in the Microsoft Fabric Community Forum we’ll be happy to assist.
Thank you for being part of the Microsoft Fabric Community.