Forum Discussion

MelisandeRicou's avatar
MelisandeRicou
Regular Visitor
1 year ago
Solved

Shortcut to AWS S3 - AWS Data Transfer Costs

Hi, I am investigating the option to create a Fabric shortcut to a AWS S3 standard bucket. The bucket contains about 200 GB of data in parquet files, that are refreshed daily. In Fabric, this da...
data engineering
lakehouse
  • Rufyda's avatar
    Rufyda
    1 year ago

    Hi MelisandeRicou 

    Copy the data once to OneLake / Azure.

    After that, use incremental loads to transfer only daily changes instead of the full 200 GB.

    Partition your data so Fabric reads only the required portion.

    For permanent, heavy workloads, move the dataset entirely to Azure to avoid ongoing egress costs.

    If this helps, consider giving some Kudos. If I answered your question, please mark this as the solution.



  • v-prasare's avatar
    v-prasare
    1 year ago

    Hi MelisandeRicou,

     

    • A shortcut does not copy data, it streams it from S3.

    • On-demand reads trigger AWS S3 egress (billed as “DataTransfer-Out-Bytes”).

    • Caching in Fabric can reduce repeated egress

     

     

     

    Thank you for your patience and look forward to hearing from you.
    Best Regards,
    Prashanth Are
    MS Fabric community support

v-prasare's avatar
v-prasare
Icon for Community Support rankCommunity Support
1 year ago

Hi MelisandeRicou,

 

  • A shortcut does not copy data, it streams it from S3.

  • On-demand reads trigger AWS S3 egress (billed as “DataTransfer-Out-Bytes”).

  • Caching in Fabric can reduce repeated egress

 

 

 

Thank you for your patience and look forward to hearing from you.
Best Regards,
Prashanth Are
MS Fabric community support