topic Re: Incremental refresh For S3 shortcut in Data Engineering

Incremental refresh For S3 shortcut

akkibuddy7 — Thu, 06 Feb 2025 08:23:23 GMT

I have parquet Partitioned data in S3 and using shortcut it only pulls the files not table and to solve this I'm using Notebook to convert that into delta tables in lakehouse. But S3 data gets refreshed every day and running notebook everyday manually is tedious work, Any automation available for such scenario?

Re: Incremental refresh For S3 shortcut

spencer_sa — Thu, 06 Feb 2025 10:01:38 GMT

Without knowing more about the individual scenario, I would recommend the following as a first pass;

Create a Pipeline that calls the notebook using a notebook activity

If there's anything that you manually change in the notebook on each run, take a view as to what can be parameterised and feed those parameters in from the pipeline
Test that the Pipeline runs and executes the notebook properly
Create a daily schedule on the Pipeline

Re: Incremental refresh For S3 shortcut

akkibuddy7 — Thu, 06 Feb 2025 12:03:50 GMT

Scenarios is - I need to use S3 text file for power BI report in fabric, and text file gets updated everyday.

Re: Incremental refresh For S3 shortcut

spencer_sa — Thu, 06 Feb 2025 12:09:08 GMT

For something as straightforward as that, a pipeline with a schedule will most likely suffice.

If this helps, please consider Accepting as a Solution to help others find it more easily.

Re: Incremental refresh For S3 shortcut

Anonymous — Thu, 06 Feb 2025 12:51:39 GMT

Hi akkibuddy7,

Thanks for reaching out to the Microsoft fabric community forum.

It looks like you have partitioned Parquet data in Amazon S3, and you're trying to bring this data into a Fabric Lakehouse. Please go through the solution provided by spencer_sa and if it solves your issue please accept it as solution.

I would also take a moment to thank spencer_sa., for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference.

If I misunderstand your needs or you still have problems on it, please feel free to let us know.

Best Regards,
Hammad.
Community Support Team

If this post helps then please mark it as a solution, so that other members find it more quickly.

Thank you.

Re: Incremental refresh For S3 shortcut

spark123 — Thu, 06 Feb 2025 16:43:41 GMT

If you are trying to run the notebook without a pipeline use schedule from inside the notebook