Incremental refresh For S3 shortcut

[akkibuddy7]

I have parquet Partitioned data in S3 and using shortcut it only pulls the files not table and to solve this I'm using Notebook to convert that into delta tables in lakehouse. But S3 data gets refreshed every day and running notebook everyday manually is tedious work, Any automation available for such scenario?

[spark123 · Accepted solution]

If you are trying to run the notebook without a pipeline use schedule from inside the notebook 

[v-mdharahman]

Hi @akkibuddy7,

Thanks for reaching out to the Microsoft fabric community forum.

It looks like you have partitioned Parquet data in Amazon S3, and you're trying to bring this data into a Fabric Lakehouse. Please go through the solution provided by @spencer_sa and if it solves your issue please accept it as solution.

I would also take a moment to thank @spencer_sa., for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference.

If I misunderstand your needs or you still have problems on it, please feel free to let us know.  

Best Regards,
Hammad.
Community Support Team

If this post helps then please mark it as a solution, so that other members find it more quickly.

Thank you.

[spencer_sa]

For something as straightforward as that, a pipeline with a schedule will most likely suffice.

If this helps, please consider Accepting as a Solution to help others find it more easily.

[spencer_sa]

Without knowing more about the individual scenario, I would recommend the following as a first pass;

• Create a Pipeline that calls the notebook using a notebook activity

• If there's anything that you manually change in the notebook on each run, take a view as to what can be parameterised and feed those parameters in from the pipeline
• Test that the Pipeline runs and executes the notebook properly
• Create a daily schedule on the Pipeline
  

[akkibuddy7]

Scenarios is - I need to use S3 text file for power BI report in fabric, and text file gets updated everyday.