Incremental refresh For S3 shortcut

Without knowing more about the individual scenario, I would recommend the following as a first pass;

• Create a Pipeline that calls the notebook using a notebook activity

 

 

• If there's anything that you manually change in the notebook on each run, take a view as to what can be parameterised and feed those parameters in from the pipeline
• Test that the Pipeline runs and executes the notebook properly
• Create a daily schedule on the Pipeline
  

   

For something as straightforward as that, a pipeline with a schedule will most likely suffice.

 

If this helps, please consider Accepting as a Solution to help others find it more easily.

Hi akkibuddy7,

Thanks for reaching out to the Microsoft fabric community forum.

 

It looks like you have partitioned Parquet data in Amazon S3, and you're trying to bring this data into a Fabric Lakehouse. Please go through the solution provided by spencer_sa and if it solves your issue please accept it as solution.

 

I would also take a moment to thank spencer_sa., for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference.

 

If I misunderstand your needs or you still have problems on it, please feel free to let us know.  

Best Regards,
Hammad.
Community Support Team

 

If this post helps then please mark it as a solution, so that other members find it more quickly.

Thank you.