Forum Discussion
Solved
Incremental refresh For S3 shortcut
I have parquet Partitioned data in S3 and using shortcut it only pulls the files not table and to solve this I'm using Notebook to convert that into delta tables in lakehouse. But S3 data gets refres...
If you are trying to run the notebook without a pipeline use schedule from inside the notebook
Without knowing more about the individual scenario, I would recommend the following as a first pass;
- Create a Pipeline that calls the notebook using a notebook activity
- If there's anything that you manually change in the notebook on each run, take a view as to what can be parameterised and feed those parameters in from the pipeline
- Test that the Pipeline runs and executes the notebook properly
- Create a daily schedule on the Pipeline
Scenarios is - I need to use S3 text file for power BI report in fabric, and text file gets updated everyday.