Forum Discussion
Incremental refresh causing "There weren't enough elements in the enumeration to complete the..."
This is not something that you do with Power BI (it is not designed to write back to the data source). You would use tools like Powershell or Power Automate.
We have folders with invoices on Sharepoint .pdf format since 2022. Invoices don't change, new are just being added each month - that's why I think Date modified would be fine.
With each PBI refresh all data get's refreshed, which gets slower and slower, when in fact it should be only checking newest data, for example last 3 months.
I've watched this youtube guide https://youtu.be/IVMdg16yBKE which says that Incremental refresh can also be used with non-foldable sources.
I've also tried Dataflows which someone suggested in another topic, but honestly I hated it, I couldn't make it work either, it kept adding some automatic steps to my query that I couldn't remove.
With each PBI refresh all data get's refreshed, which gets slower and slower, when in fact it should be only checking newest data, for example last 3 months.Only if it folds
which says that Incremental refresh can also be used with non-foldable sources.That is correct - but you will not get the expected time savings. all you get is partition management.
In this case what would you suggest to do? Refreshing is starting to take around 12 min right now and it will grow more in time.
12 minutes is not something you want to waste your energy for to create all the incremental refresh processes. You should start getting concerned when the time is closer to 2.5 hrs.
Run an independent process that extracts the necessary meta data from your PDFs, store that as CSV or Parquet on the same sharepoint, and ingest that.
Thank you - could you please explain more about creating independent process to store .pdf data as CSV?
I would like to automate this process as much as possible, so when user uploads new .pdf file to SP - PowerBI will automatically do the rest.
I'm not sure if I understand correctly what you propose, but wouldn't that just mean that there will be two reports where I would've to refresh data - one for transfering .pdfs to CSV, second for pulling data from CSV? Isn't that a little counterproductive? How would I save time here when second report would take ~12 min to refresh and then additionally I have to refresh first report as well?
I'm sorry if those are dumb questions, I'm not that experienced.