Forum Discussion
Incremental refresh causing "There weren't enough elements in the enumeration to complete the..."
This is not something that you do with Power BI (it is not designed to write back to the data source). You would use tools like Powershell or Power Automate.
Please explain the reasoning for using incremental refresh. As you know SharePoint.Files is not folding, and even if it were, your subsequent transforms are breaking folding too. So you will not have any processing time advantage
"Date Modified" is not a usable column for Incremental Refresh. You need an immutable column like "Date Created", otherwise you will have data duplication across partitions.
For general guidance - incremental refresh partitions should have millions of rows to make sense.
We have folders with invoices on Sharepoint .pdf format since 2022. Invoices don't change, new are just being added each month - that's why I think Date modified would be fine.
With each PBI refresh all data get's refreshed, which gets slower and slower, when in fact it should be only checking newest data, for example last 3 months.
I've watched this youtube guide https://youtu.be/IVMdg16yBKE which says that Incremental refresh can also be used with non-foldable sources.
I've also tried Dataflows which someone suggested in another topic, but honestly I hated it, I couldn't make it work either, it kept adding some automatic steps to my query that I couldn't remove.
With each PBI refresh all data get's refreshed, which gets slower and slower, when in fact it should be only checking newest data, for example last 3 months.Only if it folds
which says that Incremental refresh can also be used with non-foldable sources.That is correct - but you will not get the expected time savings. all you get is partition management.
In this case what would you suggest to do? Refreshing is starting to take around 12 min right now and it will grow more in time.
12 minutes is not something you want to waste your energy for to create all the incremental refresh processes. You should start getting concerned when the time is closer to 2.5 hrs.
Run an independent process that extracts the necessary meta data from your PDFs, store that as CSV or Parquet on the same sharepoint, and ingest that.