datamart and dataflow incremental refresh

[razieh1990]

Hello

I am a SharePoint list as my data source and have established a dataflow to extract sales data from the past three years. Each day's data is stored in a separate Excel file, resulting in approximately 900 files currently, with the count increasing daily.

I tried to implement incremental refresh on the dataflow, it did not work. I think this because SharePoint list as a data source isn't compatible with incremental refresh functionality.

So, I  created a datamart and set up a dataflow with the datamart as its source, successfully applying incremental refresh on data flow.

my question:

I need the report to be refreshed three times daily. When scheduling these frequent refreshes directly on the datamart (which lacks incremental refresh capability), I observed a noticeable degradation in refresh performance in terms of elapsed time.

I would greatly appreciate any suggestions or insights on how to efficiently extract data from a SharePoint list into a dataflow and enable incremental refresh. 

Thanks you 

[lbendlin]

I tried to implement incremental refresh on the dataflow, it did not work. I think this because SharePoint list as a data source isn't compatible with incremental refresh functionality.

Any data source that has a DateTime column for the partition management will work.  You may not gain much in terms of performance if the query doesn't fold.

What's the point of your datamart?  Are you running SQL queries?

Worst case you can have dataflows for certain time periods (one per quarter, for example)  and then union these in the semantic model.