DataSet does not refresh from DataFlow (both with incremental refresh) from Storage Blob

[stanbeamish]

Hi Community,

we use PowerBI embedded to provide a report that shows data that is read in from a DataSet receiving data from a DataFlow that reads the data from an Azure Storage Blob.

This all works ok-ish (memory errors) only if we do NOT make use of incremental refresh.

To handle these memory errors, we want to make use of the incremental refresh on both DatFlow and DataSet.

Questions:

1. In general, is this a viable scenario?
2. The DataFlow has a configured incremental refresh, and the data is refreshed correctly. But the manual trigger of refreshing the data of the DataSet does not result in updated data. Only data of the very first upload is available. Is there any hint, where to find the error/problem (no error message is provided)?

We make use of PowerBI Embedded A1 with a Service Principal

DataFlow-Settings:

6 months archive data, 7 days incremental refresh, NO detecting data changes (because there is not data change on existing data)

DataSet-Settings:

6 months archive data, 7 days incremental refresh, NO detecting data changes (because there is not data change on existing data)

Any idea welcome.

Cheers,

Olli

[stanbeamish]

So, we ended up using a Stream Analytics Job, that executes on a Gen2 Data Lake Blob.
This all works fine for some amount of days. The PowerBI report updates accordingly.  
Then, the data is updated in a FIFO-manner as observed. Previous data is no longer available to the report and later data is available. Where is this limitation set? Or is this a default limitation?
Also we get this error message:
" Warning: Blob scan operation scanned a large number ({#>485}) of older blobs. Please consider using the date/time path pattern to reduce the number of blobs that have to be listed."

The blob pattens is YYYY/MM/DDTHH/mm/ss, so the consideration is already matched.

Any idea on how to show more data than these ~215.000 records with Stream Analytics job on a PowerBI report? We are stuck.

[stanbeamish]

Hi, thank you for your response.
Unfortunately we cannot find a solution by just using incremental refresh on the DataSet. Still data would not be refreshed as expected. We need to puzzle with other options and ask further questions if we are able to provide those.

[arvindsingh802]

Microsoft recommand not to use incremental refresh in dataset if its been implimented in dataflow already.
Can you try in this way and share the error you facing