Power BI Streaming dataset - Updates to source feed duplicating

[phanis]

Hi,

We have a scenario where streaming dataset passes through different stages before the workflow at the source either gets successfully completed or fails. Changes happening to the steaming datasets need to be reflected on the Power BI Dashboard. While we are able to see data in real time on the Power BI Dashboard, we are seeing one row for each status change for the same record in the dashboard which is kind of duplicating the data. At the source same record is getting updated.

We wanted to show only the latest status of the record. We did ensure that the Streaming dataset query has the required logic to return only the most recent row, since the Power BI dataset in the workspace is retaining the data, which is leading to this duplication.

We did try work arounds suggested like filtering data in the form of report and pinning that to dashboard - but they didnt work.

Looking forward for any suggestions that would help meeting our need.

Regards,

Phaneendra

Regards,

Phaneendra

[phanis]

Hi collinq,

Thanks for the response. 

We are doing that in the Azure Stream Analytics query using TopOne() function ordering by timestamp.

Our analysis of root cause is as follows:

1. When the record gets processed for the first time, it is getting reflected in the Power BI dataset and eventually to dashboard
2. When the same record goes through an update, the Steam analytics is returning only one row with updated record after we implemented the TopOne() function (earlier Stream analytics was returning two rows) it is returning only one row but in Power BI dashboard it is showing both the first record and the recent one.
3. We believe that this is happening on Power BI side because the old record is retained in the Power BI dataset side even though it is filtered in the Streaming dataset.

We need to filter that record so that the old record is eliminated.

Please let us know if you have any inputs/suggestions on this.

[collinq]

Hi @phanis,

Why not use the MAX function to select the Newest (max or highest) date on that field?  The Dax would be to use the MAX, MAXA or MAXX function - I am not sure which is the right one in your context.

[collinq]

Hi @phanis ,

I am not sure exactly how you would get around the front part of that because you are getting different datapoints throughout the flow.  I think I would solve this on the output side.  Why not use a grouping (or merging possibly) of the row to have each separate time stamp of each stage and put them into one row rather than having the row duplicated each time?