Is Streaming Data the answer to my event driven data analytics?

[Anonymous]

Hi,

I'm trying to work out how to have real-time analytics on a data set where the queries can respond to events immediately without over cooking it.

To me, streaming data is for data that flows and has the potential to constantly changing like a moving vehicle's GPS, video or audio.

Is this now how you would also process a data set where either new data has been received/processed or the state of existing data changes because of a change of date (for example)?

I looked at the example about tracking fraudulant calls (https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-power-bi-dashboard) and it was interesting because it only produces an output when a condition is met, which to me is the same as an event driven solution - which is what I want - but I suppose the query is constantly running meaning constantly computing. Isn't that expensive? Isn't that inefficient? Is that just fine? I really don't know.

Scenario #1

I have a set of transactions which are in a state of "unpaid" and they have a payment date and I get to that payment date and I want to output the sum of money I should be paid today and then for those in an "unpaid" state (where I didn't receive a newer version of same document with same transaction reference where state is now "paid") where today is greater than payment_date show me the sum of late due money.

Is streaming data the way to process the above lifecycle of transactions?

Scenario #2

What about if it was data collected from a bank feed comparing payment amounts in and out of bank accounts to those expected from transaction accounting software to detect and show incorrect payments, is streaming data the way to go?

How would you approach it? How do I know the best way of doing this? Thanks, Matt

[dax]

Hi mattlightc3, 

If you want to get real-time data, I think you could try to use real-time streaming, it won't store in PowerBI permanently, it just store temporarily for one hour for rendering visual and it has data limit. You could refer to Real-time streaming in Power BI . 

You might need to create job like your link and refer to in by querying in custom stream data.

Best Regards,
Zoe Zhi

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Anonymous]

Thanks Zoe for your answer, I guess I'm asking if real-time streaming is what I want. Specifically ESP (Event Stream processing) is what I'm thinking about - where a data event happens and it is processed immediately but it might not be producing data all the time. My confusion is whether real-time streaming is basically just a channel where data that drops in is dealt with (aggregated, calculated, refreshed query or calculation) immediately or if it is for producing a multitude of data records that may or may not change in content.

Hope that makes sense