Share your thoughts on streaming dataflows aka SDF (preview feature)

[MiguelMartinez]

Back at MBAS, our CVP Arun Ulag gave everybody a sneak peek of streaming dataflows, the new experience for real-time data preparation and comsumption in Power BI. Later at BUILD, we showed everyone an extended demo of the current bits being tried by several of our customers as part of the private preview. And today, finally streaming dataflows is released to the world for public preview with even more updates a new UI.

We would love to hear your feedback and opinions to help us decide what comes next and how to improve this functionality in general. Thanks you so much in advance for your feedback and we will be alert to answer any questions you might have and listen to any ideas you might come up with.

Cheers

The streaming dataflows team

[MiguelMartinez]

Thanks for the flag @markusr Let me check with the team and get back you on this. Doesn't seem we have seen this error before but I'll confirm 👍

[MiguelMartinez]

Hi again @markusr . Some quick checks before we go further down the road here:

1. Did you go into the dataset settings in the service to enter and confirm your credentials?
2. Might be unrelated, but make sure you used the new Power Platform dataflows connector

If is not any of these 2, then it might be something else we need to take a look at 👍

[markusr]

Hi @MiguelMartinez , you found it! I really had to go into the dataset settings in the service to enter and confirm my credentials, now the report works and refreshes. I had to modify the dataset settings of the dataset that automatically gets created along with the publishing of the report, and with the same name as the report! I completely ignored that! So make sure this clearly visible in the documentation (it probably is already, I didn't check that). Great work, thanks for your help! 

[MiguelMartinez]

Thanks for confirming @markusr Great idea on including that in docs. I assumed that people would know because of regular dataflows but other users might come directly to streaming. Will add that note ASAP. Thanks again for the flag for using this awesome feature! 👍🙏

[HDK]

Hi Miguel,

I am trying to use Streaming Dataflows with an Event Hub connection to bring live weather data into Power BI. When creating the streaming flow, I get data in the data preview for the Event Hub item, but for all others it remains blank (Manage Fields / Filter / Output Table). In all documentation I've seen, you should be able to click on each item in the flow and see a static data view, but somehow it doesn't work for me. Have you got any tips?

Thanks.

[MiguelMartinez]

Apologies for the late reply @HDK Totally missed this message. Let's start checking did you click on the "Get statis preview" button on top of the data preview table? If yes, we might need more details to figure out what's going on. I showed it on the monthly update video too in case you want to take a look.

[HDK]

Thanks for the reply, the issue I was having became fixed the day after my post, not sure if a bug fix was implemented or something very peculiar happened. All working fine now, thanks!

[MiguelMartinez]

Great to hear and thanks for letting us know. If you have any more questions, comments, or feedback let us know here and we will try to get back to you ASAP 👍

[AndyDDC]

Hi Miguel,  I'm currently working through the documentation to get up and running but hitting a snag.  I'm using a PPU account and creating a blank workspace with PPU assignment.  The streaming dataflow option is greyed out with a message stating "Workspaces connected to a storage account are not supported".  This workspace is not connected to Azure nor is the tenant.  I've also tried using an Embedded A3 tier attached to the workspace but I get the same message.

[MrKrukauskas]

Hello, does not seem that the streaming dataflows work with deeper than 1 level nested items? F.e. trying to pull the _env field and it does not produce an output. Any ideas?

[Eklavya]

Hi, you can pull it by using Manage Fields operator

[MrKrukauskas]

@Eklavya still does not work. When I use Manage fields to pull f.e. 1st level items (not nested), it works fine. When I try to do it with 2nd level nested items - it does not. I'm pulling all from the same event hub, same streaming dataflow, use manage fields, filters, output tables and the only output is for the items that are not nested.
Works here:

Does not work here:

Streaming Dataflow:

[Eklavya]

I see. Let's do some quick checks here:
1. You've verified that env field does have data coming in from the source?

2. Do you see the data if you remove the filter?

3. I am assuming you are using preview to check presence of data - but can you save and run the streaming dataflow and then check if you see the output in the report?

Is it possible to share the schema/sample event of the data you are using? (You can dm me this if you'd like).

[MrKrukauskas]

@Eklavya @MiguelMartinez here are the answers:

I see. Let's do some quick checks here:
1. You've verified that env field does have data coming in from the source?

• Yes

2. Do you see the data if you remove the filter?

• I see data with and without filter in the Preview.

3. I am assuming you are using preview to check presence of data - but can you save and run the streaming dataflow and then check if you see the output in the report?

• Yes, in each step of the dataflow in the preview, i can see streaming and static data, even in the output tables.
• If I save and run, I only see the output for not nested data.

The Data preview in streaming dataflows: green - able to pull, not nested, red - not able to pull, nested two levels.

Thank You for checking this @Eklavya 

[MrKrukauskas]

Just to add, in some cases empty rows are coming through with and without Filter feature.

[Eklavya]

Okay, I'll check and get back to you today

[Eklavya]

@MrKrukauskas We tried reproducing the issue by reconstructing a payload similar to your description and it works for us:

Payload:

outputs:

Can you ensure that your payload is valid json?

To help you further, we will need the payload you are using for your events. The best way to do this is through Power BI Support

[MrKrukauskas]

@Eklavya Thank You for checking this. I'll make sure to double check if the payload is a valid json. I see that our examples differ in payload structure, so that is probably where the issue is, but I'll check first.

[MrKrukauskas]

@Eklavya  Just double checked and the incoming json is valid. Will see if I can get help from Power BI Support.

[MrKrukauskas]

@Eklavya we (data engineer from my team and myself) found a solution. Since our event hub consisted of partition 0, partition 1 and those "free fields" and I mainly needed the data from partitions, the data engineer created a new event hub where partition 0 and partition 1 was combined into one and the output looks like only free columns, no second level columns.

So our guess is that PBI SDF (streaming dataflows) cannot for now handle a bit more complex partitions that include lists, dictionaries.

Contact me if you need more details.

Thank You for looking into this!

[MiguelMartinez]

Hey @AndyDDC I replied to this in the blog comments too. You found a bug that blocks any workspace in a tenant connected to ADLS even if the workspace is not connected. We quickly sent a fix that should be deployed in 7 days or less. Thanks for the flag and for trying out streaming dataflows. I'll let you and everyone else know when this is fixed here and in the blog.