Forum Discussion

MiguelMartinez's avatar
MiguelMartinez
Icon for Microsoft Employee rankMicrosoft Employee
5 years ago

Share your thoughts on streaming dataflows aka SDF (preview feature)

Back at MBAS, our CVP Arun Ulag gave everybody a sneak peek of streaming dataflows, the new experience for real-time data preparation and comsumption in Power BI. Later at BUILD, we showed everyone a...
Dataflow
General Comment
Need Help
Show and Tell
Tips and Tricks
MrKrukauskas's avatar
MrKrukauskas
Icon for Helper I rankHelper I
4 years ago

Eklavya MiguelMartinez here are the answers:

I see. Let's do some quick checks here:
1. You've verified that env field does have data coming in from the source?

  • Yes

2. Do you see the data if you remove the filter?

  • I see data with and without filter in the Preview.

3. I am assuming you are using preview to check presence of data - but can you save and run the streaming dataflow and then check if you see the output in the report?

  • Yes, in each step of the dataflow in the preview, i can see streaming and static data, even in the output tables.
  • If I save and run, I only see the output for not nested data.

The Data preview in streaming dataflows: green - able to pull, not nested, red - not able to pull, nested two levels.

 

Thank You for checking this Eklavya 

 

MrKrukauskas's avatar
MrKrukauskas
Icon for Helper I rankHelper I
4 years ago

Just to add, in some cases empty rows are coming through with and without Filter feature.

  • Eklavya's avatar
    Eklavya
    Icon for Microsoft Employee rankMicrosoft Employee
    4 years ago

    MrKrukauskas We tried reproducing the issue by reconstructing a payload similar to your description and it works for us:

    Payload:

     

     

    outputs:

     


    Can you ensure that your payload is valid json?

    To help you further, we will need the payload you are using for your events. The best way to do this is through Power BI Support

  • MrKrukauskas's avatar
    MrKrukauskas
    Icon for Helper I rankHelper I
    4 years ago

    Eklavya we (data engineer from my team and myself) found a solution. Since our event hub consisted of partition 0, partition 1 and those "free fields" and I mainly needed the data from partitions, the data engineer created a new event hub where partition 0 and partition 1 was combined into one and the output looks like only free columns, no second level columns.

    So our guess is that PBI SDF (streaming dataflows) cannot for now handle a bit more complex partitions that include lists, dictionaries.

    Contact me if you need more details.

    Thank You for looking into this!

  • Eklavya's avatar
    Eklavya
    Icon for Microsoft Employee rankMicrosoft Employee
    4 years ago

    Okay, I'll check and get back to you today

  • MrKrukauskas's avatar
    MrKrukauskas
    Icon for Helper I rankHelper I
    4 years ago

    Eklavya Thank You for checking this. I'll make sure to double check if the payload is a valid json. I see that our examples differ in payload structure, so that is probably where the issue is, but I'll check first.