Forum Discussion
Helper IStreaming Dataflow does not produce and output
Hello PBI Community
Here's the the issue:
I have created a streaming dataflow that uses Event Hub to stream ticket sales data. It uses single Event Hub, multiple Manage Fields, multiple Filters and then creates multiple output tables.
Event Hub schema:
It uses 2 partitions and then free fields (not sure the exact name for those fields.): partition 0, partition 1 and then scan_entry in this case.
Each partition contains other fields:
When I use manage fields to pull the fields I need, it works for free fields, but not for the fields in each partition.
Some more details.
- I have verified that there's data coming through from event hub.
- I see data in each step/component of streaming dataflow in preview.
- I'm able to produce and view the static data in the output tables.
- Once I start the streaming dataflow it produces all the output tables, but only the table, that pulls Free fields is being populated. Others are either empty or have empty values in it (produces empty rows)
- I have tried all combinations to pull data; from each partition and free field separately, combined, mixing all together, using filters, not using filters and only works when I do it for the free fields.
- I'm using the correct Dataflow when adding data to PBI Desktop.
- I'm using DQ when importing data.
- I'm refreshing the report every 10 seconds to see the updates, and I do (for only the Free fields table).
- Data preview below shows which fields I'm able to pull and which I'm not.
My Streaming Dataflow:
Appreciate anyone who took time to view this post - You're the best!
HELP!!! 😄
Anonymous we (data engineer from my team and myself) found a solution. Since our event hub consisted of partition 0, partition 1 and those "free fields" and I mainly needed the data from partitions, the data engineer created a new event hub where partition 0 and partition 1 was combined into one and the output looks like only free columns, no second level columns.
So our guess is that PBI SDF (streaming dataflows) cannot for now handle a bit more complex partitions that include lists, dictionaries.
Contact me if you need more details.
Thank You for your help!
How the output looks now:
3 Replies
Hi MrKrukauskas
So have you compared the difference between the fields that can be pulled and the fields that cannot be pulled ?
Best Regards,
Community Support Team _ Ailsa Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
MrKrukauskasAnonymous the fields that cannot be pulled are from valid json files. The fields that can - simple columns. From many examples that I have seen, it should be possible to pull individual fields from json and free fields together. That does not happen in this case.
- MrKrukauskas
Anonymous we (data engineer from my team and myself) found a solution. Since our event hub consisted of partition 0, partition 1 and those "free fields" and I mainly needed the data from partitions, the data engineer created a new event hub where partition 0 and partition 1 was combined into one and the output looks like only free columns, no second level columns.
So our guess is that PBI SDF (streaming dataflows) cannot for now handle a bit more complex partitions that include lists, dictionaries.
Contact me if you need more details.
Thank You for your help!
How the output looks now:
