Forum Discussion
Microsoft EmployeeShare your thoughts on streaming dataflows aka SDF (preview feature)
Back at MBAS, our CVP Arun Ulag gave everybody a sneak peek of streaming dataflows, the new experience for real-time data preparation and comsumption in Power BI. Later at BUILD, we showed everyone an extended demo of the current bits being tried by several of our customers as part of the private preview. And today, finally streaming dataflows is released to the world for public preview with even more updates a new UI.
We would love to hear your feedback and opinions to help us decide what comes next and how to improve this functionality in general. Thanks you so much in advance for your feedback and we will be alert to answer any questions you might have and listen to any ideas you might come up with.
Cheers
The streaming dataflows team
44 Replies
- AndyDDC
Most Valuable Professional
Hi Miguel, I'm currently working through the documentation to get up and running but hitting a snag. I'm using a PPU account and creating a blank workspace with PPU assignment. The streaming dataflow option is greyed out with a message stating "Workspaces connected to a storage account are not supported". This workspace is not connected to Azure nor is the tenant. I've also tried using an Embedded A3 tier attached to the workspace but I get the same message.
- MiguelMartinez
Hey AndyDDC I replied to this in the blog comments too. You found a bug that blocks any workspace in a tenant connected to ADLS even if the workspace is not connected. We quickly sent a fix that should be deployed in 7 days or less. Thanks for the flag and for trying out streaming dataflows. I'll let you and everyone else know when this is fixed here and in the blog.
- AndyDDC
Thank you Miguel
MrKrukauskasHelper I
Hello, does not seem that the streaming dataflows work with deeper than 1 level nested items? F.e. trying to pull the _env field and it does not produce an output. Any ideas?
EklavyaHi, you can pull it by using Manage Fields operator
Hi Miguel,
I am trying to use Streaming Dataflows with an Event Hub connection to bring live weather data into Power BI. When creating the streaming flow, I get data in the data preview for the Event Hub item, but for all others it remains blank (Manage Fields / Filter / Output Table). In all documentation I've seen, you should be able to click on each item in the flow and see a static data view, but somehow it doesn't work for me. Have you got any tips?
Thanks.
- MiguelMartinez
Apologies for the late reply Anonymous Totally missed this message. Let's start checking did you click on the "Get statis preview" button on top of the data preview table? If yes, we might need more details to figure out what's going on. I showed it on the monthly update video too in case you want to take a look.
Hi Miguel, I'm just evaluating SDF in a premium per user workspace. I was able to create the dataflow based on IoT Hub, start it, create a report on top of it in Power BI Desktop, and the report refreshes automatically after I configured change detection. Great! Now when I try to publish this report to exactly the same (ppu) workspace, publishing never finishes. Same applies when I try to upload the pbix file to the workspace: "circle of death" forever, and no error messages. I can easily upload to other non-premium workspaces, but of course I lose the streaming functionality. Do you have any hints for me...?
OK, I have to modify my post a little, after all, this is preview cloud software, and it's changing fast! Today, publishing the report works fine. But as I try to access the report via browser in my ppu workspace, the error message is "The data source Extension is missing credentials and cannot be accessed". I was able to publish there, and I do have a Premium per User license. What can I do?
- MiguelMartinez
Thanks for the flag markusr Let me check with the team and get back you on this. Doesn't seem we have seen this error before but I'll confirm 👍
Hello. When can we expect Streaming dataflows to move out of preview? The feature isn't yet included in "Trusted Microsoft Services" yet so it won't work for us since our firewall settings in Azure only allows trusted microsoft services. I assume it will be "Trusted" once it moves out of preview status?
- AndyDDC
Hi,
I'm trying to work out the "hot" and "cold" storage process. I have a Streaming Dataflow that I have set the Retention Duration in the Dataflow settings tab to 1 Day. I have then run that Streaming Dataflow and can see data in the Hot table when connected via Power BI Desktop. However, after several days I can certainly see the same data in the Cold storage table but the data remains in the Hot storage table. I have tried stopping/resuming the Streaming Dataflow but the data remains in the Hot table - I would have thought this would not be visible after being moved to Cold storage?
- Eklavya
Hi Andy,
The idea is slightly different:
Hot - Keeps data only for the retention duration. While refreshing, data is actively dropped from Hot entities as it moves out of retention.Cold - Data is always kept in cold, regardless of retention.
Data is never moved from hot to cold.
- AndyDDC
Hi,
The process still isn't very clear to me.
So does a message always exist in Cold storage no matter the retention period?
What defines the hot retention period? I would not expect to still see data in the hot table several days after the "1 Day" retention period setting in the Dataflow.
thanks
ShivanCan not see cold table in Power BI Desktop, only hot table (which works perfect).
Re: Hi Anonymous, can you confirm that you're using "Power Platform dataflows" connector and not "Power BI dataflows"? If yes, is there an error that you see or it takes a long time to get data? Usually, loading cold entities take a little while since it pulls in all the historical data.
Retention is set to 7 days, could this be switched off in PowerBI Admin?
Re: Unfortunately no, the max retention policy that is allowed is 7 days (which only pertains to hot entities). You should still be able to tap into historical data using cold entities for data past that 7 day time window.
Shivan, MiguelMartinez , Eklavya - I do not see the option for Power Platform Dataflows, is there a preview feature I am missing
Hello, I'm using streaming dataflow. My data source has a Record field type, but it is not possible to bring it into an output table. It is normal?
Hi,
Neither of the time fields will appear in the input preview. You need to add them manually. is this also true for the deviceID (if there is more then one device connected to the IoThub)??? and how to do that??
- MrKrukauskas
Hello, does not seem that the streaming dataflows work with deeper than 1 level nested items? F.e. trying to pull the _env field and it does not produce an output. Any ideas?
