Forum Discussion

MiguelMartinez's avatar
MiguelMartinez
Icon for Microsoft Employee rankMicrosoft Employee
5 years ago

Share your thoughts on streaming dataflows aka SDF (preview feature)

Back at MBAS, our CVP Arun Ulag gave everybody a sneak peek of streaming dataflows, the new experience for real-time data preparation and comsumption in Power BI. Later at BUILD, we showed everyone a...
Dataflow
General Comment
Need Help
Show and Tell
Tips and Tricks
AndyDDC's avatar
AndyDDC
Icon for Most Valuable Professional rankMost Valuable Professional
5 years ago

Hi,

 

The process still isn't very clear to me. 

So does a message always exist in Cold storage no matter the retention period?

 

What defines the hot retention period? I would not expect to still see data in the hot table several days after the "1 Day" retention period setting in the Dataflow.

 

thanks

Eklavya's avatar
Eklavya
Icon for Microsoft Employee rankMicrosoft Employee
5 years ago

> So does a message always exist in Cold storage no matter the retention period?

Yes.

 

> I would not expect to still see data in the hot table several days after the "1 Day" retention period setting in the Dataflow.

Dataflows enforce retention policy only while refresh is running. So if your refresh is not running, you will still see data that is out of retention.
Do you have a use case where you need the data to be dropped out of retention even when the refresh is not running?

 

  • AndyDDC's avatar
    AndyDDC
    Icon for Most Valuable Professional rankMost Valuable Professional
    5 years ago

    Hi,

     

    OK I understand now after testing.  I see both the same data in the hot and cold storage when I set-up a new Streaming Dataflow,

     

    I have started the older Streaming Dataflow and after a few minutes I now see significantly less rows in the hot table - there are still around 90 rows (which is strange).

     

    I don't have a specific use case but I do think the documentation could be clearer in terms of the differences between hot and cold and under what conditions data will be retained further than the retention point (eg when it's no running).

     

    thanks for the responses, much appreciated