Forum Discussion

MiguelMartinez's avatar
MiguelMartinez
Microsoft Employee
5 years ago

Share your thoughts on streaming dataflows aka SDF (preview feature)

Back at MBAS, our CVP Arun Ulag gave everybody a sneak peek of streaming dataflows, the new experience for real-time data preparation and comsumption in Power BI. Later at BUILD, we showed everyone a...
Dataflow
General Comment
Need Help
Show and Tell
Tips and Tricks
AndyDDC's avatar
AndyDDC
Most Valuable Professional
5 years ago

Hi,

 

I'm trying to work out the "hot" and "cold" storage process.  I have a Streaming Dataflow that I have set the Retention Duration in the Dataflow settings tab to 1 Day.  I have then run that Streaming Dataflow and can see data in the Hot table when connected via Power BI Desktop.  However, after several days I can certainly see the same data in the Cold storage table but the data remains in the Hot storage table.  I have tried stopping/resuming the Streaming Dataflow but the data remains in the Hot table - I would have thought this would not be visible after being moved to Cold storage?

Eklavya's avatar
Eklavya
Microsoft Employee
5 years ago

Hi Andy,

 

The idea is slightly different:
Hot - Keeps data only for the retention duration. While refreshing, data is actively dropped from Hot entities as it moves out of retention.

 

Cold - Data is always kept in cold, regardless of retention.

 

Data is never moved from hot to cold.

  • AndyDDC's avatar
    AndyDDC
    Most Valuable Professional
    5 years ago

    Hi,

     

    The process still isn't very clear to me. 

    So does a message always exist in Cold storage no matter the retention period?

     

    What defines the hot retention period? I would not expect to still see data in the hot table several days after the "1 Day" retention period setting in the Dataflow.

     

    thanks

    • Eklavya's avatar
      Eklavya
      Microsoft Employee
      5 years ago

      > So does a message always exist in Cold storage no matter the retention period?

      Yes.

       

      > I would not expect to still see data in the hot table several days after the "1 Day" retention period setting in the Dataflow.

      Dataflows enforce retention policy only while refresh is running. So if your refresh is not running, you will still see data that is out of retention.
      Do you have a use case where you need the data to be dropped out of retention even when the refresh is not running?

       

      • AndyDDC's avatar
        AndyDDC
        Most Valuable Professional
        5 years ago

        Hi,

         

        OK I understand now after testing.  I see both the same data in the hot and cold storage when I set-up a new Streaming Dataflow,

         

        I have started the older Streaming Dataflow and after a few minutes I now see significantly less rows in the hot table - there are still around 90 rows (which is strange).

         

        I don't have a specific use case but I do think the documentation could be clearer in terms of the differences between hot and cold and under what conditions data will be retained further than the retention point (eg when it's no running).

         

        thanks for the responses, much appreciated