Forum Discussion

sangeetaraman's avatar
sangeetaraman
Advocate I
3 years ago

Dataflow stopped saving historical data

Hello,

 

I had set up a Dataflow (source being Azure Databricks) recently and set it up for refresh everyday. The source displays only current day's data, meaning Azure gives only today's data. In order to save the historical data, I had set up a dataflow which was also working fine by "appending" today's data to past data. I suddenly notice since two days that this has stopped and now each day, the data is "replaced" by current day's data from Azure. I need the daily new data to be appended into the dataflow instead of getting replaced as my visuals need to be showing history as well. Please guide me where I'm going wrong. Sharing the refresh settings set up currently:

Thanks

Sangeeta

Need Help

4 Replies

    • v-xiaosun-msft's avatar
      v-xiaosun-msft
      Community Support
      3 years ago

      Hi sangeetaraman ,

       

      Are the settings now the same as when it worked before? Have any changes been made?

      When you need Historical Data Preservation using Power BI Dataflow, Create two entities, one for storing transactional data and another for storing historical data.

      Entity for transactional data:

      Always stores data for the current year. Once the entity is created, schedule it daily as needed, so as to initiate the incremental refresh.

      Entity for historical data:

      Only stores the historical data for the previous year and older. This entity can be scheduled if needed or can be triggered manually once a year as needed.

       

      You can reference the following document which may be helpful to you.

      Historical Data Preservation using Power BI Dataflow - Visual BI Solutions

       

      Best Regards,
      Community Support Team _ xiaosun

      If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

      • sangeetaraman's avatar
        sangeetaraman
        Advocate I
        3 years ago

        Hello v-xiaosun-msft ,

         

        Thanks for your response. There was a change introduced in the source- Two new columns were inserted into the same query. I also see another error which was not the case earlier, "Unable to confirm if M Query can be folded"-screenshot below. Can you please guide me how to rectify this?

         

        Thanks

        Sangeeta

  • sangeetaraman's avatar
    sangeetaraman
    Advocate I
    3 years ago

    Hi everyone,

     

    An update to my query above- My dataflow is not refreshing incrementally-each day after the refesh the data gets replaced by today's data. But in the desktop, the incremental refresh is working fine. This is strange, can anyone help me on the difference between incremental refresh in the Power BI service and Power BI desktop? What will the consequences of this be after I publish my report online?

    Any help is appreciated.

     

    Rgds

    Sangeeta