Forum Discussion

CrazyHorse66's avatar
CrazyHorse66
Helper I
4 years ago
Solved

Dataflows incremental refresh

Hi All

 

I have been looking at this for far too long now, any help gratefully received.

1. Created a simple dataflow in a Premium Workspace - connect to SQL DWH and pull all data (it's a very small dataset to test with, 102 rows only - no where clauses just select fields a,b,c,d from table z)

2. Set up Incremental Refresh using CreatedDate - to store rows from the past 1 month, refresh from the past 7 days - should be 26 rows only - against a DateTime field

3. Refreshes the dataset but I still have 102 rows

4. The RangeStart value has been created but not accounting for the refresh schedule:

#datetime(0001,01,01,00,00,00) meta [IsParameterQuery=true, List={}, Type="DateTime", IsParameterQueryRequired=true, Description="autogenerated_for_incremental_refresh"]
5. The RangeEnd value has been created but not accounting for the refresh schedule:
#datetime(9999,12,31,23,59,59) meta [IsParameterQuery=true, List={}, Type="DateTime", IsParameterQueryRequired=true, Description="autogenerated_for_incremental_refresh"]
 
What am I doing wrong? Why is the data not being reduced?
 
Thanks

 

Dataflow
  • lbendlin's avatar
    lbendlin
    4 years ago

    3. did you mean to say "refreshed the dataflow" ?

     

    How many times did you refresh the dataflow?  Incremental refresh only kicks in at the second refresh.

    How many partitions are reported in the refresh history log file?

11 Replies

  • lbendlin's avatar
    lbendlin
    Super User
    4 years ago

    3. did you mean to say "refreshed the dataflow" ?

     

    How many times did you refresh the dataflow?  Incremental refresh only kicks in at the second refresh.

    How many partitions are reported in the refresh history log file?

    • CrazyHorse66's avatar
      CrazyHorse66
      Helper I
      4 years ago

      Hi and thanks

      Yes I meant it successfully refreshes but the data does not get filtered

      I've tried multiple refresh and the data stays the same, RangeStart & RangeEnd do not have the dates I have selected for Incremental Refresh

      It looks like there are 7 partitions but none are being processed?

       

       

      • lbendlin's avatar
        lbendlin
        Super User
        4 years ago

        Please post the last line of the power query.

  • lbendlin's avatar
    lbendlin
    Super User
    4 years ago

    Can you confirm that your sample's [Created Date]  values fall into the buckets defined by the partitions?

    • CrazyHorse66's avatar
      CrazyHorse66
      Helper I
      4 years ago

      Hi, here's the 'raw' data, dates in September 2021

      And here's what I set for incremental to test, but nothing changes

      Strange though it has the values twice in dropdown to filter - Created Date

       

      • lbendlin's avatar
        lbendlin
        Super User
        4 years ago

        That would be your answer.  Your source data is ambiguous with two columns having the same name, and the Power BI service got confused and chose the wrong one.

  • Anonymous's avatar
    Anonymous
    Not applicable
    4 years ago

    Hi CrazyHorse66 ,

     

    Try applying a filter on the Created Date column and check the incremental refresh again.

     

    Best Regards,

    Jay

    • CrazyHorse66's avatar
      CrazyHorse66
      Helper I
      4 years ago

      Update: I started all over again and as suggested needed to rerun the refresh 2 or 3 times to take effect - thanks