KNP's avatar
KNP
Icon for Super User rankSuper User
3 years ago
Status:
Investigating

Datamart Incremental Refresh not updating current day based on my time zone

I have incremental refresh setup on a Datamart.

 

 

My timezone is UTC +12 currently.

When the scheduled refresh runs it does not pickup the current days transactions until after midday.

 

 

When a refresh is done at 11:59am the latest transaction date is showing before midnight the previous day.

 

When the refresh is done just after midday the latest transaction date shows just after midday (all transactions between midnight and midday are updated).

 

 

As you can see, there were many transactions between midnight and midday that weren't updated until after midday local time.

 

Need a fix for this ASAP.

 

4 Comments

  • KNP's avatar
    KNP
    Icon for Super User rankSuper User

    Thanks for the reply.

    That link has nothing to do with this issue.

     

    Also, I should have mentioned above, Dataflows do not suffer from this issue. Once a scheduled refresh is setup for a Dataflow in a timezone, the timezone is respected and refreshes will pick up the current days data irrespective of the time.

     

    The issue is Datamart incremental refresh is not working this way and I cannot get today's data until after midday no matter what I do.

     

    This is a bug.

     

  • PierreL's avatar
    PierreL
    Frequent Visitor

    I am having the same issue on a dataset, I am currently in UTC+7 and suddenly we see that the refresh that we normally do at 6.30am will not contain today's data. when checking the partitions on the table we see that no partition is created for today until we pass 7am. 

  • KNP's avatar
    KNP
    Icon for Super User rankSuper User

    Hi PierreL,

     

    I went through the support process with them.

    The workaround currently is to add a column by converting your date time column to UTC.

    Something like this...

     

    Table.TransformColumnTypes(
        Table.AddColumn(
            PreviousStep, 
            "YourDTColumnUTC", 
            each [YourDTColumn] - #duration(0, 7, 0, 0)
        ), 
        {{"YourDTColumnUTC", type datetime}}
    )
    

     

    Then use this new column for your incremental refresh.

     

    Hope this helps.