Forum Discussion

kallens's avatar
kallens
Frequent Visitor
1 year ago

DataflowGen2 in a Data Pipeline is intermittently inserting data twice despite uniqueness measures

Hello,   I am building a Data Pipeline to update my table in a data warehouse with a rolling 31 days of data.  The data pipeline deletes the last 31 days of data  Then it calls on a DataFlow Gen...
Need Help
Pipeline
lbendlin's avatar
lbendlin
Icon for Super User rankSuper User
1 year ago

what do you consider "today"  and "night"?  Wonder if there are timezones other than UTC involved.

  • kallens's avatar
    kallens
    Frequent Visitor
    1 year ago

    I have the data pipeline set to run at 1 AM Pacific Time Zone (that is the 'night') and then 'today' was when I ran it around 10:23 AM Pacific Time.

     

    what is strange is that the previous night, on it's scheduled run, the data pipeline ran as expected and inserted the correct amount of data, no duplicates. and no material changes to the pipeline between those times.