Forum Discussion
DataflowGen2 in a Data Pipeline is intermittently inserting data twice despite uniqueness measures
Hello, I am building a Data Pipeline to update my table in a data warehouse with a rolling 31 days of data. The data pipeline deletes the last 31 days of data Then it calls on a DataFlow Gen...
i stand corrected that the only time the data is duplicating from the Power Query dataflow gen 2 is in the pipeline. I have had it running independently on a schedule outside the data pipeline and got duplicated results the last 2 nights as well. 😞
I have a measure in place to not insert data into the table if the date already exists, so it shouldn't be loading in twice. I might need to check the results coming from the API call I am using to get the data into the power query.