Dataflow stop refreshing : "Expression.Error: The field 'X' already exists in the record"

[Fantmas]

Hi everyone,

I have created a dataflow that is running since few weeks, today The schedueld refresh returns me the following error :

Error: Expression.Error: Loading Process ,Failed,Datatable,NA,Failed,2024-02-01 11:05:01,2024-02-01 11:05:20,00:00:18.6100,NA,NA,NA,NA,NA,NA,Error: Expression.Error: The field 'X' already exists in the record,

The table in error is built based on append 4 other tables,

I had a look in previous step and other table append but I have a unique column name for them and even when I close the dataflow "Validation queries" is running successfully, it is only when I run the refresh that was working previously that I am facing this issue,

Any clue how to debugg this ?

[aj1973]

Hi @Fantmas 

Validation queries runs successfully because it is executed on the 1000 rows the Power Query Online is screening but when you run the refresh it fails because it imports the whole data from the 4 Tables.

My guessing is that one of the tables is carrying on the problem. That's why in this business case it would have been better to create 4 dataflows (1 for each table) then create a link dataflow where you merge the 4 of them. That way it is easier to identify the issue.

[Fantmas]

thank you for your reply,

What is weird, is in my test environment the query is running well, the issue happen in production,
so my main question regarding my issue is before the query was running successfully and was refreshing during 2 weeks and today no when I did not change anything

Any clue ?

[aj1973]

Things can happen at the source without your knowledge. what you can do is Delete the step where you appended the tables and run the refresh of the dataflow. see what happens

[Fantmas]

I did it for the fourth table and refresh it one by one, all the table are loaded without error, I had the step with the append query has new table and it is there that I have the issue,

I am do not understand what is happening as the four source table for the append query has a unique column name but when it append the data, I have the error

[aj1973]

Can you show me a screenshot of the Dataflow? The stepas Applied please