Dataflow problem with appending to existing table

[suidino2]

Dataflow gen 2 with On Premise SQL is troublesome. 

The scenario is that

1. I have a Dataflow Gen 2 (DF2) which loads data from on premise SQL (by using gateway the latest version) to lakehouse for those data from year of 2010 to 2015. This DF2 has no problem. 

Then,

2. I created another DF2 using the same M code to load data from the same on premise SQL table for data fitlered between year 2016 to 2020 to existing lakehouse table created by previous DF2. But I got this error.

"Null Error: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: An error happened while reading data from the provider: 'A transport-level error has occurred when receiving results from the server." 

Before I reach to this scenario, I have also encountered gateway issue and firewall network issue described in other thread. 

The overall impression is that I felt a bit unstable using DF2 especially when it needs to work with on premise SQL....

And other than DF2, we dont have any other option to laod data from on premise to Fabric (Lakehouse), it feels stressful.

Hope MS can develop and enable connector of on premise SQL in the pipeline Copy Activities soon. 

[suidino2]

It turned out that it wa because I didnt pay good attention to the message.

When pulling data from source and to create a table in Lakehouse, Lakehouse will automatically change the column type to whole number if the column has only 0 value even you set the column type as decimal. Because of that, the second append dataflow will cause a column type inconsistency. Meaning this particular column in your dataflow is set as decimal type, but the existing table in lakehouse is actually whole number. Coumn type inconsisency. this was the main issue cuasing the error. 

So, eventually instead of 

[SidJay]

Could you please try upgrading to the latest Gateway (version 3000.178.9 released on June 15th). That should help address at least one class of errors involving output destinations and/or queries that depend on other staged queries.

Please let us know if the issues persist after that. Thanks.