Issue with Dataflow Gen2 moving data to Azure SQL

[Charley194]

I am trying to move data from a Gen1 dataflow into an Azure DB using a Gen2 flow.  I have got this to work and have moved the data.  I did come against this error and I found dealing with this in a own flow was the answer.  Now when I am trying to refresh the data I am coming up with this error.  Note I am trying to replace all data in the table.    

Nothing has change in the SQL DB the data is going to, and nothing in the flow has changed.  Can anyone assist with this issue?  I have checked that there are no errors in the column mentioned.

Just to update, I can get this to work if I recreate the flow.  But when I refresh the flow I get this error.  Note I am not using the flow to create the table.  

[kakaur]

Hi, This might be an issue with Data Gateway version. Try contacting MS Support. We had a similar issue and had to downgrade the Data Gateway version on our end to get it resolved.

[Charley194]

Sure, this is the dataflow, with the M code.  As you can see, it is some sort of issue with the column type on InvoiceDate, but the type Date is the same in the Dataflow and in the Azure SQL so I am a little confused and it did work the first time I use the flow.

The result 

Destination table in Azure DB

[Charley194]

Can anyone assist with this issue?

[v-cboorla-msft]

Hi @Charley194 

Apologies for the delay in response.

I have repro the same scenario from our end and didn't find any issue if the schema is matching. Please find the screenshots attached.

Result:

I tried to mismatch the schema and then got the same error.

Error result:

Try to recheck once the schema and try to refresh the dataflow.

If the issue still persists please check the gateway version and upgrade it to the latest version. Please refer to Link for more details on the latest gateway version.

I hope this helps. If you have any further questions please do let us know.

Thanks

[v-cboorla-msft]

Hi @Charley194 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.

In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help .

Thanks

[Charley194]

hi, no resolution.  It is now consitently coming up everytime I have a date column, regardless of tables or datasource involve.  The only work around is that I have to re create the Gen2 flow every time I need to get data into the Azure SQL.   I have spoken to a 3rd party and they know what the issue is and can hopefully help.  Should know more by the end of the week

[v-cboorla-msft]

Hi @Charley194 

Following up to see if you have a resolution yet. 

If the issue still persists please check the gateway version and upgrade it to the latest version this might help you in resolving the issue. Please refer to Link for more details on the latest gateway version.

In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks.

[v-cboorla-msft]

Hi @Charley194 

When using Microsoft Fabric Dataflow Gen2 with an on-premises data gateway, you might encounter issues with the dataflow refresh process. The underlying problem occurs when the gateway is unable to connect to the dataflow staging Lakehouse in order to read the data before copying it to the desired data destination. This issue can occur regardless of the type of data destination being used.

For more details please refer Link

This might help you in resolving the issue.

Please do let us know in case if you have any further questions.

Thanks.

[Charley194]

Hi sorry for the late reply.

Sadly the two suggestions have not work.  I have confirmed the gateway and we are on the latest verison.  I have confirm there is nothing on the server where the gateway is located that would block outbound traffic.  Though I dont think this is the issue, as this looks like a data type issue

[v-cboorla-msft]

Hi @Charley194 

Thanks for using Fabric Community.

Apologies for the issue you have been facing.

Could you please explain what is the source data here? What do you mean by Dataflow Gen1 here? These details would help me to understand the issue better.

Thanks

[Charley194]

Many thanks.  The source is a Dataflow.  So I have loaded the data into a dataflow and I load that into the Gen2 dataflow to get it into the Azure DB.  Does that make sense?  
I have discovered that I can't load the data directly in the Gen2 Dataflow.  

[v-cboorla-msft]

Hi @Charley194 

Thanks for the details. Can you please provide some repro steps about how you are loading the data? Some screen shots would also help.