Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.

Reply
Charley194
Regular Visitor

Issue with Dataflow Gen2 moving data to Azure SQL

I am trying to move data from a Gen1 dataflow into an Azure DB using a Gen2 flow.  I have got this to work and have moved the data.  I did come against this error and I found dealing with this in a own flow was the answer.  Now when I am trying to refresh the data I am coming up with this error.  Note I am trying to replace all data in the table.    

Charley194_0-1700569230303.png

Charley194_1-1700569272009.png

Nothing has change in the SQL DB the data is going to, and nothing in the flow has changed.  Can anyone assist with this issue?  I have checked that there are no errors in the column mentioned.

Just to update, I can get this to work if I recreate the flow.  But when I refresh the flow I get this error.  Note I am not using the flow to create the table.  

12 REPLIES 12
kakaur
New Member

Hi, This might be an issue with Data Gateway version. Try contacting MS Support. We had a similar issue and had to downgrade the Data Gateway version on our end to get it resolved.

Charley194
Regular Visitor

Sure, this is the dataflow, with the M code.  As you can see, it is some sort of issue with the column type on InvoiceDate, but the type Date is the same in the Dataflow and in the Azure SQL so I am a little confused and it did work the first time I use the flow.

Charley194_5-1700652847461.png

 

 



Charley194_1-1700652539034.pngCharley194_2-1700652577788.png

The result 

Charley194_3-1700652628070.pngCharley194_4-1700652646983.png

Destination table in Azure DB

Charley194_6-1700653043499.png

 

 

Can anyone assist with this issue?

Hi @Charley194 

 

Apologies for the delay in response.

I have repro the same scenario from our end and didn't find any issue if the schema is matching. Please find the screenshots attached.

vcboorlamsft_0-1700821839078.pngvcboorlamsft_1-1700821848723.pngvcboorlamsft_2-1700821856262.png

Result:

vcboorlamsft_3-1700821865720.pngvcboorlamsft_4-1700821875348.png

vcboorlamsft_5-1700821974201.png

I tried to mismatch the schema and then got the same error.

vcboorlamsft_6-1700821992017.png

Error result:

vcboorlamsft_7-1700822012920.pngvcboorlamsft_8-1700822019875.png

 

Try to recheck once the schema and try to refresh the dataflow.

 

If the issue still persists please check the gateway version and upgrade it to the latest version. Please refer to Link for more details on the latest gateway version.

 

I hope this helps. If you have any further questions please do let us know.

 

Thanks

 

Hi @Charley194 

 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.

In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help .

 

Thanks

hi, no resolution.  It is now consitently coming up everytime I have a date column, regardless of tables or datasource involve.  The only work around is that I have to re create the Gen2 flow every time I need to get data into the Azure SQL.   I have spoken to a 3rd party and they know what the issue is and can hopefully help.  Should know more by the end of the week

Hi @Charley194 

 

Following up to see if you have a resolution yet. 

If the issue still persists please check the gateway version and upgrade it to the latest version this might help you in resolving the issue. Please refer to Link for more details on the latest gateway version.

In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

 

Thanks.

Hi @Charley194 

 

When using Microsoft Fabric Dataflow Gen2 with an on-premises data gateway, you might encounter issues with the dataflow refresh process. The underlying problem occurs when the gateway is unable to connect to the dataflow staging Lakehouse in order to read the data before copying it to the desired data destination. This issue can occur regardless of the type of data destination being used.

For more details please refer Link

This might help you in resolving the issue.

Please do let us know in case if you have any further questions.

 

Thanks.

Hi sorry for the late reply.

Sadly the two suggestions have not work.  I have confirmed the gateway and we are on the latest verison.  I have confirm there is nothing on the server where the gateway is located that would block outbound traffic.  Though I dont think this is the issue, as this looks like a data type issue

v-cboorla-msft
Community Support
Community Support

Hi @Charley194 

 

Thanks for using Fabric Community.

 

Apologies for the issue you have been facing.

Could you please explain what is the source data here? What do you mean by Dataflow Gen1 here? These details would help me to understand the issue better.

 

Thanks

 

 

Many thanks.  The source is a Dataflow.  So I have loaded the data into a dataflow and I load that into the Gen2 dataflow to get it into the Azure DB.  Does that make sense?  
I have discovered that I can't load the data directly in the Gen2 Dataflow.  

Hi @Charley194 

 

Thanks for the details. Can you please provide some repro steps about how you are loading the data? Some screen shots would also help.

 

Helpful resources

Announcements
Microsoft Fabric Learn Together

Microsoft Fabric Learn Together

Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City

March 2024  FBC Gallery Image

Fabric Monthly Update - March 2024

Check out the March 2024 Fabric update to learn about new features.

April Fabric Community Update

Fabric Community Update - April 2024

Find out what's new and trending in the Fabric Community.