Forum Discussion
Fabric Data Factory pipeline Copy activity failed: MySQL tables upload operation unsuccessful.
- 9 months ago
All Right.
If I were you, I would do the following. You load your data in blocks until the table is completely loaded.
This should be relatively easy to implement.
I know it's not ideal, but at least it would be your solution.This will allow you to bypass any network issues.
However, sometimes even an error message is not always accurate.
Try testing a different destination to rule out the Fabric Warehouse.
- 9 months ago
Hi Gaurav-J ,
Thank you for reaching out to the Microsoft Community Forum.
The error you are facing in Fabric Data Factory when copying large MySQL tables to Fabric Data Warehouse, MySQL server is terminating the connection.
Please try below things to fix the issue.
1. You mentioned that 9,000 rows batches work, By Using SQL queries with LIMIT and OFFSET in the source configuration. Implement pagination logic in the pipeline to loop through batches.
2. In MySQL Server Settings, Check and increase below things.
net_read_timeout
net_write_timeout
max_allowed_packet (set to at least 64MB or higher)
wait_timeout and interactive_timeoutNote: These settings help prevent premature connection drops during large transfers.
3. Instead of selecting the table directly, use a custom SQL query like
SELECT * FROM table_name LIMIT 9000 OFFSET 0
Note: This gives more control and avoids full table scans.
4. In the Copy Activity settings, Enable compression to reduce payload size. Use parallel copy with multiple threads if supported.
5. Use Self-hosted Integration Runtime if you are accessing on-prem MySQL.
6. Check the staging storage account has sufficient permissions and performance. Try to switch to another staging location temporarily to isolate the issue.
7. In Copy Activity, break the pipeline into multiple smaller activities. Use ForEach loop with dynamic ranges.
Please refer below link.
Configure MySQL in a copy activity - Microsoft Fabric | Microsoft Learn
I hope this information helps. Please do let us know if you have any further queries.
Regards,
Dinesh
Gaurav-J ,
All right,
20,000 lines isn't really that much.
Dataflow Gen2, as you say, would be one option. Another option would be the copy job. Have you checked that too?
Have you also checked the connection speed? It seems that your network connection is too slow for copying.
Best regards
spaceman127 Yes I have also tried copy job. I have also tried to do the same process with three different networks. Still I am not able to copy the tables from MySQL database.
Any guidance or suggestions to resolve this issue would be greatly appreciated.
- spaceman1279 months agoSuper User
All Right.
If I were you, I would do the following. You load your data in blocks until the table is completely loaded.
This should be relatively easy to implement.
I know it's not ideal, but at least it would be your solution.This will allow you to bypass any network issues.
However, sometimes even an error message is not always accurate.
Try testing a different destination to rule out the Fabric Warehouse.