Forum Discussion

Gaurav-J's avatar
Gaurav-J
Frequent Visitor
9 months ago
Solved

Fabric Data Factory pipeline Copy activity failed: MySQL tables upload operation unsuccessful.

Hi, Whenever we try to load tables with a high row count from our on-prem MySQL databases using the Copy Data activity in Fabric Data Factory, we can see the data in the Data Preview option. However...
  • spaceman127's avatar
    spaceman127
    9 months ago

    All Right.

     

    If I were you, I would do the following. You load your data in blocks until the table is completely loaded.
    This should be relatively easy to implement.
    I know it's not ideal, but at least it would be your solution.

    This will allow you to bypass any network issues.

     

    However, sometimes even an error message is not always accurate.

    Try testing a different destination to rule out the Fabric Warehouse. 



  • v-dineshya's avatar
    9 months ago

    Hi Gaurav-J ,

    Thank you for reaching out to the Microsoft Community Forum.

     

    The error you are facing in Fabric Data Factory when copying large MySQL tables to Fabric Data Warehouse, MySQL server is terminating the connection.

     

    Please try below things to fix the issue.

     

    1. You mentioned that 9,000 rows batches work, By Using SQL queries with LIMIT and OFFSET in the source configuration. Implement pagination logic in the pipeline to loop through batches.

     

    2. In MySQL Server Settings, Check and increase below things.

     

    net_read_timeout
    net_write_timeout
    max_allowed_packet (set to at least 64MB or higher)
    wait_timeout and interactive_timeout

     

    Note: These settings help prevent premature connection drops during large transfers.

     

    3. Instead of selecting the table directly, use a custom SQL query like

     

    SELECT * FROM table_name LIMIT 9000 OFFSET 0

     

    Note: This gives more control and avoids full table scans.

     

    4. In the Copy Activity settings, Enable compression to reduce payload size. Use parallel copy with multiple threads if supported.

     

    5. Use Self-hosted Integration Runtime if you are accessing on-prem MySQL.

     

    6. Check the staging storage account has sufficient permissions and performance. Try to switch to another staging location temporarily to isolate the issue.

     

    7. In Copy Activity, break the pipeline into multiple smaller activities. Use ForEach loop with dynamic ranges.

     

    Please refer below link.

    Configure MySQL in a copy activity - Microsoft Fabric | Microsoft Learn

     

    I hope this information helps. Please do let us know if you have any further queries.

     

    Regards,

    Dinesh