Forum Discussion

Billybilly's avatar
Billybilly
Frequent Visitor
1 year ago

Issue with Dataflow Gen2 Copy

I am experiencing an issue with copying two tables from one lakehouse (with schema) to another lakehouse. The copy operation fails, but it succeeds when copying without the schema (using the same tab...
Dataflow
Need Help
Anonymous's avatar
Anonymous
Not applicable
1 year ago

Hi Billybilly 


Here I have some suggestions:

 

Ensure that the table name silver.Processsed_4ly_v follows the naming rules:

 

Only letters, numbers, and underscores are allowed.
The name should not exceed 256 characters.
Avoid special characters or spaces.

 

If you haven't already, try renaming the table to something simpler, like silver.Processed4ly. This can help rule out any issues related to specific characters or naming conventions.

 

Try copying a very simple table to see if the issue is related to the complexity of the original table.

 

Check the error log for more detailed information.

 

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

 

 

  • Billybilly's avatar
    Billybilly
    Frequent Visitor
    1 year ago

    Thank you for your response.

    The table name matches the requirement shown in the error message.

    I want to clarify that the data copy succeeds when the source is from the lakehouse without a schema. This makes me wonder if there might be a bug in Dataflow Gen 2 or the lakehouse configuration.

    I also tried using a simple table name, but the issue persists.

     

    • Anonymous's avatar
      Anonymous
      Not applicable
      1 year ago

      Hi Billybilly 

       

      In response to your case, please consider:

       

      The column data types of the source and target tables are the same.

       

      The number and name of columns in the source table and the target table must be the same. If the source table has additional columns or is missing columns, replication may fail. Ensure that the table structure matches exactly in both silos.

       

      Check indexes and constraints (such as primary keys, foreign keys, unique constraints, etc.) on the source and target tables. These constraints should also exist in the target table to ensure data integrity and query performance.

       

      You can test with some simple data.

       

      Regards,

      Nono Chen

      If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

      • Billybilly's avatar
        Billybilly
        Frequent Visitor
        1 year ago

        Hi NONO Chen,

        Thank you for the response and the detailed suggestions.

        To clarify further, no constraints are applied in this scenario, and there have been no changes to the data types, column names, or the number of columns between the source and target tables. Both tables are identical in structure.

        Let me reiterate the key observation:

        • Copy 1: Exists in the lakehouse with a schema applied.
        • Copy 2: Exists in the lakehouse without a schema applied.
        • It works successfully for Copy 2 but fails for Copy 1.

        Based on this, I suspect the issue may not be related to column data types, names, or constraints.


        Looking forward to your insights!

        Best regards,
        Billybilly