Forum Discussion
Issue with Dataflow Gen2 Copy
I am experiencing an issue with copying two tables from one lakehouse (with schema) to another lakehouse. The copy operation fails, but it succeeds when copying without the schema (using the same table name).
Error Message:
There was a problem refreshing the dataflow: 'Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: DataSource.Error: Pipeline execution failed (runId: a3eba383-b856-4821-9ec6-95d2c6c4bdaa). Operation on target ca-9a694c62-cba0-497c-8416-ffe320aea73d failed: Lakehouse table name should only contain letters, numbers, and underscores. The name must also be no more than 256 characters long. Details: Reason = DataSource.Error;RunId = a3eba383-b856-4821-9ec6-95d2c6c4bdaa'. Error code: Fast Copy User Error. (Request ID: e869082f-206a-4ce5-9bc1-ec5cd5a94b83).
one of the Table Names:
- silver.Processsed_4ly_v
I have attempted to rename the table, but the issue persists.
8 Replies
Hi Billybilly
Here I have some suggestions:Ensure that the table name silver.Processsed_4ly_v follows the naming rules:
Only letters, numbers, and underscores are allowed.
The name should not exceed 256 characters.
Avoid special characters or spaces.If you haven't already, try renaming the table to something simpler, like silver.Processed4ly. This can help rule out any issues related to specific characters or naming conventions.
Try copying a very simple table to see if the issue is related to the complexity of the original table.
Check the error log for more detailed information.
Regards,
Nono Chen
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Thank you for your response.
The table name matches the requirement shown in the error message.
I want to clarify that the data copy succeeds when the source is from the lakehouse without a schema. This makes me wonder if there might be a bug in Dataflow Gen 2 or the lakehouse configuration.
I also tried using a simple table name, but the issue persists.
Hi Billybilly
In response to your case, please consider:
The column data types of the source and target tables are the same.
The number and name of columns in the source table and the target table must be the same. If the source table has additional columns or is missing columns, replication may fail. Ensure that the table structure matches exactly in both silos.
Check indexes and constraints (such as primary keys, foreign keys, unique constraints, etc.) on the source and target tables. These constraints should also exist in the target table to ensure data integrity and query performance.
You can test with some simple data.
Regards,
Nono Chen
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Is it possible to set the destition lakehouse to have the same schema as the source? I have seen this issue when using the copy activity in a data pipeline bwteen two lakehouses which do not have the same schema (or when one has a schema and the other does not). I understand that this is not exactly the same, since I am referring to data pipelines and your issue is with Gen2 dataflows, but hopefully this gives you another angle to try.
