Forum Discussion
Japanese characters become □□□□ when copying data using a Copy Job
Hi everyone,
I’m having an issue when loading data from an on-prem Teradata into a Lakehouse using a Copy Job.
The copy job runs successfully, but any data containing Japanese characters ends up as □□□□ in the destination table. The data containing only English characters look fine.
Has anyone run into this before? Could it be an encoding issue, or is there a specific setting I need to check in the Copy Job or Lakehouse?
Any advice would be greatly appreciated.
P.S: The Japanese data loads correctly when using Dataflow Gen2.
Thanks!
Hi Luriahk ,
Thank you for reaching out to Microsoft Fabric Community and Thanks to timeseriesIQR and Parchitect for sharing meaningful insights
Since the Japanese characters are preserved when using Dataflow Gen2, it suggests that the Lakehouse can store the characters correctly. This indicates the issue is more likely related to the Copy Job ingestion path (such as the Teradata connector, source data types, character encoding during transfer, or schema mapping) rather than the destination itself.
Regarding the Gen AI suggestion, it doesn't appear to be necessary for this scenario, as your goal is to preserve the original Japanese and Chinese characters rather than translate or transform them.
To help narrow down the root cause, could you please confirm the following?
- What Teradata data types are used for the affected columns?
- Are those columns mapped to string-compatible data types in the Copy Job destination?
- Does the issue also occur when copying a small test table containing only Japanese characters?
These details will help determine whether the issue originates from the Teradata source, character encoding during data transfer, or the Copy Job connector.
For Reference:
What is Copy job in Data Factory for Microsoft Fabric?
Best Regards,
Abdul Rafi
10 Replies
Run your data through Gen AI to convert to English if that is how your data is going to be stored.
Thanks for the suggestion. In this case, I need to keep the original Japanese and Chinese data, so I won’t be converting it to English before storage.
If you supply (for example) Copilot with the credentials to your database so that it has access to a table and field in question and if you actually have read write permissions on it, then you can simply ask GEN AI to translate at the source in the table.
- Since Dataflow Gen2 loads the Japanese characters correctly, this looks like a potential connector / driver / encoding issue on the Copy Job path rather than a problem with the Lakehouse itself.Also, just to note — it is perfectly valid to use Dataflow Gen2 purely for ingestion (even without transformations). In fact, using Dataflow to land data in a Lakehouse and then performing transformations downstream is a common Fabric pattern.So as a practical approach:
- you can continue using Dataflow Gen2 to load the data correctly
- treat this as either a workaround or your ingestion layer depending on your design.
That said, a couple of trade-offs to keep in mind vs Copy Job:
- Dataflow can be less efficient for high-volume data movement
- less control over execution/parallelism compared to pipelines
- ALM/CI-CD integration can be more limited depending on your setup
So it’s a good pattern, just worth being aware of the differences.
Thanks for the explanation, I really appreciate the help. I’ll go ahead and use Dataflow Gen2 for the ingestion for now, but I still hope this can be resolved since I need to copy a large number of tables that contain mostly Japanese and Chinese characters.
Let me know if it works..I do not believe it will. But let me know. GEN AI the better option.
Hi Luriahk ,
Thank you for reaching out to Microsoft Fabric Community and Thanks to timeseriesIQR and Parchitect for sharing meaningful insights
Since the Japanese characters are preserved when using Dataflow Gen2, it suggests that the Lakehouse can store the characters correctly. This indicates the issue is more likely related to the Copy Job ingestion path (such as the Teradata connector, source data types, character encoding during transfer, or schema mapping) rather than the destination itself.
Regarding the Gen AI suggestion, it doesn't appear to be necessary for this scenario, as your goal is to preserve the original Japanese and Chinese characters rather than translate or transform them.
To help narrow down the root cause, could you please confirm the following?
- What Teradata data types are used for the affected columns?
- Are those columns mapped to string-compatible data types in the Copy Job destination?
- Does the issue also occur when copying a small test table containing only Japanese characters?
These details will help determine whether the issue originates from the Teradata source, character encoding during data transfer, or the Copy Job connector.
For Reference:
What is Copy job in Data Factory for Microsoft Fabric?
Best Regards,
Abdul RafiHi Luriahk ,
Thank you for reaching out to Microsoft Fabric Community
Just wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.Best Regards,
Abdul Rafi.
Hi Luriahk ,
Could you please confirm if the issue has been resolved? If not, feel free to reach out if you have any further questions.Your update would be helpful for other members who may face a similar issue.
Best Regards,
Abdul Rafi
