Forum Discussion
Copy Job - Column Mapping
Hi Magic_Mads
Thank you for reaching out to the Microsoft Fabric Community Forum.
Copy Job failures can occur when loading data into Delta Lake tables if column names contain unsupported special characters, such as slashes ("/") or spaces for example, "Fix Version/s" or "Issue Type". Spark and Delta Lake enforce strict column naming rules, and schema validation is performed before any column mapping or renaming in Copy Job. As a result, jobs will fail if the original schema includes invalid column names.
To avoid this, ensure that all column names are compliant before initiating the Copy Job. For sources like SQL Server, consider creating a SQL view that uses aliases to rename columns (e.g., SELECT [Fix Version/s] AS Fix_Version_s FROM OriginalTable). For file-based sources such as CSV or Parquet, you can utilize a Spark notebook to load the data, rename columns as needed, and write the cleansed data to a new Delta table.
In low-code environments, Dataflow Gen2 in Microsoft Fabric enables column renaming using the Select transformation prior to writing to Delta. Please note that preprocessing through views, notebooks, or dataflows is currently the most effective way to manage unsupported column names, as column mapping in Copy Job alone is not sufficient due to the timing of schema validation.
Regards,
Karpurapu D,
Microsoft Fabric Community Support Team.
Hi Magic_Mads
We have not yet received a response from you regarding your query if your resolved ?If it did not, please share more details so we can assist you more effectively.
Thank You.
I still have not clear answer of wether SQL Server Mirroring will be able to handle columnnames containing "/"
Hi Magic_Mads
Thank you for reaching out to the Microsoft Fabric Community Forum.
Copy Job failures can occur when loading data into Delta Lake tables if column names contain unsupported special characters, such as slashes ("/") or spaces for example, "Fix Version/s" or "Issue Type". Spark and Delta Lake enforce strict column naming rules, and schema validation is performed before any column mapping or renaming in Copy Job. As a result, jobs will fail if the original schema includes invalid column names.
To avoid this, ensure that all column names are compliant before initiating the Copy Job. For sources like SQL Server, consider creating a SQL view that uses aliases to rename columns (e.g., SELECT [Fix Version/s] AS Fix_Version_s FROM OriginalTable). For file-based sources such as CSV or Parquet, you can utilize a Spark notebook to load the data, rename columns as needed, and write the cleansed data to a new Delta table.
In low-code environments, Dataflow Gen2 in Microsoft Fabric enables column renaming using the Select transformation prior to writing to Delta. Please note that preprocessing through views, notebooks, or dataflows is currently the most effective way to manage unsupported column names, as column mapping in Copy Job alone is not sufficient due to the timing of schema validation.
Regards,
Karpurapu D,
Microsoft Fabric Community Support Team.