Forum Discussion
Copy Job - Column Mapping
Hi Magic_Mads
Thank you for reaching out to the Microsoft Fabric Community Forum.
Copy Job failures can occur when loading data into Delta Lake tables if column names contain unsupported special characters, such as slashes ("/") or spaces for example, "Fix Version/s" or "Issue Type". Spark and Delta Lake enforce strict column naming rules, and schema validation is performed before any column mapping or renaming in Copy Job. As a result, jobs will fail if the original schema includes invalid column names.
To avoid this, ensure that all column names are compliant before initiating the Copy Job. For sources like SQL Server, consider creating a SQL view that uses aliases to rename columns (e.g., SELECT [Fix Version/s] AS Fix_Version_s FROM OriginalTable). For file-based sources such as CSV or Parquet, you can utilize a Spark notebook to load the data, rename columns as needed, and write the cleansed data to a new Delta table.
In low-code environments, Dataflow Gen2 in Microsoft Fabric enables column renaming using the Select transformation prior to writing to Delta. Please note that preprocessing through views, notebooks, or dataflows is currently the most effective way to manage unsupported column names, as column mapping in Copy Job alone is not sufficient due to the timing of schema validation.
Regards,
Karpurapu D,
Microsoft Fabric Community Support Team.
Hi Magic_Mads ,
Yeah, this is a pretty common headache when working with Spark/Delta and Copy Job tasks. The "/" character and some others like ":", ";", line breaks and so on just aren't allowed in Delta table column names (or Spark in general), and Copy Job is quite strict about this. Even if you try to do column mapping to a new valid name, the underlying engine still checks the original schema and throws the error if the source has unsupported characters.
From what I know, currently there's no workaround inside Copy Job itself for this – it doesn't remap the columns before schema validation. The only way I got around it in the past was to rename those columns in the source before the Copy Job runs, so the schema is already Spark-friendly. If that's not possible in your flow, some folks script a pre-step to create a view or intermediate table with the cleaned column names, then copy from there.
I haven’t seen any updates about a fix in the pipeline right now, but I’d recommend keeping an eye on the release notes or submitting feedback to Microsoft as well. They occasionally expand support for naming quirks, but for now, it’s a bit of a manual workaround.
Hope that helps a bit! If you want, I can share a sample script for renaming columns in Spark or SQL before running the job.
If my response resolved your query, kindly mark it as the Accepted Solution to assist others. Additionally, I would be grateful for a 'Kudos' if you found my response helpful.
Translation & text editing supported by AI