Copy Job - Column Mapping

Hi Magic_Mads 

Thank you for reaching out to the Microsoft Fabric Community Forum.

Copy Job failures can occur when loading data into Delta Lake tables if column names contain unsupported special characters, such as slashes ("/") or spaces for example, "Fix Version/s" or "Issue Type". Spark and Delta Lake enforce strict column naming rules, and schema validation is performed before any column mapping or renaming in Copy Job. As a result, jobs will fail if the original schema includes invalid column names.

To avoid this, ensure that all column names are compliant before initiating the Copy Job. For sources like SQL Server, consider creating a SQL view that uses aliases to rename columns (e.g., SELECT [Fix Version/s] AS Fix_Version_s FROM OriginalTable). For file-based sources such as CSV or Parquet, you can utilize a Spark notebook to load the data, rename columns as needed, and write the cleansed data to a new Delta table.

In low-code environments, Dataflow Gen2 in Microsoft Fabric enables column renaming using the Select transformation prior to writing to Delta. Please note that preprocessing through views, notebooks, or dataflows is currently the most effective way to manage unsupported column names, as column mapping in Copy Job alone is not sufficient due to the timing of schema validation.

Regards,

Karpurapu D,

Microsoft Fabric Community Support Team.