Copy Job: Using Copy Job reading .parquet files and writing to a fabric warehouse fails

[Jeff_Schreck]

I have several copy jobs (the "copy job" experience) reading .parquet from a lakehouse, these files are deliverd via Golden Gate, and writing them to a fabric warehouse.

I have ~15 but the larger ones (3 of them) eventually fail and have to be rebuilt. When I say larger I mean several hundred columns with hundreds / thousands of row updaes per hour.  

It will run ok for a day or three then fail out with the following error:

ErrorCode=MissingSchemaForAutoCreateTable,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to auto create table for no schema found in Source.,Source=Microsoft.DataTransfer.TransferTask,'

Any advice would be apprecited.  

[Jeff_Schreck]

It seems the solution was to increase the frequency of the Copy Job. 

Overall I am dealing with hundreds of records per minute from Golden Gate, tracking changes. This seems to overwhelm the Copy Job when I am doing incrementals. I increased the frequency from 2 hours to 30 minutes and the problem has not presented itself again. 

[v-veshwara-msft]

Thanks for the update, @Jeff_Schreck . It's helpful to know that increasing the Copy Job frequency resolved the issue.

This makes sense, especially when working with frequent updates from Golden Gate. Running the Copy Job more often reduces the chance of schema issues caused by large data buildup or incomplete files during processing.

To help others who might face a similar issue, please consider marking your reply as the accepted solution.

Thanks again for sharing the resolution.

[v-veshwara-msft]

Hi @Jeff_Schreck ,

Thanks for reaching out to Microsoft Fabric Community.

In addition to what was already shared by @burakkaragoz, there’s a known issue related to pipeline-based Copy Activity in Fabric that shows the same error - MissingSchemaForAutoCreateTable. This occurs when the destination is an empty Lakehouse table and the schema can’t be inferred from the source.

Known issue - Pipeline can't copy an empty table to lakehouse - Microsoft Fabric | Microsoft Learn

While this applies to pipelines, the underlying issue - missing or undetectable schema in the source - may be relevant to Copy Jobs as well. If your job is relying on auto table creation, it’s worth checking if the Mapping step has column definitions set manually using the “+ New Mapping” option. This can help avoid failures caused by inconsistent or missing schema in the source .parquet files.

Also, double-check that none of the .parquet files in the source folder are empty or missing metadata, and confirm that the Golden Gate pipeline preserves schema consistently across all files.

Hope this helps. Please reach out for further assistance.
If this post helps, then please consider to give a kudos and Accept as the solution to help the other members find it more quickly.

Thank you.

[v-veshwara-msft]

Hi @Jeff_Schreck ,

Just checking in to see if you query is resolved and if any responses were helpful. If so, kindly consider marking the helpful reply as 'Accepted Solution' to help others with similar queries. 

Otherwise, feel free to reach out for further assistance.

Thank you.

[burakkaragoz]

Hi @Jeff_Schreck ,

Sorry to hear you’re running into this. For those larger copy jobs, that error usually points to an issue where the parquet files either don’t have a defined schema or the schema isn’t being detected by the copy job process. Sometimes, schema inference fails if the data in your files is too varied or if some files are missing required metadata.

A couple things to check:

• Make sure all your parquet files have the same columns in the same order and that each one has a complete schema definition.
• If possible, try running the job on a single “large” file that’s giving trouble and see if it fails right away. That can help narrow if it’s specific files or the batch size.
• Also, sometimes re-generating the parquet files with explicit schema set (not relying on auto-inference) helps avoid this error.
• If you’re using Golden Gate to deliver, make sure the data pipeline isn’t dropping schema info along the way.

Let me know if you already tried these or if you need some sample script for schema validation.

If my response resolved your query, kindly mark it as the Accepted Solution to assist others. Additionally, I would be grateful for a 'Kudos' if you found my response helpful.