Forum Discussion
Copy Job source internal error
I was getting the exact same error since yesterday for any warehouse I wanted to use as the source for the Copy Job. I tried connecting to different warehouses with data in it but I got the same internal error.
Later I found that I needed to re-authenticate my warehouse connection that was being used to connect to the source. I went to the 'Manage Connections and Gateways screen', in the Conenctions tab located my connection for the warehouse, saw that it was offline, reauthenticated the connection from the 'settings' option of the connection, and then I was able to resolve the error.
Hope that helps!
Hi jj44,
When you create a Copy Job (or a pipeline Copy activity) using a Lakehouse as the source or destination, Fabric needs to read the schema of the table to generate the mapping automatically.
If the source or destination Lakehouse table is empty (it contains schema metadata but no rows), the Copy Data activity fails because:
The engine cannot infer the schema from the data,
Fabric throws an internal error such as
MissingSchemaForAutoCreateTable or a generic “Internal error”,And in the Copy Job wizard, you see no files or tables discovered.
That’s exactly the behavior you described — even in a brand-new Lakehouse, the Copy Job fails when no data rows exist.
In the Microsoft Fabric Known Issues, there is an known bug on that pipeline can't copy an empty table to lakehouse :
“Currently, copy has a limitation when handling the empty lakehouse table."
To work around this issue, edit the mapping page. Select + New Mapping to add the columns individually.”
Reference : https://support.fabric.microsoft.com/known-issues/?product=Data%2520Factory
If the problem persists, you could use a notebook :
df = spark.read.csv("Files/source.csv", header=True)
df.write.format("delta").mode("overwrite").saveAsTable("DestinationTable")
Hope it can help you !
Best regards,
Antoine
Hi Antoine,
Not quite the same scenario. Whilst trying get to the bottom of the problem, I created a brand new Lakehouse with no tables, just a single file uploaded. But it still failed. But worth knowing about as I create empty tables during development al the time.
Cheers,
Jeff