Copy Job source internal error

[jj44]

Hi All,

I've suddenly started getting an error when setting up new Copy Job. In recent days, I've used them to get data from files uploaded into our Lakehouses. But yesterday, after selecting the Lakehouse as a source, I get internal error 

If you switch to Files, No files are discovered. 
I've created a brand new Lakehouse this morning, uploaded a file and attempted to use the Copy Job to retrieve the file data, same outcome!

Anyone got any hints as to how to proceed/debug this?

Thanks,

Jeff

[KautukG · Accepted solution]

I was getting the exact same error since yesterday for any warehouse I wanted to use as the source for the Copy Job. I  tried connecting to different warehouses with data in it but I got the same internal error.

Later I found that I needed to re-authenticate my warehouse connection that was being used to connect to the source. I went to the 'Manage Connections and Gateways screen', in the Conenctions tab located my connection for the warehouse, saw that it was offline, reauthenticated the connection from the 'settings' option of the connection, and then I was able to resolve the error.

Hope that helps!

[jj44]

Yes, that was it. When I checked my connection it was showing as 'Offline' - why did it need reauthenticating is a topic for another discussion, could get interesting with pipelines suddenly failing with vague error messages!

Thanks

Jeff

[AntoineW]

Hi @jj44,

When you create a Copy Job (or a pipeline Copy activity) using a Lakehouse as the source or destination, Fabric needs to read the schema of the table to generate the mapping automatically.

If the source or destination Lakehouse table is empty (it contains schema metadata but no rows), the Copy Data activity fails because:

• The engine cannot infer the schema from the data,

• Fabric throws an internal error such as
  MissingSchemaForAutoCreateTable or a generic “Internal error”,

• And in the Copy Job wizard, you see no files or tables discovered.

That’s exactly the behavior you described — even in a brand-new Lakehouse, the Copy Job fails when no data rows exist.

In the Microsoft Fabric Known Issues, there is an known bug on that pipeline can't copy an empty table to lakehouse :    

“Currently, copy has a limitation when handling the empty lakehouse table."

To work around this issue, edit the mapping page. Select + New Mapping to add the columns individually.”

Reference : https://support.fabric.microsoft.com/known-issues/?product=Data%2520Factory

If the problem persists, you could use a notebook : 

df = spark.read.csv("Files/source.csv", header=True)
df.write.format("delta").mode("overwrite").saveAsTable("DestinationTable")

Hope it can help you ! 

Best regards,

Antoine

[jj44]

Hi Antoine,

Not quite the same scenario. Whilst trying get to the bottom of the problem, I created a brand new Lakehouse with no tables, just a single file uploaded. But it still failed. But worth knowing about as I create empty tables during development al the time.

Cheers,

Jeff