Fabric Pipeline Creating Schemas with Unidentified Folder and Delta files in Lakehouse

[joseph_campbell]

Hi,

I have a pipeline where I have a copy data activity which queries a control table to copy data down from one of our on-premise servers into the Lakehouse we've created in Fabric.  In this control table I specify the schema, table name, and the query I want to use to copy the data down.  I've got the concept working, and on one of our on-premise servers I can do this successfully, when I try this from a different on-premise server when I run the pipeline instead of creating a table, it creates a schema based off the table name and then puts the delta files in an Unidentified folder along with the delta files...

I'm not finding much about why this is occurring on the web, is there a solution for this or has anyone else encountered this issue?  I'm using just one table to try to figure this out with three rows of data, no special characters nothing odd about the table, I find it very weird it's just creating a new schema and not just writing a table under the dbo schema or putting the files in the unidentified folder under the dbo schema.

Any help would be appreciated.

[joseph_campbell · Accepted solution]

I believe I figured out the issue over the weekend, but will be doing some additional testing tomorrow.  Previously, I was connected to SQL server on prem using an on-premise gateway in Fabric using gateway version 3000.234 on one of our VMs which was writing data to the Unidentified data folder inside of it's own schema in Fabric and was unable to write the data correctly into the lakehouse.  The delta parquet files were still visible inside of the sql end point, but the lakehouse looked jacked up and the lakehouse showed Unidentified folder as mentioned previously above.  We were able to update the gateway version over the wekeend to 3000.246 on our VM and it appears it has fixed the issue and I am now able to see the data in the lakehouse exactly as required.

[Anonymous]

HI @joseph_campbell,

I'm glad to hear that you find the root causes and thanks for sharing the workaround here. I think they may helps for other who faced a similar issues.
Regards,

Xiaoxin Sheng

[Anonymous]

Hi @joseph_campbell,

AFAIK, the 'unidentified' issue appeared when data cannot be recognized or getting with not supported data type. What type of data source are you worked on? Can you please share some more detail information about this issue? They should help us clarify your scenario and test to troubleshoot.

Regards,

Xiaoxin Sheng