topic Re: Overwrite of SQL Server table into lakehouse fails in Pipelines

Overwrite of SQL Server table into lakehouse fails

amaaiia — Thu, 06 Mar 2025 08:38:52 GMT

Hi,

I have a Data Pipeline in Production that COPIES SQL Server tables into Lakehouse tables. The Pipeline does an overwrite of each table every time. All tables work well except one.

1. I count rows with preview:

2. I configure source and I preview data correctly:

3. I configure destination:

4. Run pipeline and it fails:

I've done some testing:

If I run a bit different query that returns the same number of records, it works:
This query also fails:

This query also fails:

If I preview the data it works, but if I run the Pipeline it fails, always the same error. The only way I've seen it works, is when I filter the query with one filed value. I don't think is a column names issue because the query that is working selects all the columns.

Re: Overwrite of SQL Server table into lakehouse fails

ahmetyilmaz — Thu, 06 Mar 2025 08:48:51 GMT

Hello, have you tried selecting the import schemas option from the mapping section?
If there is an empty column, maybe you can see it here.

Re: Overwrite of SQL Server table into lakehouse fails

amaaiia — Thu, 06 Mar 2025 12:17:46 GMT

Unable to import schema. I click on Import schemas, it says Loading but with no error message, it doesn't import it.

Re: Overwrite of SQL Server table into lakehouse fails

ahmetyilmaz — Thu, 06 Mar 2025 12:22:56 GMT

Try doing your own mapping manually with + New mapping. You already have access to column names and data types.

Re: Overwrite of SQL Server table into lakehouse fails

amaaiia — Thu, 06 Mar 2025 17:32:47 GMT

I ingest some tables inside a loop. I can't specify the schema because it's different for each table. I use the same COPY activity for all the tables, setting table_name as a variable in de COPY activity.

Re: Overwrite of SQL Server table into lakehouse fails

ahmetyilmaz — Fri, 07 Mar 2025 02:48:52 GMT

Can you try what I said just to test the problem? At least we can determine which column is the problem.

Re: Overwrite of SQL Server table into lakehouse fails

v-karpurapud — Mon, 28 Apr 2025 10:34:55 GMT

Hi amaaiia

We are following up to see if your query has been resolved. Should you have identified a solution, we kindly request you to share it with the community to assist others facing similar issues.

If our response was helpful, please mark it as the accepted solution and give a kudos, as this helps other members in community.

Thank you

Re: Overwrite of SQL Server table into lakehouse fails

amaaiia — Tue, 29 Apr 2025 06:03:49 GMT

Finally, I ask SQL Server admin to create the same table in source in a different way. We don't know which was the reason of the failure but changing the table create mode, now it's working.