Forum Discussion

CrhIT's avatar
CrhIT
Frequent Visitor
2 years ago

API data changed and it's not compatible with my warehouse table

I made a pipeline that GET data from an API. At first the table was built with the data extracted. Now the data retrieve from the API changed its structure and thus the pipeline is retrieving errors....
General Comment
General Question
Need Help
pipelines
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi CrhIT ,

The error message indicates that the column ‘orders.items.modifiers.item_id’ is not found in your SQL Data Warehouse table ‘[dbo].[SaintRemi]’. This can happen if there’s a mismatch between the source and sink schema or if the column name is incorrect.

 

I think spelling and case issues also need to be checked when ensuring that 'orders.items.modifiers.item_id' exists.Another thing is that in the copy activity, go to the Mapping tab and ensure that the source and sink columns are correctly mapped. You can use the properties ‘name’, ‘path’, or ‘ordinal’ to reference the columns/fields. You can look at this document: Schema and data type mapping in copy activity - Azure Data Factory & Azure Synapse | Microsoft Learn

 

 

 

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

  • CrhIT's avatar
    CrhIT
    Frequent Visitor
    2 years ago

    Thank you very much for your reply, the thing is that the API add a new child in its json and the data warehouse didn't have that column when I first built it, hence I cannot mapped it now. I was wondering if there's a work arroound to ignore this "new column" cause I don't truly need it or to create a new column in an existing table in warehouse