Lookup activity returns incorrect column/values from altered Fabric table

[sharp1FL]

I use a lookup table to get orchestration data from a Fabric table. The table was altered to add a column. If I perform an insert into the table where the added column is not the last value in the insert statement, the lookup will fail.

In this example, the ORCHESTRATION table was altered to add the SRC_FOLDER column. Id = 41 was inserted with SRC_FOLDER as the final column, where Id in (54,55) were not.

Below is part of the the output from Lookup.  SRC_FOLDER should always be "dataEngDev".

{
"count": 39,
"value": [
{
"Id": 54,
"STAGE": "BRONZE",
"DOMAIN": "TEST",
"SRC_FOLDER": "Edw_TEST_OutreachCharges",
"SRC_MASK": "STG_TEST_OUTREACH_CHARGES",
"TGT_TABLE": "CSN",
"TGT_KEY": "1",
"ACTIVE": null,
"NOTEBOOK": "|",
"DELIMITER": "9/23/2024 2:47:35 PM",
"UPDT_DTM": null,
"COL_HEADERS": "dataEngDev"
},
{
"Id": 41,
"STAGE": "BRONZE",
"DOMAIN": "TEST",
"SRC_FOLDER": "dataEngDev",
"SRC_MASK": "Edw_TEST_Appointments_",
"TGT_TABLE": "STG_TEST_APPT",
"TGT_KEY": "CSN",
"ACTIVE": 1,
"NOTEBOOK": null,
"DELIMITER": "|",
"UPDT_DTM": "2024-08-22T14:59:35.490716Z",
"COL_HEADERS": null
},
{
"Id": 55,
"STAGE": "BRONZE",
"DOMAIN": "TEST",
"SRC_FOLDER": "Edw_TEST_Utilization",
"SRC_MASK": "STG_TEST_PRV_UTILIZATION",
"TGT_TABLE": "PROV_NAME",
"TGT_KEY": "1",
"ACTIVE": null,
"NOTEBOOK": ",",
"DELIMITER": "10/17/2024 12:59:14 PM",
"UPDT_DTM": null,
"COL_HEADERS": "dataEngDev"
}
]
}

If I perform a SELECT, the columns are populated appropriately.

The work around is to ensure that SRC_FOLDER is the last column on the insert, but this should not be necessary.

Is this a known issue?

[Anonymous · Accepted solution]

Hi @sharp1FL 

Thank you very much for sharing! It looks like you've found a solution.

There is currently no record of this issue.

In SQL, the order of columns in an INSERT statement must match the order defined in the table schema unless you explicitly specify the column names.

If the new column is added and not included in the INSERT statement, it may default to NULL or another default value, which could lead to unexpected behavior in your lookup.

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[sharp1FL]

The column names were populated in the INSERT statement. Can this be logged as a known issue?