Data Ingestion Troubles

Is it an option for you to use SQL and the gateway connector to get your data from the on-prem SQL server and load that in , what i would call, a raw table (for instance tablename:  mydata_raw) and next  use a merge in a notebook to merge your raw data with your existing data? Since you only ingest specific data from your sql server, the overwrite of your "mydata_raw" table, should not impact CU's to much.

The notebook code would look something like below ("name" is just an example column)

%%sql
MERGE INTO mydata AS target
USING mydata_raw AS source
ON target.id = source.id
WHEN MATCHED THEN
UPDATE SET target.name = source.name 
WHEN NOT MATCHED THEN
INSERT (id, name) VALUES (source.id, source.name);

Cheers

Hans