Forum Discussion
Cannot implement delta updates from upstream Postgres server behind gateway
Hi silly_bird,
To get the MAX value of a column, you could use the lookup activity in combination with a sql server connection. As I can see in your screenshots, you have a lakehouse connection defined. Instead of defining a lakehouse connection, you should set up a sql server connection and use the sql endpoint from the lakehouse in the settings. If you have a sql server connection, you are able to query the table in the lookup activity. In that query, you can directly take the MAX value within your select statement. In the next steps of the data pipeline, you can use the queried max value.
Hi silly_bird,
To get the MAX value of a column, you could use the lookup activity in combination with a sql server connection. As I can see in your screenshots, you have a lakehouse connection defined. Instead of defining a lakehouse connection, you should set up a sql server connection and use the sql endpoint from the lakehouse in the settings. If you have a sql server connection, you are able to query the table in the lookup activity. In that query, you can directly take the MAX value within your select statement. In the next steps of the data pipeline, you can use the queried max value.
Yes, I also found this thought manually defining datalake as Azure Sql Server
Then my activities pipeline looks like that
Parametrisation of the query in the copy was very difficult to figure out, but throught searching browerstack etc. I came up with the following formula:
@concat('SELECT * FROM data.stockmaster',
if(or(equals(activity('stockmaster timestamp').status, 'Failed'),
equals(activity('stockmaster timestamp').output.firstRow.lastmodified, null)),
'',
concat(' WHERE lastmodified > ', '''', string(activity('stockmaster timestamp').output.firstRow.lastmodified), '''')))