topic Re: trigger a pipeline on change in Pipelines

trigger a pipeline on change

arkiboys2 — Tue, 02 Jul 2024 13:58:23 GMT

I am ingesting data from sql server using fabric Data Factory

How is it possible to trigger a pipeline as soon as the sql server table has upsert?

Thank you

Re: trigger a pipeline on change

NandanHegde — Tue, 02 Jul 2024 15:03:15 GMT

Can you state which SQL server version you are using? That would help you suggest you a better way

Re: trigger a pipeline on change

mariussve1 — Tue, 02 Jul 2024 15:36:34 GMT

Hi,

I have a log table in data Warehouse that logs when sql table is updated. Then you can create a view that sets a flag to 1 or 0 depends if its updated today.

In the pipeline you can just create a lookup and if it returns 1 it continue else stop. Then you can schedule this every 15,30,60 min or What you think is best.

Marius

Re: trigger a pipeline on change

arkiboys2 — Tue, 02 Jul 2024 15:52:50 GMT

using azure sql server

Re: trigger a pipeline on change

NandanHegde — Tue, 02 Jul 2024 15:54:11 GMT

Depending on SQL server version, you can use polybase feature and generate file and then trigger the pipeline via event

https://datasharkx.wordpress.com/2023/05/04/trigger-synapse-azure-data-factory-pipeline-via-sql-server-on-events-within-the-database/

Re: trigger a pipeline on change

NandanHegde — Tue, 02 Jul 2024 15:56:11 GMT

FOr Azure SQL database, please refer the below blog :

https://datasharkx.wordpress.com/2022/12/02/event-trigger-azure-data-factory-synapse-pipeline-via-azure-sql-database/

You can leverage the Fabric data pipeline rest api to trigger it or you can have a blank file created to event trigger the fabric data pipeline

Re: trigger a pipeline on change

arkiboys2 — Wed, 03 Jul 2024 10:38:57 GMT

it look slike azure function is required?

even in fabric data factory there is no straight forward mechanism to trigger a DF pipeline?

thanks

Re: trigger a pipeline on change

NandanHegde — Wed, 03 Jul 2024 13:30:19 GMT

You can use Azure function to generate the token at run time but you can even add the already generated token within database scoped credentails in Azure SQL database which would eliminate the need for Az function

you can go throgh this doc :

https://devblogs.microsoft.com/azure-sql/azure-sql-database-external-rest-endpoints-integration-public-preview/