topic Re: Progrmatically create pipeline in Pipelines

Progrmatically create pipeline

smpa01 — Mon, 12 Aug 2024 15:55:51 GMT

Is there any way to code the pipeline activities rather than using the graphical interface at all.

Such as

//pseudocode in javascript where fnCopyActivity and fnNotebbok are in-built function calls //representative of Copy Activity and Notebook execution available in the graphical interface //set variable const setVariable1 = 'prod_db' //define fnCopyActivity parameters const copyParameters = [{ source: { connection: 'on_prem_sql', query: 'select * from `{setVariable1}`' } }, { destination: { connection: 'lakehouse', files: 'Files/daily' } } ] //execute copy activity const copyActivity = fnCopyActivity(copyParameters) //take nly output ids const copyOutputArray = copyActivity.map(a => a.output.id) //execute forEach copyOutputArray.forEach(a => { fnNotebbok(NB1(src=a)) })

With promise chaining

// Execute copy activity fnCopyActivity(copyParameters) .then((copyActivity) => { // If successful, extract the output ids const copyOutputArray = copyActivity.map(a => a.output.id); // Return the output array for the next .then() in the chain return copyOutputArray; }) .then((copyOutputArray) => { // If successful, execute fnNotebbok for each id return Promise.all( copyOutputArray.map(id => fnNotebbok(NB1(src=a)) ); }) .then((notebookResults) => { // All fnNotebbok executions were successful console.log('All notebooks executed successfully', notebookResults); }) .catch((error) => { // Handle any errors that occur at any stage of the chain console.error('An error occurred:', error); });

Thank you in advance

Re: Progrmatically create pipeline

NandanHegde — Mon, 12 Aug 2024 15:37:26 GMT

Data pipelines in the backend are basically json formats.

You can code the pipelines rather than ui drag /drop in case if you are familiar with the data pipeline json format

Re: Progrmatically create pipeline

smpa01 — Mon, 12 Aug 2024 15:55:24 GMT

Thanks.

Yes, you can feed a json that would create a pipeline. But it is harder to manually create a json that would have all the correct info that is conducive to the desired pipeline.

What I am looking for is a progrmatic way to create pipeline through some python/powershell libraries that already has all the pre-defined pipeline methods. So that I can invoke the methods with correct parameters and promise chain (on success, fail etc) and be able to schedle executing the code.

By programming it, I would then create the pipeline and the correct json. You can edit the pipeine from the code itself.

Otherwise, there is no way to create a json manually with the exact same pipeline in your head.

Re: Progrmatically create pipeline

frithjof_v — Mon, 12 Aug 2024 20:42:36 GMT

How can we do it in Fabric. Is this how you would achieve it? (Link below)

https://www.red-gate.com/simple-talk/blogs/edit-fabric-pipeline-json/

Re: Progrmatically create pipeline

smpa01 — Mon, 12 Aug 2024 21:40:26 GMT

When you crate a pipeline it generates a json, you can edit the json. But there is no way you can manually create a json from scratch that would represents the json otherise created by using the UI.

In a perfect fabric world, I would code to create the pipeline which derives the json and not the other way round.

Re: Progrmatically create pipeline

frithjof_v — Mon, 12 Aug 2024 21:54:09 GMT

"But there is no way you can manually create a json from scratch that would represents the json otherise created by using the UI."

I think you can: https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-rest-api

Re: Progrmatically create pipeline

Anonymous — Tue, 13 Aug 2024 01:18:29 GMT

Hi smpa01 ,

In addition to the ideas and comments mentioned by the two above, I think you can also try the following:
1. You can create and manage data factory pipelines programmatically using SDKs for .NET, Python, and other languages.

2. You can also use the Azure CLI and PowerShell, command-line tools that allow you to manage Data Factory resources through scripts, including creating and updating pipelines.

Read this official documentation below for more information: Use custom activities in a pipeline - Azure Data Factory & Azure Synapse | Microsoft Learn

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Progrmatically create pipeline

mbourgon — Fri, 18 Apr 2025 20:19:14 GMT

https://learn.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory-powershell seems like it would work, too. But the biggest problem is that you need to do a fair amount of JSON, and then essentially use PoSH to import the configs.

Re: Progrmatically create pipeline

smpa01 — Sat, 19 Apr 2025 20:49:41 GMT

Pipeline triggers proxy as opposed to redirect. In other words, pipeline is more expensive than notebook. If the cost were equal, it was worth the effort.

redirect vs proxy