topic Re: Any alternative solutions for Deploying Data Pipeline Connections? in Pipelines

Any alternative solutions for Deploying Data Pipeline Connections?

DanielAmbler — Mon, 10 Jun 2024 13:26:37 GMT

Currently looking at a POC for Fabric and having a few issues with deployment/CICD early on. Ive probably spent a couple of hours this morning coming to the conclusion that Deployment Pipelines are next to useless for Pipelines under certain circumstances.

I have a Data Warehouse in DEV, TEST, PROD that uses a pipeline to populate from a Data Lake. The Connection for the Data Warehouse does not update between workspaces and deployment rules are not yet supported.

When building Azure solutions a common model would be to have a resource group each for DEV, TEST, PROD and deployment options were aplenty typically via Devops. It seems the functionality in Fabric is not yet present to deploy a Data Pipeline between DEV, TEST and PROD workspaces. My question is has anybody yet come across an alternative solution to deploying a pipeline between workspaces before I waste anymore of my time? (e.g Parameterised notebook, external tools) - Deployment pipelines are not ready, connections cannot be parameterised as like linked services, the REST APIs dont seem to have an 'update connection' function and without deploying a full capacity, I cant see how this could easily be done via Azure Devops.

I've come across similar queries and the answer seems to be 'just wait', which is fine, if a little frustrating. At the moment I cant in all honesty say that Fabric is ready for us to use - I'd like to at least feel like i've done a full investigation.

Thanks in advance.

Re: Any alternative solutions for Deploying Data Pipeline Connections?

DanielAmbler — Mon, 10 Jun 2024 14:47:27 GMT

I found a workaround that allows me to proceed, not quite as elegant as having connections at the workspace level but has the desired end result.

On the destination connection, selecting 'Dynamic Input' on the connection type - this exposes the workspace id and sql connection string that can then be parameterised. These values can then be stored as config in a control database (schema within Data Warehouse or file in the Lakehouse) and setup in a previous step in the pipeline. The end result is the same as in the Azure Resource Group based solution mentioned above.

Re: Any alternative solutions for Deploying Data Pipeline Connections?

Anonymous — Tue, 11 Jun 2024 05:12:41 GMT

Hi DanielAmbler ,

Glad to know that you were able to get to a resolution. Please continue using Fabric Community on your further queries.