Forum Discussion

Matthew_G's avatar
Matthew_G
Icon for Advocate I rankAdvocate I
2 years ago
Solved

Dynamic pipelines to use across workspaces

Hi fellow Fabricators,   The set-up: Currently you can't use a deployment pipleline to deploy data pipelines. We have three different workspaces, basically dev, test, prod - we did it this way to ...
pipelines
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi Matthew_G 
    Thanks for using Fabric Community.
    You need to select the "Add dynamic content" option within the drop down and pass in the GUID.

     


    You need to create a new parameter for the stored procedure activity.
    The parameter value should be Lakehouse/Data Warehouse/KQL Database object ID. To get your Lakehouse/Data Warehouse/KQL Database object ID, open your Lakehouse/Data Warehouse/KQL Database in your workspace, and the ID is after /lakehouses/or /datawarehouses/ or /databases/ in your URL.

     

     


    Please refer to this link for more information: Link1
    Hope this helps. Please let me know if you have any further questions.

  • Matthew_G's avatar
    Matthew_G
    2 years ago

    Thanks, it looks a bit different for the Stored Procedure Activity.

    When you add dynamic content to the Warehouse it then also requests a SQL connection string.

    I then added two parameters, the connection string and the warehouse guid, adding it as dynamic content.

    This now works.

     

     

     

     

paul_hasell's avatar
paul_hasell
Regular Visitor
2 years ago

Hi, have you found a way to handle upgrades to the pipeline(s)? Although you can copy across workspaces you can't overwrite an existing pipeline so with this and not being able to 'release' DataFlows we're currently sticking with ADF/Synapse for all bar the simplest requirements (we have some customer integrations with >50 complex pipelines)

Matthew_G's avatar
Matthew_G
Icon for Advocate I rankAdvocate I
2 years ago

Unfortunately not really. For the moment I make the changes to the pipeline in dev and then rename/delete the previous version in the test etc environments and save as the new pipeline then update the variables in the pipeline. So there is still quite a bit of admin needed. Ideally it would be great if you could deploy data pipelines or at least the ability to create workspace environmental variables and use them in the pipelines.

  • paul_hasell's avatar
    paul_hasell
    Regular Visitor
    2 years ago

    I think it's going to get caught in the trap of referencing by id/guid rather than name so there'll always be an issue when moving between workspaces. They may need to take a leaf from the Power Apps book and have some kind of 'solution' package so that related items are managed together to maintain referential 'integrity'