Update warehouse artifactId and workspaceId integrating git repo to new workspace

[PeterDaniels]

I have a "source" Fabric workspace. It's integrated with a github git repo. It includes a warehouse and stored procedures along with data fractory pipelines that call the stored procedures.  I want to use this repo as a "starter" for creating new workspaces. So, I create d a new workspace, integrated it with a branch from this repo, and then sync to populate the new workspace.  One issue I'm running into is that all of the stored procedure calls from data factory pipelines have the "old" workspaces workspaceId (GUID) and warehouse artifactId (GUID). So, I have to manuallu edit the JSON to replace those with the new workpsace's worskpaceId and new warehouse's artifactId. I'd like to find a way to automate this process when the new workspace is initially sync'd from the git repo.

[nilendraFabric]

Hi @PeterDaniels 

The GitHub Action will use the service principal credentials to authenticate and connect to the new workspace. Ensure the service principal has the necessary permissions in the new workspace.

GitHub Actions Workflow

Create a GitHub Actions workflow that triggers when syncing to a new workspace. This workflow will:
1. Authenticate with Microsoft Fabric using a service principal
2. Use the Fabric REST API to retrieve the new workspaceId and warehouse artifactId
3. Update the pipeline JSON files with the new IDs
Authentication
Use Azure AD authentication to obtain an access token for the Fabric REST API. Store the necessary credentials (client ID, client secret, tenant ID) as GitHub secrets.
Retrieving New IDs
Use the Fabric REST API to get the new workspaceId and warehouse artifactId:
1. List items in the workspace to find the warehouse
2. Get the warehouse details to retrieve its artifactId
Updating Pipeline Files
Use a script within the GitHub Action to:
1. Parse the pipeline JSON files
2. Replace the old workspaceId and artifactId with the new values
3. Commit and push the updated files back to the repository

please try this option and see if this works 

thanks

[nilendraFabric]

hello @PeterDaniels could you please keep me posted if this works and accept this solution if query is resolved 

thanks

[PeterDaniels]

I have a tool, written PowerShell, but it's not yet plugged into a github action.  Not sure it should be, either. Here's the process:

1. After the initial sync of a fork of my framework repo to the workspace, the warehouse "builds", getting it's new artifactId and endpoint.
2. I then sync/commit the workspace back to the github repo.
3. Then, I clone the repo locallly.
4. I then get the new workspaceId, warehouse artfiactId and endpoint from the workspace (wil automate this at some point)
5. Feed those three items into my powershell script which replaces all of the references to the "old" warehouse in the various pipeline-content.json files with the new Ids.
6. stage, commit, push - the updated files
7. Go back into Fabirc and update the workspace from the repo
8. Also need to then commit changes back up to the repo (not exactly sure why, but seems like the workspace getting the updated JSON jiggles some bits that make it think there are some changes that need to be commited - no big whoop.
9. To test, open the pipelines and verify that the connections to the warehouse are all OK.

I may do some more automation, but this works and reduces the time and potential for error in manually updating all of the pipelines.

[nilendraFabric · Accepted solution]

Hi @PeterDaniels 

To automate updating the workspaceId and artifactId when syncing a git repository to a new Microsoft Fabric workspace, you’ll need to implement a custom solution. Unfortunately, there isn’t a built-in feature in Microsoft Fabric to automatically update these IDs during the sync process.

To automate the process of updating the workspaceId and warehouse artifactId when syncing a new workspace from your “starter” git repo, you can implement a script that runs after the initial sync. This script would update the JSON files containing the pipeline definitions. Here’s a high-level approach to achieve this:
1. Create a PowerShell script that does the following:
3. a. Retrieves the new workspaceId and warehouse artifactId from the newly created workspace.
b. Searches for all JSON files in the synced repository that contain pipeline definitions.
c. Updates the workspaceId and artifactId in these files.
2. Use a GitHub Action to run this script automatically after syncing the new workspace.

Worth trying this as well, using the Microsoft Fabric REST API. I haven't tried this though.

if this helps please give kudos and accept the solution.

thanks

POST https://api.fabric.microsoft.com/v1/workspaces/{newWorkspaceId}/items

Payload 

{
"displayName": "My New Pipeline",
"type": "DataPipeline",
"definition": {
"properties": {
"activities": []
}
}
}

PATCH https://api.fabric.microsoft.com/v1/workspaces/{newWorkspaceId}/items/{newArtifactId}

[PeterDaniels]

Thank you, @nilendraFabric . I was wondering if a REST API solution might work. So, if I'm hearing you correclty, we might have two options: 1) Use powershell or similar to just do a find and replace in the JSON source files via github action. 2) Use REST API to patch/update the pipelines with the new IDs.

Would the REST Solution also be a github action?  How des the github action connect to the new workspace to get the new IDs?