How to automate Fabric Deployment Rule creation for a Deployment Pipeline?

[MLUM]

Hi!

So my current situation is this, in Fabric, we have 3 workspaces (Dev, Stage, Prod) with it's own Lakehouse in each. We then use a few scripts that hit the Deployment Pipeline API to push our items between workspaces using a Fabric Deployment Pipeline.

Since semantic models are very annoying to push (you have to push the model through, then set a deployment rule to change the lakehouse connection, then re-push it through for that to take effect, then go into settings and select the connection you want), we need a way to speed this up. 

I was wondering if there is a way to either A) set a global Deployment Pipeline rule for a connection or parameter so that all semantic models will automatically switch from our dev lakehouse in the dev workspace to the stage lakehouse in the stage workspace etc, or B) if there is a way through an API or something to create deployment rules so that there is less manual work? It is extremely annoying to have to set the same deployment rules and connection settings for every single semantic model on it's initial push through the Deployment Pipeline.

[svenchio]

Hey @MLUM  I completely understand your fruistration, as equality frustrating is that, neither A nor B are currently available, just to make sure I went and check on the Power BI API  Power BI REST APIs for embedded analytics and automation - Power BI REST API | Microsoft Learn  - Yes, deployment pipelines API interfaces ARE NOT IN THE FABRIC API but the Power BI API (guess you did not see that one coming right? hehehe)

Anyways, I can tell you there's an option C, use the API to deploy your sematic models via API directly into the workspace with a modified connection in each of the source code! This option is perfectly possible, in fact, this is something I do, but is not an easy nor out-of-the-box option, let me tell you in general how it works, since all the items in fabric now can be git-integrated, that means, in code you can get the json definition of the item, using this, you need by code, to define your definition per each environment configuring the connection as you wish, and deploy the itesm via API's, but you need to understand how parts work on the API, how to encode the content as a 64-bit string, etc. etc. I can explain a bit more if interested, not sure if you want to got this path while Fabric keeps improving CI/CD capabilities in Fabric ... let me know. Nonethelss, I think I answer your question, mark as solution and kudos if you found this info. useful 

[MLUM]

Hey @svenchio thanks for the reply! I would be interested to know more. Currently we actually use the Git connection built into Fabric to trigger an Azure Devops Pipeline/Release that pushes our items through the Deployment Pipeline using the existing Deployment Pipeline API (there is just no way to set rules with this API). So since i have some of that set up already, your solution might work for me!

[svenchio · Accepted solution]

HI @MLUM awesome, then here it comes! I'm going to give you an algorithm witht the Fabric API interfaces you will need to use, of course there's a lot of coding to do to implemented, I can help partially with a couple of articles of my own on this topic, in particular I'm going to refere an articles I wrote related to this topic, I recommend you to check ... take a look 

#1.  The TMSL to TMDL Switch: A CI/CD Nightmare for Microsoft Fabric

#2.  The TMSL to TMDL Switch (Part 2): I Did My Homework!

So, the algotithm goes like this ... 

Step #1. Get all the semantic models from the workspace you want to deploy to upper environments.  Items - List Semantic Models - REST API (SemanticModel) | Microsoft Learn

Step #2. Iterate thru each of this Sematic Model items using Items - Get Semantic Model - REST API (SemanticModel) | Microsoft Learn

Step #3. For each Sematic Model item, get its definition, here is what I can help more, in the article #2, I'm sharing a fully functional example as to how getting the defintion to its JSON VERSION (TMSL)! this is what I rant about in my article, by default, the new version is TMDL wish is similar to YAML, better for user to read but a pain for parting and automation, luckily MSFT included in the API the option to choose the format. 

POST https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/semanticModels/{semanticModelId}/getDefinition?format={format}

 Items - Get Semantic Model Definition - REST API (SemanticModel) | Microsoft Learn 

Step #4. From the defintion's model.bin (if you follow my advise and use TMSL) you will have to replace the connection in the path model/expressions/expression as shown below 

Step #5. Encode your new definition and deploy to a the target workspace via Items - Create Semantic Model - REST API (SemanticModel) | Microsoft Learn (semantic model does not exist) or Items - Update Semantic Model Definition - REST API (SemanticModel) | Microsoft Learn (update existing semantic models) 

Step #6. Repeat for next semantic model 

Hopefully in the near future, MSFT will include more and more CI/CD capabilities in the platform, but, for the time being, there's a LOT YOU CAN DO with the API's, of course, there's a balance and a compromise of how much you would like to spend coding custom made solutions vs wait and do thing manually while Fabric gets there. 

For me there's a  lot of information to digest here so, I'll let you dig in and have fun and of course, ping me if you need any help or directly thru my blog entries (woudl help if you become a memeber) ... but I would definetly appreciate a kudos here and mark this as solution  ... cheers. 

[MLUM]

@svenchio is your semantic model in a power bi project file? I do not have the model.bim file in my Git.

[svenchio]

Perhaps this need a bit of additional explanaition, sematic models at Fabric's code leval can be represented in two formats, either TMSL to TMDL, if you created recently your semantic model, then the format in your code will be TMDL as the new default for semantic models

Explain on my article  The TMSL to TMDL Switch: A CI/CD Nightmare for Microsoft Fabric

In the prev. format TMSL, there is a model.bim and the path I mentioned, in TMDL there's no such file but the expressions.tmdl ... I guess you can see this last one right? 

If so, and you want to manipulate TMDL files, then step #3 won't be needed... I like to get the definition in JSON (TMDL) because it's easiert to manipulate with code 🤞  Let me know how it goes, remember to kudos on post and mark as solution when ready. 

[MLUM]

@svenchio i think i almost have this working, but the issue now is that the APIs you have linked, i do not see a way to use one of those to update the semantic model connection, it seems your process only allows you to update the files in Git but this is not what i need to to. I need to be able to use the API to update the Semantic Model connection in Fabric, Git is only linked to our Dev environment

[svenchio]

Hi @MLUM  well, yes, but when you are daling with the API, you can't just change the connection on the semantic model, you update the entire semantic model defintion with a new connection (hope it makes sense)  ... so the strategy is to get the defintion, manipulate it, and then send a new defintion back with a modified connection ... this is what I described in step #5. 

Step #5. Encode your new definition and deploy to a the target workspace via Items - Create Semantic Model - REST API (SemanticModel) | Microsoft Learn (semantic model does not exist) or Items - Update Semantic Model Definition - REST API (SemanticModel) | Microsoft Learn (update existing semantic models) 

[MLUM]

Hi @svenchio so you are saying i actually need to edit the connection in the Git files and then push it to another workspace? This could make things complicated as our process does not actually read from Git, it just uses a Git commit as a trigger and Artifact builder to tell the API which items to push through the deployment pipeline. We don't actually use the files in Git

Ideally i would be able to push the new model definition via API without needing Git.

[svenchio]

HI @MLUM , you don't need to get it from code, you can also get the defintion as I mentioned on Step #3, in this case, you use the API to get the current definition of a semantic model, modify the files and send back via API, this process could be done in the memory of you devops agent 😉 ... hope it makes sense, best of lucks  🤞

"Step #3. For each Sematic Model item, get its definition, [...] I'm sharing a fully functional example as to how getting the defintion to its JSON VERSION (TMSL)!  

POST https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/semanticModels/{semanticModelId}/getDefinition?format={format}

 Items - Get Semantic Model Definition - REST API (SemanticModel) | Microsoft Learn " 

[MLUM]

@svenchio Hi, I'm not sure how to interpret the API response, how are you getting to the TMSL response you posted above that has the expression?

This is the response i get from that API:

cache-control: no-store, must-revalidate, no-cache
content-length: 24
content-type: application/jso

n; charset=utf-8
location: https://api.fabric.microsoft.com/v1/operations/be34
pragma: no-cache
requestid: fffffffff54566677777
retry-after: 20
x-ms-operation-id: Some ID

If i add in the Format=TMSL at the end i just get a "null" response

The API i used was: https://api.fabric.microsoft.com/v1/workspaces/Workspace/semanticModels/SemanticId/getDefinition
AND The API you provided in the last reply with format=TMSL

[svenchio]

But in the info you sent is just the header of the response, you need to fetch the Body of the response containing a JSON that looks something like this ... instead of <base64 encoded string> there will be a LONG STRING, that's the payload (aka content of the file) 

So, you need to "process" this JSON, meaning, take the json, and decode each of the part's payload into a regular string so you can manipulate it (e.g. save it or replace json objects), and then, re-encoded and sent the new definition back via de API 😊 ... 

Perhaps if you are getting NULL when trying TMSL (JSON)  is because the semantic model is just not allowing it, as shown below, the default NOT for all new semantic models is TMDL (YAML)

Hope this clarification helps 😉 

[MLUM]

@svenchio I was able to get this to work, thanks for all of your help

[svenchio]

Hye @MLUM  GREAT NEWS!!! I'm glad that I've been of help, thanks for sharing ... all the best. 

[MLUM]

I will take a look and see if it can work for me, thank you!