composite model and deployment Pipeline meta data refresh issue

[Raki39]

We encountered an unusual issue with the power deployment pipeline. Here's the situation:

We've developed a composite data model in Power BI, which is connected to an AAS (Azure Analysis Services) data model via a live connection. we converted it to a direct query local model.

After deploying this model to the Power BI service and using it to create reports, we found a drawback. Whenever a new table or column is added to the AAS data model, we must download the Power BI data model file from the service, refresh it in the desktop app to reflect those changes, and then republish it (client is ok for this porcess) .

We're utilizing the deployment pipeline feature to move the reports to subsequent workspaces (such as QA and prod).

The specific issue can be illustrated with an example: In the DEV environment, we have four tables (A, B, C, D) in the AAS model, and the Power BI local data model correctly displays these tables. However, when we moved this Power BI data model to the QA environment using the deployment pipeline (with respective rules configured), only tables A, B, and C were available in the QA AAS data model. Surprisingly, we could still see table D along with A, B, and C in the QA-deployed Power BI data model.
 I hope this explanation provides a clearer understanding of the situation.

requesting the experts help with this situation. @LeeBenjamin, Jeroen ter Heerdt is there any solution for this?

[Raki39]

@LeeBenjamin 
Sorry for the late reply. 
We have received the same response from the microsoft support team. 

It’s a by design behaviour of deployment pipeline for Composite model and we don’t have any workaround other than that to have same data  in all environments.

Thanks,
Raki

[v-huijiey-msft]

Hi @Raki39 ,

You can create gen2 in data factory and link it to AAS, then create pipeline in data factory, and pipeline will select the dataflow you just created.

If the pipeline is connected to dataflow, it will reflect the data changes in dataflow.

Dataflow is a thing similar to PowerQuery, it basically has what PowerQuery has.

For more information about Data Factory and Dataflow, please refer to :

What is Data Factory - Microsoft Fabric | Microsoft Learn

Creating a dataflow - Power BI | Microsoft Learn

Please refer to the documentation below for specific steps on how to do the above:

Copy Data from Azure Analysis Service (AAS) through Microsoft Fabric (c-sharpcorner.com)

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

[Raki39]

@v-huijiey-msft  Thank you some much for the response.
I apologize for any confusion in my previous explaination. To ensure clarity, I've provided addational detail below with an attachment.

"In our setup, we have Product and Sales tables in the AAS dev data source, while only the Sales table exists in the AAS QA and AAS prod data sources. Our Power BI semantic data model is connected to the AAS dev source in the Dev workspace. After deploying the semantic model to the QA and prod workspaces using a deployment pipeline, we noticed that the Product table appears in the deployed QA and prod semantic data models, even though it's not present in the respective AAS data sources. Despite refreshing the metadata, the Product table continues to appear. We're unable to identify the root cause of this issue.
Could you please review it and offeer any possible solution or insight
Regards,
Raki

[LeeBenjamin]

Hi @Raki39 
If I understand right the scenario, then this is by design.

Deployment pipelines pairs two items across neighbour stages so then on future deployments, it knows which item to override on the target stage. The pairing process is done based on the combination of item type+name while in case of conflict, the item's full path (workspace folders) will be considered as well.
With that, it seems like your semantic-model items on Dev and QA are paired, and same is for those on QA and Prod, since they have the same name and item type. Once paired, the item on the target stage will be overwritten, regardless if it has the same tables or not as its source. Fully compare of 2 paired items is only done as part of Deployment pipelines' compare process which in the end presents indidcation of the target item vs. its source item.
Best regards,

Lee

[Raki39]

Thanks For the explanation @LeeBenjamin 
Despite refreshing the semantic model data with the QA/PROD source after deployment, the metadata isn't updating. Consequently, columns or tables not present in the QA and Prod data sources remain visible in the semantic data model for both environments. Is this expected behavior within a deployment pipeline when using a semantic model? It's worth noting that we've successfully deployed multiple reports with direct Power BI datasets using live connections through the deployment pipeline without encountering this issue,  the deployment pipeline without encountering this issue,  

[LeeBenjamin · Accepted solution]

@Raki39 This works by design - The Model's schema in QA is similar to that in Dev, and it still uses the PRODUCT table (even though the data source doesn't have it). Deploy does not erase existing data, so either the new data source have all the tables the model is using, or add a parameter (again controlled by a rule) that somehow in the model prevent using that table.