Forum Discussion
CI/CD dataflow issue
Hello,
In Fabric, i have 2 workspaces (tests and prod)
i have azure devops repos (tests and prod)
i use the pipeline deployment to switch between both.
i have a datapipeline that calls 4 dataflows Gen2 that write data in a datawarehouse.
so far so good.
To avoid time consuming, i tried to set up library variables with the workspaces and the warehouses.
i found this documentation that indicates how to modify the mashup.pq file to drive destination by variables
but despites all my efforts, my prod warehouse is not being populated by the prod dataflows, it is always linked to the testing WS
I publish all git updates
i opened/saved the prod datapipeline and dataflows
without success.
Any idea of what should be checked ?
Thanks !
23 Replies
Hi GeekAlfPro ,
You can't do this natively with deployment pipelines, but you should be able to use variable libraries to achieve this.
See this documentation: https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-variable-library-integrationBe sure that after you edit the mashup.pq file in DevOps, that you do a pull in the workspaces so they pull down the changes - modifying things in git will not change them in your workspace without doing a pull.
You can also look at this post: https://community.fabric.microsoft.com/t5/Fabric-platform/Deployment-using-dataflow-Gen-2/m-p/4820064
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
Hello tayloramy
thanks for your answer.
For the mashup.pq
i edited my dev file
saved it
git add
git commit
git push
then i verified my code
that is the same in both environments
[StagingDefinition = [Kind = "FastCopy"]] section Section1; [DataDestinations = {[Definition = [Kind = "Reference", QueryName = "moeny Numérique_DataDestination", IsNewTarget = false], Settings = [Kind = "Manual", AllowCreation = false, ColumnSettings = [Mappings = {[SourceColumnName = "Année Budgétaire", DestinationColumnName = "Année Budgétaire"], [SourceColumnName = "Ligne Budgétaire", DestinationColumnName = "Ligne Budgétaire"], [SourceColumnName = "Commentaires", DestinationColumnName = "Commentaires"], [SourceColumnName = "Budget Initial", DestinationColumnName = "Budget Initial"], [SourceColumnName = "ID", DestinationColumnName = "ID"], [SourceColumnName = "Modifié", DestinationColumnName = "Modifié"], [SourceColumnName = "Budget_Quasi_ID", DestinationColumnName = "Budget_Quasi_ID"]}], DynamicSchema = false, UpdateMethod = [Kind = "Replace"], TypeSettings = [Kind = "Table"]]]}] shared #"Budget Direction Numérique" = let Source = SharePoint.Tables("https://url.com/sites/DptNumrique-bibli", [Implementation = "2.0", ViewMode = "All"]), #"Navigation 1" = Source{[Id = "d8705c22-aa38-484b-b16e-6bb0035814d4"]}[Items], #"Colonnes supprimées" = Table.RemoveColumns(#"Navigation 1", {"Title", "Balise de couleur", "ID de ressource de conformité", "Direction", "Modifié par", "Créé", "Type de contenu", "Créé par", "Version", "Pièces jointes", "Modifier", "Type", "Nombre d'éléments enfants", "Nombre d’enfants du dossier", "Paramètres de l’étiquette", "Étiquette de rétention", "Étiquette de rétention appliquée", "Étiquette appliquée par", "L’élément est un enregistrement", "Application créée par", "Application modifiée par"}), #"Colonne personnalisée ajoutée" = Table.AddColumn(#"Colonnes supprimées", "Budget_Quasi_ID", each [Année Budgétaire] & " - " & [Ligne Budgétaire]), #"Type de colonne changé" = Table.TransformColumnTypes(#"Colonne personnalisée ajoutée", {{"Budget_Quasi_ID", type text}, {"Budget Initial", type number}}), #"Valeur remplacée" = Table.ReplaceValue(#"Type de colonne changé", null, 0, Replacer.ReplaceValue, {"Budget Initial"}) in #"Valeur remplacée"; shared #"moeny Numérique_DataDestination" = let Pattern = Fabric.Warehouse([HierarchicalNavigation = null, CreateNavigationProperties = false]), Navigation_1 = Pattern{[workspaceId = ws_Dirnum_id]}[Data], Navigation_2 = Navigation_1{[warehouseId = wh_Dirnum_id]}[Data], TableNavigation = Navigation_2{[Schema = "dbo", Item = "Fact_Budget_DN"]}[Data] in TableNavigation;But it doesn't work unfortunately...
Hi GeekAlfPro,
THere's no reference to the variable library in that M code.
If you want it to pull the value from the variable library, you will need to reference the variable library like this:
Variable.ValueOrDefault("$(/**/My Library/My Variable)", "Sample")Then, after you've done the git commit and git push, you still need to do a git pull in the workspace. You can do that by clicking the update all button in the source control menu of the workspace.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
i modified my code like this
[...] in #"Valeur remplacée"; shared #"moeny Numérique_DataDestination" = let // 🔧 Lecture des variables de bibliothèque avec valeur par défaut ws_Dirnum_id = Variable.ValueOrDefault("$(/**/My library/ws_Dirnum_id)", "00000000-0000-0000-0000-000000000000"), wh_Dirnum_id = Variable.ValueOrDefault("$(/**/My library/wh_Dirnum_id)", "00000000-0000-0000-0000-000000000000"), // 🧭 Navigation dynamique Fabric (Warehouse cible) Pattern = Fabric.Warehouse([HierarchicalNavigation = null, CreateNavigationProperties = false]), Navigation_1 = Pattern{[workspaceId = ws_Dirnum_id]}[Data], Navigation_2 = Navigation_1{[warehouseId = wh_Dirnum_id]}[Data], TableNavigation = Navigation_2{[Schema = "dbo", Item = "Fact_Budget_DN"]}[Data] in TableNavigation;i've done a pull but without success
Hi GeekAlfPro ,
Sorry for the inconvenience , It would be a good idea to submit a support ticket to Microsoft. Their team can thoroughly investigate the problem within your specific environment and provide detailed assistance to identify the exact root cause. This approach will help ensure that the issue is properly diagnosed and resolved.To raise a support ticket for Fabric and Power BI, kindly follow the steps outlined in the following guide:How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn
