Forum Discussion

Cookistador's avatar
Cookistador
Super User
8 months ago

Dataflow gen2: use libraby variable for destination ?

Hello everyone,   One of my coworkers decided to use Dataflow Gen2 to simplify maintenance. After a quick analysis of the capacity unit (CU) consumption, the results are fully acceptable — so w...
Dataflow
v-ssriganesh's avatar
v-ssriganesh
Community Support
8 months ago

Hi Cookistador,
Thank you for posting your query in the Microsoft Fabric Community Forum.

At the moment, Dataflow Gen2 does not support changing the destination output settings by manually editing the mashup script in OneLake. Even though the files appear in OneLake File Explorer, they are not the source of truth for Dataflow Gen2. The product currently ignores manual edits because:

  • Destination configuration is stored as metadata, not in the mashup script.
  • Editing the script in OneLake does not trigger a metadata refresh, so the UI continues to show and use the previous destination.

If dynamic destination configuration is important for your scenario, we recommend using the Fabric REST API or Git connected workspace, as these are supported approaches for parameterizing destinations.


Best regards,
Ganesh Singamshetty.

  • Cookistador's avatar
    Cookistador
    Super User
    8 months ago

    Hello v-ssriganesh 

     

    What I don't understand, 

     

    I modified the .PQ via OneLake explorer, as it doesn't work,
    I decided to use the API approach, but when I get my PQ definition via the API, I can see the modification I've done via the one lake Explorer, in the destination source I have:

    Navigation_2 = Navigation_1{[warehouseId = Variable.ValueOrDefault("$(/**/Variables_1/Warehouse_ID)", "1f5196d3-59a1-41b1-9fee-9900c1eXXXXX")]}[Data]

    So it means that OneLake explorer is able to modify my files ?
    I just republish the dataflow via API and I see an improvement, as a a warehouse, I have nothing:

    But it continues to load it in the wrong table

     

    • v-ssriganesh's avatar
      v-ssriganesh
      Community Support
      8 months ago

      Hi Cookistador,
      Thank you for the additional details.

      Yes, OneLake Explorer can modify the .pq file, and that’s why the REST API shows your updated M script. However, in Dataflow Gen2 the destination is not controlled by the M script. It is stored in separate metadata, which is not updated when you edit the .pq file.

      So even though your Navigation step changed, the Dataflow still loads into the old table because the destination metadata was never updated. Republishing via API only refreshes the script. it does not overwrite the destination settings.

      To actually change where the data is written, you must update the destination/output configuration through the Fabric REST API or via Git deployment. Editing .pq alone will not redirect the load.

      • Cookistador's avatar
        Cookistador
        Super User
        8 months ago

        Ok so if I undestand correctly
        I have to use the API to get the .pq files coded in Base64, then I have to Decode it, update it with the parameter I want to use and after that, I use the API to update my new flow

         

        Is it right? 
        Not possible to do the same in file explorer even if I edited the Mashup File AND the DataflowEditingSession ?

         

        I'm just looking for an easier workaround for my business team 😕