Forum Discussion

Cookistador's avatar
Cookistador
Icon for Super User rankSuper User
8 months ago

Dataflow gen2: use libraby variable for destination ?

Hello everyone,

 

One of my coworkers decided to use Dataflow Gen2 to simplify maintenance.

After a quick analysis of the capacity unit (CU) consumption, the results are fully acceptable — so we agreed to continue using this approach.

 

The only issue I’m facing concerns the destination output configuration.

I’d like to set it dynamically as a variable library. According to Microsoft’s documentation, this can be done through the API or via Git.

 

However, after exploring the OneLake File Explorer, I located the mashup script in the correct folder. Even after making changes and saving them, nothing seems to update — the Dataflow Gen2 still uses the previous destination value. Is there another step required to apply or refresh these changes?

 

For context, when I open the Dataflow Gen2 in Fabric, it still shows the old destination, even though the script file in OneLake is correctly synced.

 

Many thanks for your ideas and insights.

Dataflow

7 Replies

  • v-ssriganesh's avatar
    v-ssriganesh
    Icon for Community Support rankCommunity Support
    8 months ago

    Hi Cookistador,
    Thank you for posting your query in the Microsoft Fabric Community Forum    .

    At the moment, Dataflow Gen2 does not support changing the destination output settings by manually editing the mashup script in OneLake. Even though the files appear in OneLake File Explorer, they are not the source of truth for Dataflow Gen2. The product currently ignores manual edits because:

    • Destination configuration is stored as metadata, not in the mashup script.
    • Editing the script in OneLake does not trigger a metadata refresh, so the UI continues to show and use the previous destination.

    If dynamic destination configuration is important for your scenario, we recommend using the Fabric REST API or Git connected workspace, as these are supported approaches for parameterizing destinations.


    Best regards,
    Ganesh Singamshetty.

    • Cookistador's avatar
      Cookistador
      Icon for Super User rankSuper User
      8 months ago

      Hello v-ssriganesh 

       

      What I don't understand, 

       

      I modified the .PQ via OneLake explorer, as it doesn't work,
      I decided to use the API approach, but when I get my PQ definition via the API, I can see the modification I've done via the one lake Explorer, in the destination source I have:

      Navigation_2 = Navigation_1{[warehouseId = Variable.ValueOrDefault("$(/**/Variables_1/Warehouse_ID)", "1f5196d3-59a1-41b1-9fee-9900c1eXXXXX")]}[Data]

      So it means that OneLake explorer is able to modify my files ?
      I just republish the dataflow via API and I see an improvement, as a a warehouse, I have nothing:

      But it continues to load it in the wrong table

       

      • v-ssriganesh's avatar
        v-ssriganesh
        Icon for Community Support rankCommunity Support
        8 months ago

        Hi Cookistador,
        Thank you for the additional details.

        Yes, OneLake Explorer can modify the .pq file, and that’s why the REST API shows your updated M script. However, in Dataflow Gen2 the destination is not controlled by the M script. It is stored in separate metadata, which is not updated when you edit the .pq file.

        So even though your Navigation step changed, the Dataflow still loads into the old table because the destination metadata was never updated. Republishing via API only refreshes the script. it does not overwrite the destination settings.

        To actually change where the data is written, you must update the destination/output configuration through the Fabric REST API or via Git deployment. Editing .pq alone will not redirect the load.