Dataflow Gen2: Variable Library references fail in Connection Settings
Hi everyone, I am working on variabilizing my Dataflow Gen2 for deployment pipelines, following the documentation on Variable references. I’ve noticed a discrepancy in how variables are handled, and I suspect it might be a bug or a UI limitation. The Issue: When I define parameters using a Blank Query and try to use them to replace values in the Source Connection settings (via the user interface), the connection fails or isn't recognized. And replace in the source: let Source = Lakehouse.Contents([]), #"Navigation 1" = Source{[workspaceId = workspace_id]}[Data], #"Navigation 2" = #"Navigation 1"{[lakehouseId = lakehouse_id]}[Data], #"Navigation 3" = #"Navigation 2"{[Id = "Table1", ItemKind = "Table"]}[Data] in #"Navigation 3" That's not working. The Workaround (Manual M Code): However, if I bypass the UI and manually insert the Variable.ValueOrDefault syntax directly into the Source Step using the Advanced Editor, it works perfectly. let Source = Lakehouse.Contents([]), // Retrieving variables manually works fine workspace_id = Variable.ValueOrDefault("$(/**/My Library/Workspace ID)", "default_value"), lakehouse_id = Variable.ValueOrDefault("$(/**/My Library/Lakehouse ID)", "default_value"), #"Navigation 1" = Source{[workspaceId = workspace_id]}[Data], #"Navigation 2" = #"Navigation 1"{[lakehouseId = lakehouse_id]}[Data], #"Navigation 3" = #"Navigation 2"{[Id = "Table1", ItemKind = "Table"]}[Data] in #"Navigation 3" My Question: Has anyone else encountered this issue where the UI connection dialog doesn't seem to parse the variable reference, forcing us to hardcode M logic manually? Is there a specific syntax for the UI that differs from the Advanced Editor? Thanks for your help! References: - Dataflow Gen2 Variable Library IntegrationFabrics to Dataverse - Dataflow Alternative
Hi, I have been using Dataflow in power apps to move data from Fabrics lakehouse to Dataverse table. Also mentioned in other posts as Reverse Integration. I am fully aware Dataflow does the job but I am facing long runs which usually takes ~3 mins to complete. Keeping that in mind, I need to find a reliable & consitent approach rather than using Dataflow. My requirements: 1. Scheduling 4 to 6 times day 2. Merge capability based on ID 3. Lookup columns should also be map-able. I have tested: 1. Dataflow ( Inconsitent with run time) 2. Copy Job from Fabrics (Does not allow mapping for lookup columns, even when i convert the mappings w.r.t. the value of the lookup). Please do let me know if i am missing anything. Thanks in advance.Managing pagination for a Web API connector used in a dataflow gen2
Hi, I need to create a dataflow gen2 that has to use a Web API connector to read Dynamics 365 Business Central data. This connector requires to manage the pagination else it allows to read only the first 20.000 rows. I've read that I need to manage the pagination by a Power Query function, while with a data pipeline isn't required a such management. Does it exist a more easy manner in order to use a Web API connector in a dataflow gen2 to read an entire BC table? Many thanks
Combining Multiple Values from a SharePoint List into a Single Cell in Dataverse Using Power Query a
I have an online SharePoint list that I want to import into a Dataverse table using a dataflow. The SharePoint list contains a column with multiple values (list or table type) that I need to expand. However, instead of expanding these values into multiple rows, I want to combine them into a single cell, separated by comma
CDM - Common Data Model format connector
I would be even happier with this in Pipelines but will there be a connector for the legacy CDM (common data model) files, ideally supporting both Azure Synapse Link for Dataverse generated files and D365 (F&O) Export to Data Lake files? Use case: Our D365 environments are going to lag behind in updates and there will be further lag migrating to Synapse Link. In the meantime we have built a process to parse and consume D365 F&O and will be building to consume Azure Synapse Link Dataverse CDM files to update our Synapse Dedicated Pool. We are quite likely going to be able to start using Fabric technologies far sooner than we will upgrade Azure Synapse Link for Dataverse to write to Delta and far sooner than we will upgrade F&O Export to Lake to Azure Synapse Link. I'd like to be able to easily bridge updating my Lakehouse before we get all the back end technologies modernized. After that migrating to Azure Synapse Link with Delta support will be much easier.
How to fully load data once and then incrementally update with yesterday’s data in Dataflow Gen2?
Hi all, I'm new to Dataflow Gen2 and exploring it to centralize data transformations for Power BI reporting. I’m using F2 capacity and pulling data from multiple sources like SQL Server, Salesforce, and Microsoft Business Central. While building the dataflows, I didn’t define any explicit output data destination (bottom left panel in the dataflow UI). But I noticed that files like “StagingLakehouseForDataflows_...” are automatically getting created in my workspace under type “Lakehouse” or “Warehouse”. What exactly are these files? Is it okay to leave the output destination undefined if I can still see the tables and use them in Power BI? Incremental Refresh Doubts: I tried setting up incremental refresh on one of the queries (Currency Exchange Rate). This table has three columns: Currency Code Starting Date Currency Exchange Rate There’s historical data available from 2020 onwards. Since the exchange rates update daily, my goal is to: Load the full historical dataset initially Then, refresh only yesterday’s data daily (i.e., append new records to the existing dataset) So I used these settings in Incremental Refresh: Date column: Starting Date Extract data from the past: 1 day Bucket size: Day But after enabling and running the refresh, I noticed that only records with yesterday’s date were loaded — the older data (from 2020 onwards) wasn't loaded at all. What I need: I want the dataflow to load the entire dataset initially, and then on each scheduled refresh, only add new records from yesterday (without removing the older ones). How can I properly configure incremental refresh to support this pattern in Dataflow Gen2? Thanks in advance for any help or clarification!
