Forum Discussion
How to Dynamically Select Lakehouse Tables with Different Schemas Using Pipeline Parameters?
Hi Fabric Community,
I am trying to build a metadata-driven / parameterized Dataflow Gen2 where the user provides a table name only once in the Pipeline, and the Dataflow should dynamically read that Lakehouse table and write the result to the corresponding destination table.
My requirement
I have a Lakehouse containing multiple tables, for example:
Lakehouse
Sales
Customer
Employee
The tables have different schemas.
For example:
Sales:
OrderID
CustomerID
OrderDate
Region
Amount
Status
Customer:
CustomerID
CustomerName
City
Country
Phone
Employee:
EmployeeID
EmployeeName
Department
Salary
JoiningDate
The user should not have to open Dataflow Gen2 and manually change a parameter every time.
I want the user to provide only: Pipeline parameter: table_name = Customer
The Pipeline should pass this value to the Dataflow Gen2 parameter: p_table_name = @pipeline().parameters.table_name
Then the Dataflow should dynamically select the Customer table. If the user runs the Pipeline again with:
table_name = Sales
the same Dataflow should dynamically select Sales.
My Dataflow Gen2 setup
I created a Dataflow Gen2 with a parameter:
Parameter name: p_table_name
Type: Text/String
I enabled: Enable parameters to be discovered and overridden for execution
My source is a Lakehouse. The Dataflow query structure is approximately:
p_table_name
Source
Navigation 1
Navigation 2
Filtered rows
Custom
Navigation 2 contains metadata such as:
Name
Id
Data
Schema
ItemKind
ItemName
IsLeaf
I filter the table using the following M expression:
Table.SelectRows(#"Navigation 2", each [Name] = p_table_name)
Then I select the actual table data using: #"Filtered rows"{0}[Data]
This works correctly in the Dataflow editor.
For example, when:
p_table_name = Sales
the Custom step shows the Sales data.
When:
p_table_name = Customer
the Custom step shows the Customer data.
So the dynamic source selection is working
Pipeline setup
I created a Pipeline parameter:
Name: table_name
Type: String
In the Dataflow activity, I map:
p_table_name = @pipeline().parameters.table_name
So the intended flow is:
Pipeline parameter
table_name
Dataflow activity
p_table_name
Dynamic Lakehouse table selection
Destination setup
The Dataflow has a Lakehouse destination
I parameterized the destination table name using p_table_name
The destination is configured approximately as:
Lakehouse
Schema: dbo
Table name: p_table_name
Update method: Replace
Schema option: Dynamic schema
I also tested the destination configuration with Dynamic schema.
The problem
When the Pipeline parameter and Dataflow parameter have the same value, the Pipeline succeeds.
For example:
Dataflow p_table_name = Customer
Pipeline table_name = Customer
Result: Pipeline succeeds
Customer data is loaded to the destination However, when I leave the Dataflow design-time parameter as:
p_table_name = Customer and run the Pipeline with: table_name = Sales
the Pipeline/Dataflow fails with:
Dataflow refresh job failed with status: Failed.
Failure reason:
requestId: ...
errorCode: EntityUserFailure
message: Something went wrong, please try again later.
The same happens in the reverse situation:
Dataflow p_table_name = Sales
Pipeline table_name = Customer
The Pipeline fails.
Important observation
If both values are the same:
Dataflow = Sales
Pipeline = Sales
it works.
If:
Dataflow = Customer
Pipeline = Customer
it works.
But if:
Dataflow = Customer
Pipeline = Sales
it fails.
This makes me wonder whether the runtime Pipeline parameter is actually overriding the Dataflow parameter correctly, or whether the problem is related to the destination or schema.
My main question
Since Dataflow Gen2 supports public parameters that can be overridden at Pipeline execution time, shouldn't I be able to do the following?
Pipeline:
table_name = Sales
Dataflow:
p_table_name = Sales
Source:
Sales
Destination:
Sales
without manually changing the Dataflow's p_table_name?
In other words, I want the Pipeline parameter to be the only value the user has to change.
The user should never have to:
1. Open the Dataflow.
2. Change p_table_name.
3. Save or publish the Dataflow.
4. Go back to the Pipeline.
5. Run it.
Instead, they should only do:
Pipeline -> table_name = Sales -> Run
or:
Pipeline -> table_name = Customer -> Run
Additional question about different schemas
Because Sales, Customer, and Employee have completely different schemas, is this scenario supported with one parameterized Dataflow Gen2?
For example:
table_name = Sales
Sales schema
Sales destination
and:
table_name = Customer
Customer schema
Customer destination
using the same Dataflow?
I have selected:
Update method = Replace
Schema option = Dynamic schema
I also noticed the destination has:
Use automatic settings
Would enabling automatic settings allow the destination schema to dynamically adapt to the selected table?
Expected behavior
I am trying to achieve the following:
Pipeline
table_name
Sales / Customer / Employee
Dataflow Gen2
p_table_name
Dynamic Lakehouse source
Dynamic destination
The key requirement is:
The user should provide the table name only once in the Pipeline, and the same Dataflow should dynamically process whichever Lakehouse table was selected, even when the tables have different schemas.
Is this supported in Dataflow Gen2?
If yes, what is the correct configuration for the source and destination?
If not, what is the recommended Fabric architecture for achieving this requirement?
Thank you.
1 Reply
- GilbertQ
Super User
Hi Imran2706
As far as I am currently aware, it is not possible to configure the destination via parameter.