Forum Discussion

Imran2706's avatar
Imran2706
New Member
5 hours ago

How to Dynamically Select Lakehouse Tables with Different Schemas Using Pipeline Parameters?

Hi Fabric Community,
I am trying to build a metadata-driven / parameterized Dataflow Gen2 where the user provides a table name only once in the Pipeline, and the Dataflow should dynamically read that Lakehouse table and write the result to the corresponding destination table.
My requirement
I have a Lakehouse containing multiple tables, for example:
Lakehouse

Sales
Customer
Employee

The tables have different schemas.
For example:
Sales:
OrderID
CustomerID
OrderDate
Region
Amount
Status

Customer:
CustomerID
CustomerName
City
Country
Phone

Employee:
EmployeeID
EmployeeName
Department
Salary
JoiningDate

The user should not have to open Dataflow Gen2 and manually change a parameter every time.

I want the user to provide only: Pipeline parameter: table_name = Customer
The Pipeline should pass this value to the Dataflow Gen2 parameter:  p_table_name = @pipeline().parameters.table_name

Then the Dataflow should dynamically select the Customer table. If the user runs the Pipeline again with:
table_name = Sales
the same Dataflow should dynamically select Sales.

My Dataflow Gen2 setup
I created a Dataflow Gen2 with a parameter:
Parameter name: p_table_name
Type: Text/String
I enabled: Enable parameters to be discovered and overridden for execution

My source is a Lakehouse. The Dataflow query structure is approximately:

p_table_name
Source
Navigation 1
Navigation 2
Filtered rows
Custom

Navigation 2 contains metadata such as:
Name
Id
Data
Schema
ItemKind
ItemName
IsLeaf

I filter the table using the following M expression:
Table.SelectRows(#"Navigation 2", each [Name] = p_table_name)

Then I select the actual table data using: #"Filtered rows"{0}[Data]
This works correctly in the Dataflow editor.

For example, when:
p_table_name = Sales
the Custom step shows the Sales data.

When:
p_table_name = Customer
the Custom step shows the Customer data.
So the dynamic source selection is working

Pipeline setup
I created a Pipeline parameter:
Name: table_name
Type: String

In the Dataflow activity, I map:
p_table_name = @pipeline().parameters.table_name
So the intended flow is:

Pipeline parameter
table_name
Dataflow activity
p_table_name
Dynamic Lakehouse table selection
Destination setup

The Dataflow has a Lakehouse destination

I parameterized the destination table name using p_table_name

The destination is configured approximately as:
Lakehouse
Schema: dbo
Table name: p_table_name
Update method: Replace
Schema option: Dynamic schema

I also tested the destination configuration with Dynamic schema.

The problem
When the Pipeline parameter and Dataflow parameter have the same value, the Pipeline succeeds.
For example:
Dataflow p_table_name = Customer
Pipeline table_name = Customer

Result: Pipeline succeeds
Customer data is loaded to the destination However, when I leave the Dataflow design-time parameter as:
p_table_name = Customer and run the Pipeline with: table_name = Sales

the Pipeline/Dataflow fails with:

Dataflow refresh job failed with status: Failed. 

Failure reason:

requestId: ...

errorCode: EntityUserFailure

message: Something went wrong, please try again later.

The same happens in the reverse situation:

Dataflow p_table_name = Sales
Pipeline table_name = Customer

The Pipeline fails.

Important observation

If both values are the same:

Dataflow = Sales
Pipeline = Sales

it works.

If:

Dataflow = Customer
Pipeline = Customer

it works.

But if:

Dataflow = Customer
Pipeline = Sales
it fails.

This makes me wonder whether the runtime Pipeline parameter is actually overriding the Dataflow parameter correctly, or whether the problem is related to the destination or schema.

My main question

Since Dataflow Gen2 supports public parameters that can be overridden at Pipeline execution time, shouldn't I be able to do the following?

Pipeline:
table_name = Sales
Dataflow:
p_table_name = Sales
Source:
Sales
Destination:
Sales

without manually changing the Dataflow's p_table_name?

In other words, I want the Pipeline parameter to be the only value the user has to change.

The user should never have to:

1. Open the Dataflow.

2. Change p_table_name.

3. Save or publish the Dataflow.

4. Go back to the Pipeline.

5. Run it.

Instead, they should only do:

Pipeline -> table_name = Sales -> Run

or:

Pipeline -> table_name = Customer -> Run

Additional question about different schemas

Because Sales, Customer, and Employee have completely different schemas, is this scenario supported with one parameterized Dataflow Gen2?

For example:

table_name = Sales

Sales schema

Sales destination

and:

table_name = Customer

Customer schema

Customer destination

using the same Dataflow?

I have selected:

Update method = Replace

Schema option = Dynamic schema

I also noticed the destination has:

Use automatic settings

Would enabling automatic settings allow the destination schema to dynamically adapt to the selected table?

Expected behavior

I am trying to achieve the following:

Pipeline

table_name

Sales / Customer / Employee

Dataflow Gen2

p_table_name

Dynamic Lakehouse source

Dynamic destination

The key requirement is:

The user should provide the table name only once in the Pipeline, and the same Dataflow should dynamically process whichever Lakehouse table was selected, even when the tables have different schemas.

Is this supported in Dataflow Gen2?

If yes, what is the correct configuration for the source and destination?

If not, what is the recommended Fabric architecture for achieving this requirement?

Thank you.

1 Reply