topic Re: Pass data pipeline parameter to dataflow gen2 in Pipelines

Pass data pipeline parameter to dataflow gen2

SamWilliams — Fri, 09 Jun 2023 03:09:29 GMT

Is it possible to pass a data pipeline parameter (or variable) to a gen2 dataflow?

Re: Pass data pipeline parameter to dataflow gen2

miguel — Thu, 01 Jun 2023 17:45:38 GMT

Hi!

Not currently possible, but I've created a new idea so you can cast on vote on it and help us prioritize this work item:

https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=2f761603-a400-ee11-a81c-6045bdb98602

Thanks!

Re: Pass data pipeline parameter to dataflow gen2

n0elleli — Thu, 01 Jun 2023 17:46:44 GMT

Currently, this is not possible, but is a capability that will available in the future.

Re: Pass data pipeline parameter to dataflow gen2

Element115 — Fri, 19 Jan 2024 03:40:27 GMT

I sure hope so because as things stand now, more than 6 months later, I still cannot control the date range of my on-prem data ingest, which makes no sense as each day that passes, the data grows and grows, so no point in ingesting everything from scratch every single day.

Re: Pass data pipeline parameter to dataflow gen2

data_power — Wed, 03 Apr 2024 08:52:41 GMT

n0elleli Any estimates for that future. More than one year and still missing that feature. By any change any workaround untill that feature is onboard?

Re: Pass data pipeline parameter to dataflow gen2

Marusyk — Mon, 27 May 2024 20:33:41 GMT

I wonder when this future will come

Re: Pass data pipeline parameter to dataflow gen2

donovan-bergin — Thu, 12 Sep 2024 14:09:51 GMT

Have you tried a filter like (pick your date function/span that works for you):

Table.SelectRows(#"Remove Columns", each [Date_Column] >= DateTime.Date(DateTime.LocalNow()) - 1)

Re: Pass data pipeline parameter to dataflow gen2

Element115 — Thu, 12 Sep 2024 19:45:56 GMT

No because DFg2 are too expensive in terms of CUs. So I ended up implementing an incremental ETL inside a pipeline using timestamps from the tables in my LH and comparing to the timestamps in the on-prem DB tables.