How to automate the dataflow to load data for every day

[Koritala]

Hi ,

Just want to know the process to configure the dataflow that should load the data for every day by scheduling the dataflow in servive. (No Incremental Approach I am looking as we have db limitation).

In general for example, I want to load the data from 2025-Jan-01 to 2025-Dec-31.

As we have large volume data in many of the fact tables, I want to breakdown the dataflows like, one dataflow for current month and like that for remaining 11 months I want to create 11 dataflows for each month. After that, I will consolidated these individaul dataflows to Quarter level, and then Year level. consider that we are using the Import mode.

To load the data into dataset, I need to perform the dataloading process, on the 1st day of everymonth (To load entire data at month wise using individual months dataflows and then Quarterly dataflows and finally at the year end, should run the Year dataflow(it loads entire year data into dataset).

I need help in how to do it by creating the parameters to pass the dates or months manually and how to intregrate parametes in M code of Fact tables?

Appriciate if anyone can share the reference video or steps or json file of sample dataflow.

Thanks,

Sri.

[v-saisrao-msft]

Hi @Koritala,

Checking in to see if your issue has been resolved. let us know if you still need any assistance.

Thank you.

[v-saisrao-msft]

Hi @Koritala,

Have you had a chance to review the solution we shared earlier? If the issue persists, feel free to reply so we can help further.

Thank you.

[v-saisrao-msft · Accepted solution]

Hi @Koritala,
Thank you @Zanqueta,@GilbertQ, for your insights.

The following approach may help address the issue:
Manage data partitioning in the database by creating month-wise tables or views, each containing a fixed portion of data. These monthly objects can be combined into quarterly and yearly tables or views within the database. Power BI should then connect only to the final yearly table or view using Import mode. This method prevents repeated full scans of the source system, eliminates the need for incremental refresh, and ensures a stable and predictable refresh process for large data volumes.

Thank you.

[GilbertQ]

Hi @Koritala 

My question to you is why can you not use incremental refresh if you are loading data for the entire year, which means you then have to re query the database system or source system has to then re query and resend you all that information versus if you do incremental refresh that will just give you a subset of that information allowing you to then incrementally add data as time goes on.

[Koritala]

Hi GilbertQ,

Thanks for your quick reply.

Answer to your question is currently Redshift database tables are not supporting the query folding option. that's whay we couldn't implement the incremental load.

Thanks,

Srini

[Zanqueta]

Hi @Koritala

Power BI dataflows support parameters; however, they behave as constants. Once created, they cannot be dynamically changed at refresh time and can only be modified by editing the dataflow itself. This is a documented limitation.

This means you can indeed create one dataflow per month, each with its own fixed date range (using parameters or fixed filters), and schedule each one independently. It is a stable approach commonly used when incremental refresh is not an option.

 

Although you can hardcode the date filters directly into the query, using parameters makes the monthly duplication process clearer.

 

Within Power Query Online:

Open the dataflow and choose Edit.

Select Manage parameters → New.

Define parameters such as:

StartDate (Type: Date, e.g., 01/01/2025)

EndDate (Type: Date, e.g., 01/02/2025)

Repeat this structure per month, assigning the appropriate month‑specific values.

Applying the Parameters in the M Code

let
Source = Sql.Database("YourServer", "YourDatabase",
[Query = "SELECT * FROM FactSales"]),

#"Changed Type" = Table.TransformColumnTypes(Source,{{"SalesDate", type date}}),

#"Filtered Rows" = Table.SelectRows(
#"Changed Type",
each [SalesDate] >= StartDate and [SalesDate] < EndDate
)
in
#"Filtered Rows"​<p> </p>