Manage a datawarehouse with dataflows

[ptacquet]

Hi,

As indicated in the doc, we try to build a small dwh from scratch with dataflows.

https://docs.microsoft.com/en-us/power-query/dataflows/best-practices-for-data-warehouse-using-dataf...

We have PPU licences.

We received data based on csv files from applications.

We try to implement this process:

1. A staging table with daily files content

2. A transformation dataflow for loading stg rows into an historical table

3. DWH table based on stg historical table

We didn't manage to implement the second step to add rows in an historical table.

Quite simple with an ETL and a DB, but not with a dataflow storage and tranform.

I tried to use incremental refresh to manage my history. But it doesn't work because my staging datasource is volatile (only day content).

Except this doc, i didn't found an exemple of implementation. Any exemple, best practices ? 

[Anonymous]

Hi @ptacquet ,

Based on the above information, if you are a Power BI Pro licensee, you can create a support ticket for free and a dedicated Microsoft engineer will come to solve the problem for you.
It would be great if you continue to share in this issue to help others with similar problems after you know the root cause or solution.

The link of Power BI Support: Support | Microsoft Power BI

For how to create a support ticket, please refer to How to create a support ticket in Power BI - Microsoft Power BI Community

Hope it helps,

Community Support Team _ Caitlyn

If this post helps then please consider Accept it as the solution to help the other members find it more quickly.

[ptacquet]

Hi Caitlyn,

Thks. I saw the thread before posting.

I know ETL processing for years, Oracle docs don't help me 😉

 My problem is thechnical with dataflows. But i think the use case of building an historical table based on files with dataflows is difficult to achieve (impossible?).

I think that the concept of building a dwh with dataflows only works with a SQL Source.

I have the impression, that I will need to construct an historical table with an ETL on prem. 

This limitation should be explained.

[Anonymous]

Hi @ptacquet ,

Please check if these two links below will be helpful in solving your problem.

This is a thread about DatawareHouse vs Dataflows：

DatawareHouse vs Dataflows 

This article gives an overview of loading and transformation in a data warehouse

Loading and Transformation 

Hope it helps,

Best Regards,
Community Support Team _ Caitlyn

If this post helps then please consider Accept it as the solution to help the other members find it more quickly.