Forum Discussion

ptacquet's avatar
ptacquet
Frequent Visitor
4 years ago

Manage a datawarehouse with dataflows

Hi,

 

As indicated in the doc, we try to build a small dwh from scratch with dataflows.

https://docs.microsoft.com/en-us/power-query/dataflows/best-practices-for-data-warehouse-using-dataflows

 

We have PPU licences.

We received data based on csv files from applications.

 

We try to implement this process:

1. A staging table with daily files content

2. A transformation dataflow for loading stg rows into an historical table

3. DWH table based on stg historical table

 

We didn't manage to implement the second step to add rows in an historical table.

Quite simple with an ETL and a DB, but not with a dataflow storage and tranform.

 

I tried to use incremental refresh to manage my history. But it doesn't work because my staging datasource is volatile (only day content).

 

Except this doc, i didn't found an exemple of implementation. Any exemple, best practices ? 

Dataflow

3 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    4 years ago

    Hi ptacquet ,

     

    Please check if these two links below will be helpful in solving your problem.

    This is a thread about DatawareHouse vs Dataflows：

    DatawareHouse vs Dataflows 

     

    This article gives an overview of loading and transformation in a data warehouse

    Loading and Transformation 

     

    Hope it helps,

    Best Regards,
    Community Support Team _ Caitlyn

    If this post helps then please consider Accept it as the solution to help the other members find it more quickly.

     

  • ptacquet's avatar
    ptacquet
    Frequent Visitor
    4 years ago

    Hi Caitlyn,

     

    Thks. I saw the thread before posting.

    I know ETL processing for years, Oracle docs don't help me 😉

     

     My problem is thechnical with dataflows. But i think the use case of building an historical table based on files with dataflows is difficult to achieve (impossible?).

     

    I think that the concept of building a dwh with dataflows only works with a SQL Source.

    I have the impression, that I will need to construct an historical table with an ETL on prem. 

     

    This limitation should be explained.

  • Anonymous's avatar
    Anonymous
    Not applicable
    4 years ago

    Hi ptacquet ,

     

    Based on the above information, if you are a Power BI Pro licensee, you can create a support ticket for free and a dedicated Microsoft engineer will come to solve the problem for you.
    It would be great if you continue to share in this issue to help others with similar problems after you know the root cause or solution.

     

    The link of Power BI Support: Support | Microsoft Power BI

    For how to create a support ticket, please refer to How to create a support ticket in Power BI - Microsoft Power BI Community

     

     

    Hope it helps,


    Community Support Team _ Caitlyn

    If this post helps then please consider Accept it as the solution to help the other members find it more quickly.