Forum Discussion

tac's avatar
tac
Frequent Visitor
6 years ago

Custom queries to detect data changes on dataflow entity

Hi community,   I'm currently looking for improvements on my incremental refresh setup, and hopefully some of you may be able to help me. Here is the context:   We have one dataflow entity which...
Dataflow
v-xicai's avatar
v-xicai
Community Support
6 years ago

Hi tac ,

 

You may need to improve the dataflow performance, see: https://community.powerbi.com/t5/Service/SQLServer-performace-issue-will-dataflows-help/td-p/783770.

 

Or use incremental refresh for dataflow, see more detail:https://docs.microsoft.com/en-us/power-bi/service-dataflows-incremental-refresh.

 

Best Regards,

Amy 

 

Community Support Team _ Amy

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

  • tac's avatar
    tac
    Frequent Visitor
    6 years ago

    Hi v-xicai ,

    Thanks for your message and links to the documentation. I could say I did most of the possible improvements in the dataflow side, knowing I have more or less no transformation in the entity subject to the incremental refresh.

    However, when reading the documentation, something caught my attention:

    https://docs.microsoft.com/en-us/power-bi/service-dataflows-incremental-refresh#configuring-incremental-refresh-for-dataflows

    We could read:

    "

    Tip

    The current design requires that the column to detect data changes be persisted and cached into memory. You may want to consider one of the following techniques to reduce cardinality and memory consumption:

    • Persist only the maximum value of this column at time of refresh, perhaps using a Power Query function.

    "

    But how we could do this? At that time, I'm returning for each row the CDC timestamp but I would be happy to be able to replace by a M function or something similar. But how?

    Thanks for your help