Help with Dimension update with data comming from a huge data source

[Anonymous]

Hello, 

I'm quite new to Power BI Service and Dataflow, I've been looking for a solution or an approach to the following scenario, but I couldn't find the right answer. I would really appreciate your help or any ideas you might have.

I want to create a Dataflow that reads data from Log Analytics and then use this data with Power BI to populate Dashboards, etc. 

The data in Log Analytics are events generated by hundred of mobile applications, with hundreds of users, 24h, 365 days a year. So we expect huge amounts of data to be stored. 

I want to create a star schema, and I wouldn't know how to keep the dimensions up to date without querying in every refresh the whole log history. 

For example the User dimension. Ideally, I would process a delta of the logs at some point and see if the users that appear already exist or not in my User entity, and then decide if add them or not. If this is the optimal approach, I wouldn't know how to implement it in a Dataflow without the use of Computed Entities. I have a PRO subscription and these are not available. 

I also learnt that with Power Apps you can create Dataflows that allow the creation of Computed Entities if you associate a Data Lake storage to your environment. Is there any important difference between Power Apps Dataflows and Power BI Dataflows? I'm mean, why is Premium so expensive? What would be the downside of using Power App Dataflows?

I'm wondering if maybe there is another approach to do this without Computed Entities and without querying the whole log.  I don't see how Incremental Refresh could be useful. Although I belive I could use it to obtain a delta of the log, but what could I do with it if I cannot append it to the existing Users entity?

I would appreciate your thoughts. 

Thanks.

[aj1973]

Hi @Anonymous 

The question would be better, why do you want to use dataflows? What do you want to achieve? How do you want to connect to your data? how are you going to refresh your reports? Why thinking of premium instead of Pro? who is going to use your reports? with whom are you going to share it? and many more questions need to be answered before engaging into Power Bi.

Sometimes a Free license could do the work but for sure Premium is not an option for you now unless you would have more than 500 users in your organization that are going to use your reports.

[Anonymous]

Hello, @aj1973 

I use Dataflows to be able to store the entities in CDM (with an associated storage account) since Power BI datasets have size limitations that I believe would be surpassed in this case. 

Also is a requirement for this data to be available to other users, clients, or applications that at some point in the future would want to use it. 

In my post I mention that I have a PRO subscription. I explain that my initial approach was to get the new data and append it into an existing entity, something that is only available in Premium subscription (if you want it to refresh it). Then I explain that I learned about Power Platform Dataflows and I ask for an opinion about it, if this would be a good alternative since it enables to create and refresh Computed Entities. 

Then I wonder if this is a good approach (get delta data and append) or is there a better way to deal with dimensions in a scenario with large source datasets (the idea would be to avoid query the whole underlying log)