Multi Layer Architecture in Pro

[BiWithMarius]

Hi dear BI-fam, 

I tried finding a solution in the community but havent come over it yet. So please excuse me if there already is a thread for my problem. 

The case: 

I currently work as Dashboard engeneer / Data Analyst and most of the datasets I use are already provided by the data engeneering department. 

Now I was asked to develop a BI integration for a small retail company (30 employees). 5-10 employees should later have access to the dashboard.
The plan is to connect their invoicing platform with PowerBi and enrich it with data.

The problem is that some data points should remain constant and should not change in the future, even if they change on the platform. Other data points should remain flexible and change when they are changed manually on the platform.
At the same time, there is a tight budget and there is no premium capacity available.

The plan:
I want to build a layer structure to achieve the goal. First the data ingestion layer where the API endpoints are called to load the raw data into the system. I have already achieved this with DataFlows.

Then I want a data processing layer (second layer) in which the data that is fixed / constant is processed.

Then, in a third layer, I would like to load this data with an incremental refresh and thus retain it.

In a fourth layer, the data from the third layer should then be retrieved, flexible data from the first layer added and then transformed for the final display. This data set should then be used for the dashboard.

The Problem: 

Unfortunately, PowerBI Pro does not let me reference the dataflows. As soon as I call up a dataflow with another dataflow, I am asked to buy Premium because otherwise no refresh can be carried out.

Have I overlooked something? How would you solve this problem? Will the customer have no choice but to buy several PPU licenses?

Thank you for your support and ideas!

Cheers,
Marius

[BiWithMarius · Accepted solution]

Thanks guys! Yes the incremental refresh was a premium feature but was introduced to Pro I think earlier this year. 

I posted the same question in the Power BI Subreddit and the user LostWelshMan85 came up with a great solution:

You actualy can reference a previous dataflow in pro, but only the loading part is restricted to premium. What you can do is to load the previous dataflow, deactivate load and then reference it in another table. In that one you can now transform and do your thing. Even refresh is working fine. Just takes a bit of dataflow orchestration and will probably use more of your pro storage, but if the dataset has a decent size, it should be a good workaround. 

[pmreis]

Hi @BiWithMarius 

Please refer to What licenses do you need to use dataflows - Power Query | Microsoft Learn for Dataflow capabilities in each of PRO / Premium licenses.

Power BI Pro

Power BI Pro gives you the ability to create analytical dataflows, but not use any of the premium features. With a Power BI Pro account, you can't use linked or computed tables, you can't use AI capabilities in Power BI, and you can't use DirectQuery to connect to the dataflow. The storage for your dataflows is limited to the space left under your Power BI Pro account, which is a subset of 10-GB storage for all Power BI content. Currently, we don't report the current storage usage of dataflows in the Power BI portal. You'll be notified if you've almost reached the limit of the left over capacity.

You mentioned using Incremental Refresh, which is a Premium feature, and will only work with Premium License.

Pedro Reis - Data Platform MVP / MCT
Making Power BI and Fabric Simple

If my response resolved your issue, please mark it as a solution to help others find it. If you found it helpful, please consider giving it a kudos. Your feedback is highly appreciated!

Find me at LinkedIn

[lbendlin]

You mentioned using Incremental Refresh, which is a Premium feature, and will only work with Premium License.

I was about to respond the same way but when checking the documentation this turns out not to be correct.

Supported plans
Incremental refresh is supported for Power BI Premium, Premium per user, Power BI Pro, and Power BI Embedded datasets.

Incremental refresh for datasets and real-time data in Power BI - Power BI | Microsoft Learn

[lbendlin]

Can you not push all the processing further upstream, into their existing systems? Doesn't have to be fancy, a OneDrive with Excel macros could do.