Forum Discussion
Managing and Exporting High-Volume Datasets (~1M Records) in Power BI
Hi v-hashadapu ,
Thanks for the detailed breakdown on my reply so I got some information with respect to the RLS (Row Level security) so they have kind of custom setup, for which I will give an overview for better context
So the thing is end users intteract with the organization portal, so what this portal does (behind the scene is) it generates an unique ID based on the end-users email {this is just created once and stored in backend I presume and not at the time of authentication with Power BI service}
When the Power BI service is accessed (in the backend) at the organization there is a RLS table : 'v_user_facility_map' which is maintained that has 3 columns: id, user_email (end_user email), user_facility_email (kind of admin whose email has the required access within the org)
Example:
table: v_user_facility_map
d_facility_id: 32, user_email: [email protected], user_facility_email: [email protected]
d_facility_id: 32, user_email: [email protected], user_facility_email: [email protected]
And this table is turn connected to one of the dimension table (which denotes what level of access those set of user have, as per what I know)
And then a role is created on top of the RLS table (v_user_facility_map)
[user_facility_security_id] IN
CALCULATETABLE(
VALUES('v_user_facility_map'[user_facility_security_id]),
'v_user_facility_map'[user_email] = USERPRINCIPALNAME()
)And then within the workspace (Fabric workspace) we simply set the security for the semantic model.
So I tested this on my end for which I created a simple table (RLSDomainAccess) wherein I had specified the Domain and the email
Similarly created a role within Power BI desktop:
And then published the respective report in Fabric workspace and set the respective users under the new role
Then to allow the end user to trigger the flow, I had to add the user under the Run-as option (because before that if the end user tried to trigger the flow it was Unsuccessful)
After this the user were ablke to trigger the flow (the users used the connection of the owner) and this also ensured that RLS was maintained like:
Normal view (with the entire access like super user)
Assume 'John' accesses the report who has access to Domain== 'RESTAURANT'
So now based on this setup one thing is clear that RLS will be preserved (based on the setup that we have in the actual organization side) now I had couple of questions now when the flow needs to be triggered right now I tested this with few of my team-mates now in the actual case do the respective admin have to simply mention the email/ or the Azure AD group under Run as so they can trigger the flow.
The other thing you did mention for local download the setup will be quite heavy (for every user) so its better to go for SharePoint, but I'm still unclear about the flow on how to integrate the same in my existing flow.
Queries:
1: How to get the end user details (within this context)
2: You have mentioned to make use of Grant access action, so in this case will the organization where its to be deployed will have to maintain/create a SharePoint site and then the respective SharePoint admin will mange the access related things
3: To create a separate folder for every user there are some intial checks that whether the folder exist for the respective who has triggered (like Tim triggers the flow then we need to check in the respective SP document library a folder exist with his name/email if not then create one -> break the security and then gran the access to only that folder so here is there some kind of Post {HTTP} call that needs to be made: if yes what is the payload that will be passed {API endpoint and the parameters within it}
I hope this context helps you a have better understanding of the context and setup which I am working on.
Poojara_D12 , Gabry if you have any inputs please share your thoughts.
Regards,
Sidhant
Hi v-hashadapu ,
So there has been a progress when it comes to the part of folder creation and storing them within respective folders (you can check the latest comment have started a new reply for the same) now I wanted to understand based on the structure that I just described in my previous reply. The thing is the solution that we have proposed (using Power Automate along with Power BI to overcome the record limit) whether it will work in a complex environment.
The reason being right now what I have done is a just a single table with few set of columns but in actual environment that's not the case there we have multiple fact tables and dimension tables, so therein we need to switch between fact tables, like there are cases wherein field parameters have 10/20 columns, so for now what we were thinking we will just provide the most commonly used columns (from different tables). The other thing is in the UserSelections table as of now I have just placed columns from single table, but is it possible that we can consolidate the columns used in field parameters within a single table (like UserSelections)
'Orders[OrderID]', 'People[Department]' {Here Orders, People are two different tables so can that be consolidated when we construct the UserSelections.
I also wanted to understand that this solution what are the limitiations that things to keep in mind when the data is so large and considering the data model as well.
Gabry , Poojara_D12 if you have anything to add please share those points
SantoshPothnak if I have missed something please do add your points as well.
Regards,
Sidhant