Forum Discussion
Modelling a large customer table in Fabric
Hi, many thanks for your reply.
I try to write further details about the question.
I need to model the data about the customer registry in a banking scenario, namely in an enterprise organization, as a data product. So, I think to create a dedicated workspace in Fabric to manage the customer registry.
The ambit of the customer registry (from the source) is composed by some basic data, more some data with 1-to-1 relationships, more some data with 1-to-m relationships (e.g. customer addresses and customer notes) and some domain data (e.g. customer types) with 1-to-1 relationships.
The source is a structured and relational system.
I need to import csv files, one file for each source table.
I could combine basic data and the customer data with 1-to-1 relationships in a flat header table, the header customer table without any problems.
Then I could manage the domain tables maintaining the 1-to-1 relationships as a dimensional data model without any issues.
The focus is about how managing the customer data with 1-to-m relationships with the header data, considering that these data haven't any dependencies between them; a customer address does't depend to a customer note and viceversa.
My personal idea is to maintain these data as separate tables with 1-to-m relationships with the header customer data, but I should think to an alternative data model.
The goal is to implement a data product to consume not only by Power BI but also and specially by other workspaces related to other banking data.
Now, could I create an unique flat customer table combining together header data and addresses and notes and other 1-to-m data?
Following this way, if a customer has 3 addresses and 4 notes, in this unique flat table should I have 7 records (3 rows with header data more 3 addresses with null values for the notes, and 4 rows with header data more 4 notes with null values for the addresses) or 12 records (resulting by a cartesian product that generates 12 rows with header data more the address and note combinations, f.e. address1 and note1, address1 and note2, address1 and note3, and so on).
The basic customer data have almost 1-2 millions of rows, the addresses and notes could have more data.
Implementing an unique flat customer table could mean to manage big data.
So, which could be the appropriate data model for a such big data scenario?
E-R model or dimensional model or a combination between them or other right data model for big data?
I say again that the goal is to implement a customer data product to consume not only by Power BI, but also and specially by other workspaces related to other banking data. The customer data aren't more appropriate to perform data analysis.
Thanks
Hi pmscorca
I would suggest you to create a support ticket for the above case, as our support team can suggest you the best solution for this based on your requirement. Creating a support ticket allows our team to gather all the necessary details about your specific situation. This ensures we can provide the most accurate and tailored solution for you.
Please do reach out to our support team : https://support.fabric.microsoft.com/support
and provide the details of the support ticket here for further tracking.
Thanks