Forum Discussion

amir_tohidi's avatar
amir_tohidi
Advocate I
10 years ago

Design advice needed

Hello,

 

Being new to Power BI, I am not sure what the optimal design for the following problem is. Any thoughts/ideas would be appreciated.

 

I would like to create a business user friendly data model that consists of the following tables:

1) Customers

2) Accounts

3) AccountTransactions

4) FraudCodes

 

Customers, Accounts and AccountTransactions are in a SQL Server database with the following relationships:

Customers 1---* Accounts 1----* AccountTransactions

 

FraudeCodes is an Excel file that provides the following mappping:

FraudCode 1 --- * AccountTransactions.TransactionCode  

 

My problem is AccountTransactions contains 55 million rows without any filtering. Even if I apply date and transaction code filer, I still have 3.8 millions rows that I need to process. The table widget is already crashed with error "Can't display the visual"

 

Any ideas how to best model this problem?

Need Help

10 Replies

  • asocorro's avatar
    asocorro
    Skilled Sharer
    10 years ago

    Since you are connecting to an Excel file, I suppose you are importing the data rather than using DirectQuery mode.  How about importing the Excel data to the SQL Server database and switching to DirectQuery mode?

  • Anonymous's avatar
    Anonymous
    Not applicable
    10 years ago

    Is there some way that you can roll-up the AccountTransactions before it gets to this point?  e.g. Can you do a Group By AccountID, Date, Transaction Type to reduce the number of raw records?

    • amir_tohidi's avatar
      amir_tohidi
      Advocate I
      10 years ago

      asocorro - Only the FraudCodes data is in Excel. Everything else is in SQL Server. My problem is the large amount of SQL Server data that I need to process.

       

      silolondon - I can't roll up the account transactions because our Financial crime team want to anlyses each and every single transaction in 1 half of 2015.

       

      I am using 64 bit Power BI which I thought can handle large number of rows. Is that not the case?

       

       

      • Anonymous's avatar
        Anonymous
        Not applicable
        10 years ago

        Have you tried using a much smaller subset of transactions to ascertain whether it's a size problem, or a setup problem?

  • asocorro's avatar
    asocorro
    Skilled Sharer
    10 years ago

    Hm, why do you want millions of records in a table visualiization?

     

    Have you tried visualizations that aggregate the data, like bar charts?

    • amir_tohidi's avatar
      amir_tohidi
      Advocate I
      10 years ago

      Our financial crime have asked for a tabular output, showing each transaction in the 1 Half of 2015. I have emailed them asking them to reconsider their requirements because even SSRS would struggle with the vast number of rows.

       

      Being new to Power BI / Excel Data Models, I would like to understand the optimaldesign aproaches to use.

       

      So far, I have picked up from the previous replies to my quesiton (thanks everyone):

      • Use Direct Query (as opposed to import)
      • Group/Aggregate the data (as opposed to large number of "atomic" rows)

      Are there are any other important deisgn principles to follow?

      • Anonymous's avatar
        Anonymous
        Not applicable
        10 years ago

        amir_tohidi Another option is to show aggregated visuals, (giving actual value to see things) and if they want to look at the data - have them use the "Export Data". Worth seeing if this works in the Service, that way, if they want to sit and wait for the records to load, they can. 

        Best case I would think is to meet them half way, set up visuals that are meaningful, slice them down to smaller chunks, and then they can export smaller pieces. Regardless of what anyone says, a report with a million rows is meaningless, there is no way they are looking at it all. If they take that and filter it down, then have them do that prior to exporting it from Power BI... Another option would be for them to Analyze in Excel off a report if they want to get more granular. (just some thoughts).