Learn from the best! Meet the four finalists headed to the FINALS of the Power BI Dataviz World Championships! Register now
Hi Everyone !
I have been trying to pivot the below data by wrt to Account_L2 and Values.
There are around 5 million such rows
But it is taking a hell lot of time and finally crashing my power BI.
The source is Dataflow.
Please help
| Month | Year | ZONE 1 | ZONE 2 | CLUSTER 1 | CLUSTER 2 | CLUSTER 3 | CLUSTER 4 | COUNTRY | FDT Cluster | NBM Countries | Franchise | Franchise#2 | Division#1 | Division#2 | PRODUCT FAMILY | PRODUCT | MAIN PRODUCT | EXT - SEGMENT | EXT - ASSET TYPE | EXT - PRODUCT FAMILY NEW | INT - SEGMENT | INT - ASSET TYPE | INT - PRODUCT FAMILY NEW | MARKET#1 | MARKET#2 | Other information #1 | Other information #2 | Account_L2 | INT - FDT SEGMENT | Rates | ATT_DIV | ATT_EnActivite | MC Code | Account Code | Product Code | Extracted Rates | Attribute | Value |
| Dec | 2021 | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GROSS MARGIN | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | 713000 | |
| Dec | 2021 | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GROSS MARGIN | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | -713000 | |
| Dec | 2021 | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GROSS MARGIN | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | 0 | |
| Dec | 2021 | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GROSS MARGIN | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | GENERAL MEDICINES | 0 |
Solved! Go to Solution.
Hi @Gopal2704 ,
Pivot/Unpivot are 'whole-table' operations i.e. Power Query must load the ENTIRE table into memory in order to perform the transformation.
For such large tables, you have a few options:
-1- Don't perform the pivot at all and see if it's possible to use the data in its native state e.g. by using matrix visuals etc.
-2- Perform the pivot operation at source e.g. write a view on the server that pivots the data before being brought in to Dataflow/Power Query.
-3- Use the original source (SQL Server etc.) as your query source (instead of a Dataflow) and utilise query folding to perform the operation at source.
-4- Break up/filter your table into much smaller pieces before performing the pivot/unpivot.
-5- Invest in RAM to go into your gateways/desktop/laptop, cranking them all up to 48GB-64GB+.
You may also find performance improvements by doing the pivot in DAX, although this will likely also be highly dependent on machine RAM availability as well.
Pete
Proud to be a Datanaut!
Hi @Gopal2704 ,
Pivot/Unpivot are 'whole-table' operations i.e. Power Query must load the ENTIRE table into memory in order to perform the transformation.
For such large tables, you have a few options:
-1- Don't perform the pivot at all and see if it's possible to use the data in its native state e.g. by using matrix visuals etc.
-2- Perform the pivot operation at source e.g. write a view on the server that pivots the data before being brought in to Dataflow/Power Query.
-3- Use the original source (SQL Server etc.) as your query source (instead of a Dataflow) and utilise query folding to perform the operation at source.
-4- Break up/filter your table into much smaller pieces before performing the pivot/unpivot.
-5- Invest in RAM to go into your gateways/desktop/laptop, cranking them all up to 48GB-64GB+.
You may also find performance improvements by doing the pivot in DAX, although this will likely also be highly dependent on machine RAM availability as well.
Pete
Proud to be a Datanaut!
Thank you for providing a detailed solution Pete !!
A new Power BI DataViz World Championship is coming this June! Don't miss out on submitting your entry.
Share feedback directly with Fabric product managers, participate in targeted research studies and influence the Fabric roadmap.
| User | Count |
|---|---|
| 5 | |
| 4 | |
| 4 | |
| 3 | |
| 3 |
| User | Count |
|---|---|
| 11 | |
| 10 | |
| 7 | |
| 7 | |
| 7 |