Forum Discussion

Coffeeaddict's avatar
Coffeeaddict
Frequent Visitor
6 years ago

dataflows and service storage query

Hi,

 

I have a question as to what is stored within the power BI service storage space and what is processed in the memory but not stored.

 

I'm in the process of creating several large sequencial dataflows, 

The first dataflow beind a upload from a source (lets call  this input). 

After this there can be muliple dataflows feeing from the input dataflow and i want to understand just want data is the storage space used for.

 

for example lets say "input" has 3 entities each of 10gb

 

transform 1 is fed from input using 2 of the entities (entity_1+entity_2), these are merged into another entity "tranform_1_output" which is 7gb in size

 

transform 2 if fead from transform 1 output and input entity 3, in this dataflow input_entity_3 is referenced and grouped into a a new entity (trans_2_groupings) which is 3gb in size this is merged as new with the transform 1 output to create a final entity (Desired_output) which is 5gb in size.

 

My questions is how much storage space is used in the service.  

 

I am thinking there are 2 options.

 

the first being all entities have enable load ticked:

             Input - 3 entities 5gb each = 15Gb

             Transform 1 - 2 intial entities (5Gb) each (10gb) + output entity (7gb) = 17gb

             Transform 2 - tranform 1 output (7Gb) + input_entity_3 (5Gb) + trans_2_grouping (3Gb) + Desired_output (5GB) = 20GB

       Total Storage = 52Gb

 

Option 2 where apart form the input dataflow all other dataflows only have enable load on the final entity

             Input = 15Gb

             Transform 1 = 7Gb

             Transform 2 = 5Gb

        Total Storage = 27Gb

 

then again both these may be wrong but if someone knows can you let me know thanks

 

(this is just an example dataflow sizes and compexity are much bigger)

 

2 Replies