Forum Discussion
Calculating kWh based on datetime and kW
- 6 years ago
This turned out to be much more challenging than expected. Power BI kept running out of memory even though it is running on 32 GB. I ended up presorting the data in SQL server (and adding the index row there)
SELECT ROW_NUMBER() OVER(ORDER BY [Telemetry AssetID ] ,[Telemetry PointName ] ,[Telemetry Timetag ] asc) Row, [Telemetry SiteID ] [SiteID] ,[Telemetry AssetID ] [AssetID] ,[Telemetry PointName ] [PointName] ,[Telemetry Timetag ] [Timetag] ,[Telemetry Units ] [Units] ,[Telemetry ELECTRICITY_MAP ] [ELECTRICITY_MAP] ,[Telemetry StoreID ] [StoreID] ,[Telemetry date_key ] [date_key] FROM [Telemetry]Then the calculated column can be written as such: (shown in two variations)
kWh = var i = Telemetry[Row] var t = Telemetry[Timetag] var ap = Telemetry[AssetID] & Telemetry[PointName] var f = filter(Telemetry,Telemetry[Row]=i-1) var pt = SELECTCOLUMNS(f,"pt",Telemetry[Timetag]) var pv = SELECTCOLUMNS(f,"pv",Telemetry[ELECTRICITY_MAP]) var pap= SELECTCOLUMNS(f,"pap",Telemetry[AssetID] & Telemetry[PointName]) //var pt = maxx(f,Telemetry[Timetag]) //var pv = maxx(f,Telemetry[ELECTRICITY_MAP]) //var pap = maxx(f,Telemetry[AssetID] & Telemetry[PointName]) return if (ap=pap,divide(Telemetry[ELECTRICITY_MAP]+pv,120)*DATEDIFF(pt,t,MINUTE),0)and the calculation yields the expected result.
Thank you for the challenge. I'm still baffled that Power BI struggles with such a seemingly small dataset (7.8M rows) but I guess there are a few cartesian products happening that explode the memory needs. Hope someone else can find a faster solution.
I don't see a reason why a calculated column would not work with incremental refresh - do you have anything to reproduce that?
Please provide sanitized sample data that fully covers your issue. If you paste the data into a table in your post or use one of the file services it will be easier to work with. Please show the expected outcome.
Thanks for your quick reply! When i activated the incremental refresh there was an error after the refresh on the visuals that the calculated column had no data in it. When I removed incremental refresh, it worked again. I think it's very similar to this case: Incremental Update Issue: Calculated Column Data D... - Microsoft Power BI Community
Here is a fraction of my dataset: https://www.transfernow.net/dl/20220208JpCf11jW
In my full dataset i have data from 2019 till today. What i would like to get kWh per Day, Month and Year.
- lbendlin4 years ago
Super User
I'll look at the data but I think might be a misunderstanding of how incremental refresh works. The post you link to seems to be based on the same misunderstanding.
Whenever you make structural (meta data) changes to an incremental refresh data source and then publish from Power BI Desktop to the workspace the following things will happen:
- All partitions are removed, and replaced by a single partition based on your upload (governed by the RangeStart and RangeEnd settings in Power Query, NOT by the incremental refresh settings in Power BI)
- If scheduled refresh is enabled for the dataset then a new refresh is triggered. Otherwise nothing will happen until you initiate a manual refresh
- upon this new refresh all partitions are recreated and recomputed based on the settings in Power BI, NOT on the Power Query settings.
- Any refresh request after that will execute based on the incremental refresh rules you indicated in Power BI.
This is the basic process. Advanced users will use XMLA queries to selectively refresh individual partitions, and use ALM toolkit to selectively modify meta data.
- Anonymous4 years agoNot applicable
Alright, i will try it again with the incremental refresh settings - maybe i messed something up, this is a very new topic for me 😉
- lbendlin4 years ago
Super User
Here is the calculated column that computes the integral. Your timestamp granularity is good enough so you don't have to worry about the day passage. Performance is still bad, so I'll try Power Query next.
kWh = var ts = 'data (1)'[Timestamp] var pts = CALCULATE(max('data (1)'[Timestamp]),'data (1)'[Timestamp]<ts) var ptsw = CALCULATE(sum('data (1)'[kW]),'data (1)'[Timestamp]=pts) return divide('data (1)'[kW]+ptsw,2,0)*24*(ts-pts)