Forum Discussion
Filter outlier data rows from a data table in power bi
Hello guys,
I am a new user of power bi desktop and I have manged to do some tasks in power bi so far.
At the moment I am going to filter some recordes of data from transaction table before i use the data for other purposes. I have a transaction table which has many outlier data (e.g out of range order value ) .
I have created a formula to filter them based on Standard deviation of the exisiting order value data. First, I would to sort the order values from max to min and then calculate STD of those data.then i filter the max row one by one until the STD<200.I stucked to do this job in Power bi....Any soloution can help me to do this task?I want this procedure be done whenever the transaction table is updated with new data.
Thank you in advance
Mehdi
- Anonymous10 years ago
myti What is the source of your data? And this sounds like something you would want to do on the import side of things, so i'd recommend you try to get some feedback from ImkeF, she's the master of "M" related solutions.
myti: You shouldn't mark your question as answered: If @Eno1978 Eno hadn't posted me in I wouldn't have looked at it :-)
Fortunately, there is a function for Standard Deviation in M already. But you need a trick in order to apply it to the needed list / to create the lists with different length. Therefore you add an Index-column after you sorted your values. Then you create the lists by specifying all values from the column up to the current row. So your code would look like this:
let Source = Excel.CurrentWorkbook(){[Name="Tabelle2"]}[Content], #"Sorted Rows" = Table.Sort(Source,{{"List", Order.Ascending}}), #"Added Index" = Table.Buffer(Table.AddIndexColumn(#"Sorted Rows", "Index", 0, 1)), #"Added Custom" = Table.AddColumn(#"Added Index", "Custom", each List.StandardDeviation({#"Added Index"[List]{0}..#"Added Index"[List]{[Index]}})) in #"Added Custom"Where "Tabelle2" is the source-table with column "List" with the values to sort.
12 Replies
- AnonymousNot applicable
- myti
Helper II
My data is online sales data that I import from Azure database to power bi.Should I post my question in those communities?
thanks
- AnonymousNot applicable
myti hmm. Is there a reason you don't just create a view on the SQL side? This is the right community, and the "M" would be applicable to filter the data prior to it being loaded if you were using import. But if you are using direct query, then I would just do everything in SQL.
My initial assumption is that you were using some sort of flat file, so I recommended ImkeF. But if you are using SQL, I would suggest you just filter it down in a view, or write a query and execute the query in Power BI.