Forum Discussion
Unpivot removes rows with no/null values - how to keep them?
- 9 years ago
Select Monday to Friday in Query Editor, Go to Transform Tab, Replace Values ---> Replace "null" with "0".
You would see the expected results.
Select Monday to Friday in Query Editor, Go to Transform Tab, Replace Values ---> Replace "null" with "0".
You would see the expected results.
What if preserving the NULL values is the required behaviour?
Context:
in a factory, different machines each store 30 parameter values internally. At the end of every shift (once every 8 hours) each machine writes its 30 parameters to a database table. The table has 3 key columns ("machine ID", "row ID" and "DateTime" which records the moment of writing into the table) and 30 numerical columns "param01" to "param30".
The data writing process has different types of issues. I would like to count the number of bad data points for each type of issue
.
One issue causes correct numerical parameters in the machines to be written as NULL values in the database table.
(e.g. shift 1 could have param10 = NULL; shift 2 could have param10 = <correct value> but param17 = NULL; and so on. The parameter values of each data record are impacted more or less at random) (note: the 3 key columns are never NULL)
Hence I would like to count the occurrence of the "bad write" error type by counting all NULL values in the table.
I followed the solution advised here, i.e.
- unpivot the 30 parameter columns (resulting table has 5 columns: the 3 key columns + "Parameter name" + "Value")
- count all NULL values in "Value"
Problem:
Power Query does not create unpivoted rows if a parameter column has value NULL, i.e. no unpivoted row is created in this case.
(e.g. if 20 out of 30 parameters are NULL in a record in the original table, the unpivoting operation results in 10 rows each having "Value" = <a number>, instead of my desired result of 30 rows where 10 row have a number and 20 rows have "Value" = NULL)
Constraints:
- Since a machine parameter can actually be 0 and valid, replacing NULL by 0 prior to unpivoting is not allowed. This would turn a bad data point into a good one and falsify my occurrence count for the "bad write" type of error.
- Replacing NULL with a non-occurring value (e.g. -999999) prior to unpivoting solves the problem for historical data but, given the range of issues, any such value could potentially appear in a future record and be caused by a different type of error (say of type "counter error"). If this ever occurs, I would in the future start to miscount the occurrences of the "bad write" error by +1 and the occurrences of "counter error" by -1.
- I cannot re-run the data output for the machines for dates older than 30 days so correct reconstruction of historical data is impossible.
Question:
Since NULL values give me unique information, is it possible to unpivot whilst preserving NULL values?
(i.e. in such a way that unpivoted rows are created with "Value" = NULL)
- richardnlove9189 years agoFrequent Visitor
I have exactly the same requirements/contraints as Nick - PLEASE HELP US!