Forum Discussion
Filtering rows seems to create a row with all nulls
Hi There,
I had built this query to interpolate some budget figures across financial years. It seemed to work okay but a query with a reference to this where I attempted to pivot the data failed. Eventually I figured out that the reason it failed is that there is a row in there that is completely full of errors.
By a process of elimination I worked out that the row of errors is not there until after "RemoveNonActiveDays" below is executed. After that has executed the row exists and it causes some operations to fail. I've tried removing blank rows but that does nothing. I suspect somehow the filter operation is adding this in. Above this command I can happily filter and the error line does not exist after it any filter will cause the error.
Have I stumbled across a bug or am I doing something stupid? With either how can I get around this so that I can pivot the data?
I've included the advance edit code version of this query. Apologies if it is inelegant I'm still warming to it ;-) advice on cleverer ways to do what I want to achieve would be gratefully received.
let
//Load the source date.
Source = OData.Feed("https://emckclac.sharepoint.com/sites/pwa/_api/ProjectData"),
Projects_table = Source{[Name="Projects",Signature="table"]}[Data],
CoreSource = Table.SelectColumns(Projects_table,{"ProjectActualCost", "ProjectActualDuration", "ProjectActualFinishDate", "ProjectActualStartDate", "ProjectBudgetCost", "ProjectFinishDate", "ProjectFixedCost", "ProjectName", "ProjectStartDate", "ProjectBudget", "PlanStatus", "PlanType", "FundingSource", "ProjectCode", "PlanningGroup", "InitialBudget"}),
//get the earliest and latest date and calculate the years between them
MinDate = List.Min(CoreSource[ProjectStartDate]), //maximum due date in the source
MaxDate = List.Max(CoreSource[ProjectStartDate]), //maximum due date in the source
YearsBetweenMinAndMax=Date.Year(MaxDate) - Date.Year(MinDate), //number of years between the date
NoOfMonthsOfFunding= Table.AddColumn( CoreSource,"NoMonthsFunding",each ((Date.Year([ProjectFinishDate])-Date.Year([ProjectStartDate]))*12) + Date.Month([ProjectFinishDate])-Date.Month([ProjectStartDate])), //This figures out how many months a project will need funding for i.e. months between start and end date
//This will add a row for every month between the start of MinDate and the end of MaxDate to the table
PutDateTypesRight = Table.TransformColumnTypes(NoOfMonthsOfFunding,{{"ProjectFinishDate", type date}, {"ProjectStartDate", type date}}),
AddMonths =
Table.ExpandListColumn(
Table.AddColumn(PutDateTypesRight,"Month", each List.Generate(()=>#date(Date.Year(MinDate),1,1), each _< #date(Date.Year(MaxDate)+1,1,1),each Date.AddMonths(_,1))),"Month"),
AddFinancialYearStart = Table.AddColumn(AddMonths, "FinancialYearStart", each if Date.Month([Month]) >= 8 then Date.Year([Month]) else Date.Year([Month])-1), //works out the financial year for the start of the project
AddFinancialYearStartDate = Table.AddColumn(AddFinancialYearStart,"FinancialYearStateDate",each #date([FinancialYearStart],8,1), type date),
//Now work out what months actually need some budget to do the work and flag those to keep then remove the others
FindActiveDays = Table.AddColumn(AddFinancialYearStartDate , "BudgetType", each if [Month] >= Date.StartOfMonth([ProjectStartDate]) then if Date.AddMonths([Month],1)<=Date.EndOfMonth([ProjectFinishDate]) then "KEEPME" else "REMOVEME" else "REMOVEME"),
RemoveNonActiveDays = Table.SelectRows(FindActiveDays, each ([BudgetType] = "KEEPME")),
CalculateBudget = Table.AddColumn(RemoveNonActiveDays, "Budget", each if [InitialBudget]<>null then [InitialBudget] / [NoMonthsFunding] else 0), //work out linear amount of budget for each project over the life of the project
FinancialYearAdded = Table.AddColumn(CalculateBudget , "Financial Year", each Text.From( [FinancialYearStart]) & "/" & Text.End(Text.From(([FinancialYearStart]+1)),2)), //Add a human readable financial year
CreateFullerProjectName = Table.AddColumn(FinancialYearAdded , "ProjectIDDescription", each [ProjectCode] & " - " & [ProjectName] ),
#"Changed Type" = Table.TransformColumnTypes(CreateFullerProjectName,{{"Budget", type number}})
in
#"Changed Type"
try implement an error-elemination like this in the previous step:
FindActiveDays = Table.AddColumn(AddFinancialYearStartDate , "BudgetType", each try if [Month] >= Date.StartOfMonth([ProjectStartDate]) then if Date.AddMonths([Month],1)<=Date.EndOfMonth([ProjectFinishDate]) then "KEEPME" else "REMOVEME" else "REMOVEME" otherwise "CheckError"),
you should be able to shorten your statement like this:
if [Month] >= Date.StartOfMonth([ProjectStartDate]) and Date.AddMonths([Month],1)<=Date.EndOfMonth([ProjectFinishDate]) then "KEEPME" else "REMOVEME"
4 Replies
- Greg_Deckler
Community Champion
Could you add a step right after the step that causes errors to "Remove error rows". It is on the Home tab in the Query Editor. Right in the middle of the ribbon in the "Reduce Rows" section.
- trevb
Advocate II
It seems to be more fundemental than that. I just used a filter on a column that I knew had three values. What comes back is four rows.
If I then chose the Remove Errors nothing changes, the row remains there. If I click an error message I get
for each one of these errors
- ImkeF
Community Champion
try implement an error-elemination like this in the previous step:
FindActiveDays = Table.AddColumn(AddFinancialYearStartDate , "BudgetType", each try if [Month] >= Date.StartOfMonth([ProjectStartDate]) then if Date.AddMonths([Month],1)<=Date.EndOfMonth([ProjectFinishDate]) then "KEEPME" else "REMOVEME" else "REMOVEME" otherwise "CheckError"),
you should be able to shorten your statement like this:
if [Month] >= Date.StartOfMonth([ProjectStartDate]) and Date.AddMonths([Month],1)<=Date.EndOfMonth([ProjectFinishDate]) then "KEEPME" else "REMOVEME"