Forum Discussion

joshua1990's avatar
joshua1990
Icon for Post Prodigy rankPost Prodigy
5 years ago

Will Date as Number break Query Folding / Incremental refresh?

Hello together!

I am accessing a transactional table (oracle / ODBC Connection type) that has several million rows.
I would like to update this table based on incremental refresh in Power BI Desktop.
This table has a column "Date Entry" which is a number in the format "CYYDDD".
For example, this means that the date is 01/01/2019 = 119001.
To put this number in a date format like "DD.MM.YYYY" I have already found a function.
But this function will stop the query folding. Anyway, after this step, I can't see a Native Query anymore.

 

Now I have the question if I can still somehow apply the incremental refresh?
What recommendations do you guys have?

 

3 Replies

  • artemus's avatar
    artemus
    Icon for Microsoft Employee rankMicrosoft Employee

    Here is one way to do it:

    Replace the Source line with your table. It expects a [DateEntry] column with the date string

     

     

    let
        Source = ...,
        #"Filtered Rows" = Table.SelectRows(Source, each [AddressID] = 1),
        #"Added Custom" = Table.AddColumn(#"Filtered Rows", "DateEntry", each 107263, type number),
        Custom2 = Table.AddColumn(#"Added Custom", "Century",each Number.RoundDown([DateEntry] / 100000) + 19, Int32.Type),
        #"Added Custom1" = Table.AddColumn(Custom2, "Year", each Number.RoundDown(Number.Mod([DateEntry], 100000) / 1000), Int32.Type),
        #"Added Custom2" = Table.AddColumn(#"Added Custom1", "LeapYearOffset", each if Number.Mod([Year], 4) = 0 and (Number.Mod([Century], 4) = 0
    or [Year] <> 0) then 1 else 0, Int32.Type),
        #"Added Custom3" = Table.AddColumn(#"Added Custom2", "MonthDayPart", each Number.Mod([DateEntry], 1000)),
        #"Added Custom4" = Table.AddColumn(#"Added Custom3", "MonthDay", let monthTable = #table(type table [MoNum = Int32.Type, Days = Int32.Type], List.Zip({{1 .. 12}, List.Accumulate({31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31}, {}, (current, next) => current & {List.Last(current, 0) + next})})) in Expression.Evaluate("each " & Text.Combine(Table.TransformRows(monthTable, each "if [MonthDayPart] <= " & Text.From([Days]) & (if [MoNum] = 1 then "" else " + [LeapYearOffset]") & " then " & Text.From([MoNum]) & " + ([MonthDayPart] - " & Text.From(monthTable{[MoNum = _[MoNum] - 1]}?[Days]? ?? 0) & ") / 100"), " else ") & " else 0")),
        #"Added Custom5" = Table.AddColumn(#"Added Custom4", "Month", each Number.RoundDown([MonthDay])),
        #"Added Custom6" = Table.AddColumn(#"Added Custom5", "Day", each Number.Mod(Number.RoundDown([MonthDay] *  100), 100)),
        #"Added Custom7" = Table.AddColumn(#"Added Custom6", "Date", each Text.From([Century]) & (if [Year] <  10 then "0" else "") & Text.From([Year]) & "/" & Text.From([Month]) & "/" & Text.From([Day]), type text)
    in
        #"Added Custom7"

     

     

     

    Native generated query:

     

     

    select [_].[AddressID] as [AddressID],
        [_].[DateEntry] as [DateEntry],
        [_].[Century] as [Century],
        [_].[Year] as [Year],
        [_].[LeapYearOffset] as [LeapYearOffset],
        [_].[MonthDayPart] as [MonthDayPart],
        [_].[MonthDay] as [MonthDay],
        [_].[Month] as [Month],
        [_].[Day] as [Day],
        (((((convert(nvarchar(max), [_].[Century]) + (case
            when [_].[Year] < 10
            then '0'
            else ''
        end)) + convert(nvarchar(max), [_].[Year])) + '/') + convert(nvarchar(max), [_].[Month])) + '/') + convert(nvarchar(max), [_].[Day]) as [Date]
    from 
    (
        select [_].[AddressID] as [AddressID],
            [_].[DateEntry] as [DateEntry],
            [_].[Century] as [Century],
            [_].[Year] as [Year],
            [_].[MonthDayPart] as [MonthDayPart],
            [_].[LeapYearOffset] as [LeapYearOffset],
            [_].[MonthDay] as [MonthDay],
            floor([_].[MonthDay]) as [Month],
            floor([_].[MonthDay] * 100) % 100 as [Day]
        from 
        (
            select [_].[AddressID] as [AddressID],
                [_].[DateEntry] as [DateEntry],
                [_].[Century] as [Century],
                [_].[Year] as [Year],
                [_].[MonthDayPart] as [MonthDayPart],
                [_].[LeapYearOffset] as [LeapYearOffset],
                case
                    when [_].[MonthDayPart] <= 31
                    then 1 + convert(decimal(38,6), [_].[MonthDayPart] - 0) / convert(decimal(38,6), 100)
                    when [_].[MonthDayPart] <= 59 + [_].[LeapYearOffset]
                    then 2 + convert(decimal(38,6), [_].[MonthDayPart] - 31) / convert(decimal(38,6), 100)
                    when [_].[MonthDayPart] <= 90 + [_].[LeapYearOffset]
                    then 3 + convert(decimal(38,6), [_].[MonthDayPart] - 59) / convert(decimal(38,6), 100)
                    when [_].[MonthDayPart] <= 120 + [_].[LeapYearOffset]
                    then 4 + convert(decimal(38,6), [_].[MonthDayPart] - 90) / convert(decimal(38,6), 100)
                    when [_].[MonthDayPart] <= 151 + [_].[LeapYearOffset]
                    then 5 + convert(decimal(38,6), [_].[MonthDayPart] - 120) / convert(decimal(38,6), 100)
                    when [_].[MonthDayPart] <= 181 + [_].[LeapYearOffset]
                    then 6 + convert(decimal(38,6), [_].[MonthDayPart] - 151) / convert(decimal(38,6), 100)
                    when [_].[MonthDayPart] <= 212 + [_].[LeapYearOffset]
                    then 7 + convert(decimal(38,6), [_].[MonthDayPart] - 181) / convert(decimal(38,6), 100)
                    when [_].[MonthDayPart] <= 243 + [_].[LeapYearOffset]
                    then 8 + convert(decimal(38,6), [_].[MonthDayPart] - 212) / convert(decimal(38,6), 100)
                    when [_].[MonthDayPart] <= 273 + [_].[LeapYearOffset]
                    then 9 + convert(decimal(38,6), [_].[MonthDayPart] - 243) / convert(decimal(38,6), 100)
                    when [_].[MonthDayPart] <= 304 + [_].[LeapYearOffset]
                    then 10 + convert(decimal(38,6), [_].[MonthDayPart] - 273) / convert(decimal(38,6), 100)
                    when [_].[MonthDayPart] <= 334 + [_].[LeapYearOffset]
                    then 11 + convert(decimal(38,6), [_].[MonthDayPart] - 304) / convert(decimal(38,6), 100)
                    when [_].[MonthDayPart] <= 365 + [_].[LeapYearOffset]
                    then 12 + convert(decimal(38,6), [_].[MonthDayPart] - 334) / convert(decimal(38,6), 100)
                    else 0
                end as [MonthDay]
            from 
            (
                select [_].[AddressID] as [AddressID],
                    [_].[DateEntry] as [DateEntry],
                    [_].[Century] as [Century],
                    [_].[Year] as [Year],
                    [_].[MonthDayPart] as [MonthDayPart],
                    case
                        when ([_].[Year] % 4 = 0 and [_].[Year] % 4 is not null) and ([_].[Century] % 4 = 0 and [_].[Century] % 4 is not null or ([_].[Year] <> 0 or [_].[Year] is null))
                        then 1
                        else 0
                    end as [LeapYearOffset]
                from 
                (
                    select [_].[AddressID] as [AddressID],
                        [_].[DateEntry] as [DateEntry],
                        floor(convert(decimal(38,6), [_].[DateEntry]) / convert(decimal(38,6), 100000)) + 19 as [Century],
                        floor(convert(decimal(38,6), [_].[DateEntry] % 100000) / convert(decimal(38,6), 1000)) as [Year],
                        [_].[DateEntry] % 1000 as [MonthDayPart]
                    from 
                       <<My Table Source>>
                ) as [_]
            ) as [_]
        ) as [_]
    ) as [_]

     

     

    • joshua1990's avatar
      joshua1990
      Icon for Post Prodigy rankPost Prodigy

      Thank you so much! It works almost perfectly fine. But somehow I get the result "32" as day for 

      120091

       Can you help artemus ?

  • aj1973's avatar
    aj1973
    Icon for Community Champion rankCommunity Champion

    Hi joshua1990 

    Some Query folding are not supported by ODBC Driver connection. I recommend that you apply those steps in your model instead.