Forum Discussion

RamonNooijen's avatar
RamonNooijen
New Member
8 months ago
Solved

Issue with Dataflow Gen2 with date variable

I have a question, I want to ingest data from API's where I have to call multiple API's based on issue ID's to get the history from an issue and this history data is pretty long. To make my question/...
  • Ugk161610's avatar
    8 months ago

    Hi RamonNooijen ,

     

    The error message is actually giving the clue:

    We cannot apply operator < to types Table and Date

     

    That means WM is not a single date value at runtime – it’s still a table (or query result) when the dataflow engine tries to fold the query and push it to the Lakehouse / DW.

     

    When you hard-code WM = #date(2025, 12, 3) it works, because WM is clearly a scalar date.
    When you load it from SQL, your WM step is returning a table (for example one row, one column), not the cell value itself.

     

    You need one extra step to turn that table into a single date before using it in Table.SelectRows.

     

    A common pattern looks like this in Power Query (M):

     

    // Query that gets the watermark table from SQL
    WMTable = Sql.Database("server", "db"){[Name="WatermarkTable"]}[Data],

    // Take the first row and the Watermark column as a scalar value
    WM = Date.From( WMTable{0}[WatermarkDate] ),

     

    Then in your filter step:

    FilteredRows =
    Table.SelectRows(
    IssuesTyped,
    each [updated_at] >= WM
    )

     

    Key idea:

     

    • WMTable = table returned from SQL

    • WM = single Date value extracted from that table

    Right now you’re comparing [updated_at] (a date) to WM (a table), which is why you get the “Table and Date” error only when it actually runs in the service.

     

    Once WM is a true date value, your dynamic filter will work the same way as the hard-coded example.

    – Gopi Krishna