Forum Discussion

kroman's avatar
kroman
Icon for Helper II rankHelper II
3 years ago
Solved

Getting sorted data in bunches

Hi All We have a dataset with 100K rows in the following format: And we have a flow that is getting data from this dataset Due to the fact that flow cannot get all 100k rows in one go we are...
Flow
Large Datasets
  • johnt75's avatar
    johnt75
    3 years ago

    You could use the WINDOW function, that allows you to order by whichever column you want and you can provide the offsets you need.

kroman's avatar
kroman
Icon for Helper II rankHelper II
3 years ago

Thanks for your reply     

  I am trying it with WINDOW with following query on a 100 rows table

     EVALUATE  
         VAR Window1 =  WINDOW(1, REL, 5 , REL, Data100Rows, MATCHBY([Index]))
    RETURN Window1

 

But it returns all 100 rows instead of 5 rows

johnt75's avatar
johnt75
Icon for Super User rankSuper User
3 years ago

I think it wants to be ABS, not REL

EVALUATE
VAR Window1 =
    WINDOW ( 1, ABS, 5, ABS, Data100Rows, MATCHBY ( [Index] ) )
RETURN
    Window1
  • kroman's avatar
    kroman
    Icon for Helper II rankHelper II
    3 years ago

    😪

    • johnt75's avatar
      johnt75
      Icon for Super User rankSuper User
      3 years ago

      If you're getting all the data each time, does it matter what order it is in ? Also, have you considered using a Live Connected xlsx file rather than exporting to CSV ?

      • kroman's avatar
        kroman
        Icon for Helper II rankHelper II
        3 years ago

        100 rows is just for testing
        we have a 200k rows dataset which we need to convert into csv with a Flow (powerautomate), and flow can only take 25-30k rows in one go, so I am doing some looping to get all the data. But we also need to sort that data by different columns before getting it to csv