Forum Discussion

nl908's avatar
nl908
Regular Visitor
3 years ago

Web.Contents 40k records

Hi, I'm having a lot of trouble pulling down about 40000 records from an api using Web.Contents and List.Generate.

 

The api has a rate limit of 4000 per hour. So that is about 1 request per second (each request can get 100 records). The dataflow gets a timeout after 10 mins, theoretically it should be able to get about...

 

10 mins * 60 seconds per min * 100 records per request * 1 request per second = 60000 records

 

But there is so much waiting/timeout I can't really examine what is going on.

 

Do you know if I'm crazy to think its possible to pull down 40000 records from an api using the Power BI service?

Dataflow
Need Help
Tips and Tricks

1 Reply

  • lbendlin's avatar
    lbendlin
    Super User
    3 years ago

    what time interval are these 40000 records covering?  Are they changing every day?  Or could you consider incremental refresh?