Forum Discussion

Tinus1905's avatar
Tinus1905
Resolver I
2 years ago

Best approach API calls

I'm using PowerBi to get files from the API. The total amount of records are more than 500.000. So to speed up the progress I made an manual export of the site from beginning date -> yesterday. The ...
Need Help
EmanuelTavares's avatar
EmanuelTavares
Advocate II
2 years ago

Hi Tinus1905 

 

Having two tables to separate historical data from recent makes no sense to me.

 

I suggest you create an incremental ETL process and read all data from a table. If can set the process frequency to update the database many times daily.

 

You also can run this process using a serverless service, such as Azure Functions or AWS Lambda.

 

Regards.

 

Emanuel