Forum Discussion
FHIR Connector is loading slowly large data sets
We are experiencing huge delays when loading big datasets from our FHIR Server.
I am talking about 1 Mio+ records, which take close to a day to fully load.
The main problem seems to be the way how the FHIR Connector is loading the data:
- Small sets with small offset (about 200 for each request)
- No utilization of FHIR export feature
Is there any way to tweak the connector e.g. by overwriting the default offset size?
Our server would have no problems to serve sth. like 25k records with each request.
3 Replies
- DataInsights
Super User
A colleague alerted me to this interesting technique, and got a 10x improvement in his query. Let me know if it works for you.
Hi DataInsights ,
thanks for the advice. It's pretty nice explained, but only works for local datasets.
However the problem with FHIR is that it takes the most time to load all datapoints 1by1 (buckets of 100-500) from the server instead of incresing the offset size. So you loose a lot of time with establishing unnecessary requests.
Any update on this issue?
The problem is still existing.I honestly don't understand why the PBI FHIR Connector does not ship with an option to set the page size. Queryfolding is supported which is much harder to implement.
Providing a parameter to set the page size should be easy to be tackled by the team, would love to see it on the roadmap.
