Forum Discussion
how to read multiple objects json data using multiple relative url path in fabric.
Hi anusha_2023
Thanks for using Fabric Community.
As I understand you want to perform incremental ETL on your data from hubplanner rest api.Here are few points that might help you:
- REST API supports pagination . You can copy data from REST API which sends response in Pages using Azure data factory. Please refer to the following links for more details:
Link1
Link2 - You can use copy activity and use REST API dataset as the source and Lakehouse table as the sink dataset and load all the data into a dummy table using a FULL LOAD technique and then use that dummy table to join to the main table to perform incremental load.
You can also refer to this link for additional help: Incremental Data
Hope this helps. Please let me know if you have any further queries.- REST API supports pagination . You can copy data from REST API which sends response in Pages using Azure data factory. Please refer to the following links for more details:
Hi anusha_2023
Does the rest API support timestamps to return specific time period data? If yes, you can maintain a control table and use high watermark solution to do incremental extracts even from REST API.
Please refer to this link: Link1
This can be extended to rest as well (in fabric data pipelines) as long it excepts querystring with high watermark field.
Hope this helps. Please let me know if you have any further questions.
Hi anusha_2023
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
Otherwise, will respond back with the more details and we will try to help.
Thanks
Hi anusha_2023
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
Otherwise, will respond back with the more details and we will try to help.
Thanks