Forum Discussion
how to read multiple objects json data using multiple relative url path in fabric.
For example I have projects data in one relative path and resources data in another relative path and I am trying to read these two dimensional data from hubplanner rest api into fabric.
I have fact data in bookings and need to do incremetal etl on this data.
I want to load the data into fabric and do transformations using notebook. Please suggest me the best way to do the landing of data into fabric.
Right now I have created a pipeline and added three different copy activities to the pipeline for three different types of json objetcs and load these objetcs into notebook through lakhouse.
I am unbale to setup incremental ETL scenario on bookings. Please let me suggest me the scenarios those can fit my task. Thank you!
Hi anusha_2023
Thanks for using Fabric Community.
As I understand you want to perform incremental ETL on your data from hubplanner rest api.Here are few points that might help you:
- REST API supports pagination . You can copy data from REST API which sends response in Pages using Azure data factory. Please refer to the following links for more details:
Link1
Link2 - You can use copy activity and use REST API dataset as the source and Lakehouse table as the sink dataset and load all the data into a dummy table using a FULL LOAD technique and then use that dummy table to join to the main table to perform incremental load.
You can also refer to this link for additional help: Incremental Data
Hope this helps. Please let me know if you have any further queries.- REST API supports pagination . You can copy data from REST API which sends response in Pages using Azure data factory. Please refer to the following links for more details:
6 Replies
Hi anusha_2023
Thanks for using Fabric Community.
As I understand you want to perform incremental ETL on your data from hubplanner rest api.Here are few points that might help you:
- REST API supports pagination . You can copy data from REST API which sends response in Pages using Azure data factory. Please refer to the following links for more details:
Link1
Link2 - You can use copy activity and use REST API dataset as the source and Lakehouse table as the sink dataset and load all the data into a dummy table using a FULL LOAD technique and then use that dummy table to join to the main table to perform incremental load.
You can also refer to this link for additional help: Incremental Data
Hope this helps. Please let me know if you have any further queries.Thanks for your response.
I am unclear about how to proceed with pagination when I have to iterate through pages and my response object does not give any clue about how many pages are there or does not give the next_page link or information. Can I have more information on this scenario?
I need to filter data based on the updated date field for example I need the latest bookings only for this week before loading data into the Lakehouse.
Could you give more information regarding these issues?
Hi anusha_2023
Does the rest API support timestamps to return specific time period data? If yes, you can maintain a control table and use high watermark solution to do incremental extracts even from REST API.
Please refer to this link: Link1
This can be extended to rest as well (in fabric data pipelines) as long it excepts querystring with high watermark field.
Hope this helps. Please let me know if you have any further questions.
- REST API supports pagination . You can copy data from REST API which sends response in Pages using Azure data factory. Please refer to the following links for more details:
Thanks for attaching the pagenation material. I could able to navigate through pages when I am hardcoding the numbers. How to write a loop to go through pages in the queryparameters section of dynamic content?
Now I want to save my last run pipeline date and use that date in the current run as a query parameter. Could you please elaborate how could I able to proceed with this step? For example
@concat('{"updatedDate" : {"$gte": "',variables('lastRunDate'),'"}}')I want to save my previous run pipeline date in lastRunDate and use that parameter in the current run as above.
