topic Re: how to read multiple objects json data using multiple relative url path in fabric. in Pipelines

how to read multiple objects json data using multiple relative url path in fabric.

anusha_2023 — Tue, 21 Nov 2023 09:11:08 GMT

For example I have projects data in one relative path and resources data in another relative path and I am trying to read these two dimensional data from hubplanner rest api into fabric.

I have fact data in bookings and need to do incremetal etl on this data.

I want to load the data into fabric and do transformations using notebook. Please suggest me the best way to do the landing of data into fabric.

Right now I have created a pipeline and added three different copy activities to the pipeline for three different types of json objetcs and load these objetcs into notebook through lakhouse.

I am unbale to setup incremental ETL scenario on bookings. Please let me suggest me the scenarios those can fit my task. Thank you!

Re: how to read multiple objects json data using multiple relative url path in fabric.

Anonymous — Wed, 22 Nov 2023 07:09:21 GMT

Hi anusha_2023
Thanks for using Fabric Community.
As I understand you want to perform incremental ETL on your data from hubplanner rest api.

Here are few points that might help you:

REST API supports pagination . You can copy data from REST API which sends response in Pages using Azure data factory. Please refer to the following links for more details:
Link1
Link2
You can use copy activity and use REST API dataset as the source and Lakehouse table as the sink dataset and load all the data into a dummy table using a FULL LOAD technique and then use that dummy table to join to the main table to perform incremental load.

You can also refer to this link for additional help: Incremental Data

Hope this helps. Please let me know if you have any further queries.

Re: how to read multiple objects json data using multiple relative url path in fabric.

anusha_2023 — Wed, 22 Nov 2023 13:00:38 GMT

Thanks for your response.

I am unclear about how to proceed with pagination when I have to iterate through pages and my response object does not give any clue about how many pages are there or does not give the next_page link or information. Can I have more information on this scenario?

I need to filter data based on the updated date field for example I need the latest bookings only for this week before loading data into the Lakehouse.

Could you give more information regarding these issues?

Re: how to read multiple objects json data using multiple relative url path in fabric.

Anonymous — Wed, 22 Nov 2023 18:29:27 GMT

Hi anusha_2023
Does the rest API support timestamps to return specific time period data? If yes, you can maintain a control table and use high watermark solution to do incremental extracts even from REST API.
Please refer to this link: Link1

This can be extended to rest as well (in fabric data pipelines) as long it excepts querystring with high watermark field.

Hope this helps. Please let me know if you have any further questions.

Re: how to read multiple objects json data using multiple relative url path in fabric.

Anonymous — Sat, 25 Nov 2023 16:01:38 GMT

Hi anusha_2023
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
Otherwise, will respond back with the more details and we will try to help.
Thanks

Re: how to read multiple objects json data using multiple relative url path in fabric.

Anonymous — Mon, 27 Nov 2023 07:48:55 GMT

Re: how to read multiple objects json data using multiple relative url path in fabric.

anusha_2023 — Tue, 05 Dec 2023 04:05:49 GMT

Thanks for attaching the pagenation material. I could able to navigate through pages when I am hardcoding the numbers. How to write a loop to go through pages in the queryparameters section of dynamic content?

Now I want to save my last run pipeline date and use that date in the current run as a query parameter. Could you please elaborate how could I able to proceed with this step? For example

@concat('{"updatedDate" : {"$gte": "',variables('lastRunDate'),'"}}')

I want to save my previous run pipeline date in lastRunDate and use that parameter in the current run as above.