Forum Discussion

anusha_2023's avatar
anusha_2023
Helper IV
2 years ago
Solved

how to read multiple objects json data using multiple relative url path in fabric.

For example I have projects data in one relative path and resources data in another relative path and I am trying to read these two dimensional data from hubplanner rest api into fabric. I have fact...
Pipelines
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi anusha_2023 
    Thanks for using Fabric Community.
    As I understand you want to perform incremental ETL on your data from hubplanner rest api. 

    Here are few points that might help you:

    1. REST API supports pagination . You can copy data from REST API which sends response in Pages using Azure data factory. Please refer to the following links for more details: 
      Link1
      Link2
    2. You can use copy activity and use REST API dataset as the source and Lakehouse table as the sink dataset and load all the data into a dummy table using a FULL LOAD technique and then use that dummy table to join to the main table to perform incremental load.

    You can also refer to this link for additional help: Incremental Data 

    Hope this helps. Please let me know if you have any further queries.

anusha_2023's avatar
anusha_2023
Helper IV
2 years ago

Thanks for attaching the pagenation material. I could able to navigate through pages when I am hardcoding the numbers. How to write a loop to go through pages in the queryparameters section of dynamic content?

Now I want to save my last run pipeline date and use that date in the current run as a query parameter. Could you please elaborate how could I able to proceed with this step? For example 

@concat('{"updatedDate" : {"$gte": "',variables('lastRunDate'),'"}}')
I want to save my previous run pipeline date in lastRunDate and use that parameter in the current run as above.