Forum Discussion

anusha_2023's avatar
anusha_2023
Helper IV
2 years ago
Solved

how to read multiple objects json data using multiple relative url path in fabric.

For example I have projects data in one relative path and resources data in another relative path and I am trying to read these two dimensional data from hubplanner rest api into fabric. I have fact...
Pipelines
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi anusha_2023 
    Thanks for using Fabric Community.
    As I understand you want to perform incremental ETL on your data from hubplanner rest api. 

    Here are few points that might help you:

    1. REST API supports pagination . You can copy data from REST API which sends response in Pages using Azure data factory. Please refer to the following links for more details: 
      Link1
      Link2
    2. You can use copy activity and use REST API dataset as the source and Lakehouse table as the sink dataset and load all the data into a dummy table using a FULL LOAD technique and then use that dummy table to join to the main table to perform incremental load.

    You can also refer to this link for additional help: Incremental Data 

    Hope this helps. Please let me know if you have any further queries.

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi anusha_2023 
Thanks for using Fabric Community.
As I understand you want to perform incremental ETL on your data from hubplanner rest api. 

Here are few points that might help you:

  1. REST API supports pagination . You can copy data from REST API which sends response in Pages using Azure data factory. Please refer to the following links for more details: 
    Link1
    Link2
  2. You can use copy activity and use REST API dataset as the source and Lakehouse table as the sink dataset and load all the data into a dummy table using a FULL LOAD technique and then use that dummy table to join to the main table to perform incremental load.

You can also refer to this link for additional help: Incremental Data 

Hope this helps. Please let me know if you have any further queries.

  • anusha_2023's avatar
    anusha_2023
    Helper IV
    2 years ago

    Thanks for your response. 

    I am unclear about how to proceed with pagination when I have to iterate through pages and my response object does not give any clue about how many pages are there or does not give the next_page link or information. Can I have more information on this scenario? 

    I need to filter data based on the updated date field for example I need the latest bookings only for this week before loading data into the Lakehouse.

     

    Could you give more information regarding these issues?

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi anusha_2023 
      Does the rest API support timestamps to return specific time period data? If yes, you can maintain a control table and use high watermark solution to do incremental extracts even from REST API.
      Please refer to this link: Link1

      This can be extended to rest as well (in fabric data pipelines) as long it excepts querystring with high watermark field.

      Hope this helps. Please let me know if you have any further questions.

      • Anonymous's avatar
        Anonymous
        Not applicable
        2 years ago

        Hi anusha_2023 
        We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
        Otherwise, will respond back with the more details and we will try to help.
        Thanks