Forum Discussion
ETL API Loop with multiple paramaters
hey icassiem I'm trying to understand what you're after and hopefully below answers could help.
- if your question is "Where I store and reuse a temporary authentication token during pipeline execution" then possible solutions are combining into one notebook (no passing token across multiple notebooks), use Fabric pipeline variables, use Notebook variable Library or even Key Vault (although this is more for secrets, not transient tokens)
- Re "3-4 loop parameters", yeah it sounds like you're building a nested-loop orchestration, which I doubt that Fabric F2 could handle and potentially you need to increase
- Not much to comment since this is more like API design to me. Does the API support pagination / bulk requests / multiple stores in one call / all products for a client in one call? I'm exploring if we can reduce from 200 calls to much less
- Re "Can't I just use Data Factory like SSIS?", It's still preview but yes Fabric will be able to execute SSIS packages using Fabric data pipeline.
https://community.fabric.microsoft.com/t5/Fabric-Updates-Blog/Invoke-SSIS-Package-Activity-in-Microsoft-Fabric-Preview/ba-p/5172121 - Re setting timeouts and capture api call failures", yes absolutely Fabric data pipeline can configure a timeout so a hung API call doesn't run forever, support Retry Count and Retry Interval settings and alert when failures happen. See below some links to MS learn to explore further.
https://github.com/MicrosoftDocs/fabric-docs/blob/main/docs/data-factory/activity-overview.md
https://community.fabric.microsoft.com/t5/Pipelines/Automatic-Retry-x-times-in-case-of-error/m-p/4614123
https://fgarofalo56.github.io/Suppercharge_Microsoft_Fabric/best-practices/error-handling-monitoring/
Appreciate if you can "Kudos" and/or "Accept as Solution" if this answered your query.
Hi icassiem ,
Thanks GilbertQ , rizalard0684 ,for addressing the issue.
If you're just getting started with notebooks. There are examples that demonstrate how to:
- Authenticate against an API and retrieve an access token.
- Store the token in a variable.
- Use the token in subsequent API requests.
- Perform nested loops (for example, looping through pages, IDs, or child resources) within a single notebook using Python.
This is a common pattern in Fabric notebooks and is typically implemented using Python libraries such as requests
Regarding SSIS and Azure Data Factory in Fabric:
- Fabric does not currently provide a native SSIS runtime that allows you to import and execute existing SSIS packages directly.
- To run SSIS packages, you would typically need Azure Data Factory's Azure-SSIS Integration Runtime, which is a separate Azure service and incurs additional Azure costs.
- This capability is not included with an F2 or F4 Fabric capacity.
- Fabric Data Factory provides Data Pipelines and Dataflows Gen2, but these are not a direct replacement for SSIS. They support many ETL/ELT scenarios but do not execute ,dtsx packages.
- There is also no local/free version of SSIS that can be imported into Fabric and executed natively.
If you only have an F2 capacity and no Azure Data Factory or Azure-SSIS Integration Runtime, then your options within Fabric are primarily:
-
Fabric Notebooks (Python, Spark)
-
Fabric Data Pipelines
-
Dataflows Gen2
For API-based ingestion, a Fabric Notebook is generally the most flexible approach, as it allows you to handle authentication, looping, pagination, and complex API logic within a single notebook.
If you're new to notebooks, starting with a simple example that retrieves an API token and then calls the API in a loop is a good way to become familiar with the workflow.
Please let us know if you need any further assistance.
Thanks,
Chaithanya.