Forum Discussion
Post ProdigyETL API Loop with multiple paramaters
hey icassiem I'm trying to understand what you're after and hopefully below answers could help.
- if your question is "Where I store and reuse a temporary authentication token during pipeline execution" then possible solutions are combining into one notebook (no passing token across multiple notebooks), use Fabric pipeline variables, use Notebook variable Library or even Key Vault (although this is more for secrets, not transient tokens)
- Re "3-4 loop parameters", yeah it sounds like you're building a nested-loop orchestration, which I doubt that Fabric F2 could handle and potentially you need to increase
- Not much to comment since this is more like API design to me. Does the API support pagination / bulk requests / multiple stores in one call / all products for a client in one call? I'm exploring if we can reduce from 200 calls to much less
- Re "Can't I just use Data Factory like SSIS?", It's still preview but yes Fabric will be able to execute SSIS packages using Fabric data pipeline.
https://community.fabric.microsoft.com/t5/Fabric-Updates-Blog/Invoke-SSIS-Package-Activity-in-Microsoft-Fabric-Preview/ba-p/5172121 - Re setting timeouts and capture api call failures", yes absolutely Fabric data pipeline can configure a timeout so a hung API call doesn't run forever, support Retry Count and Retry Interval settings and alert when failures happen. See below some links to MS learn to explore further.
https://github.com/MicrosoftDocs/fabric-docs/blob/main/docs/data-factory/activity-overview.md
https://community.fabric.microsoft.com/t5/Pipelines/Automatic-Retry-x-times-in-case-of-error/m-p/4614123
https://fgarofalo56.github.io/Suppercharge_Microsoft_Fabric/best-practices/error-handling-monitoring/
Appreciate if you can "Kudos" and/or "Accept as Solution" if this answered your query.
Resolver IVhey icassiem I'm trying to understand what you're after and hopefully below answers could help.
- if your question is "Where I store and reuse a temporary authentication token during pipeline execution" then possible solutions are combining into one notebook (no passing token across multiple notebooks), use Fabric pipeline variables, use Notebook variable Library or even Key Vault (although this is more for secrets, not transient tokens)
- Re "3-4 loop parameters", yeah it sounds like you're building a nested-loop orchestration, which I doubt that Fabric F2 could handle and potentially you need to increase
- Not much to comment since this is more like API design to me. Does the API support pagination / bulk requests / multiple stores in one call / all products for a client in one call? I'm exploring if we can reduce from 200 calls to much less
- Re "Can't I just use Data Factory like SSIS?", It's still preview but yes Fabric will be able to execute SSIS packages using Fabric data pipeline.
https://community.fabric.microsoft.com/t5/Fabric-Updates-Blog/Invoke-SSIS-Package-Activity-in-Microsoft-Fabric-Preview/ba-p/5172121 - Re setting timeouts and capture api call failures", yes absolutely Fabric data pipeline can configure a timeout so a hung API call doesn't run forever, support Retry Count and Retry Interval settings and alert when failures happen. See below some links to MS learn to explore further.
https://github.com/MicrosoftDocs/fabric-docs/blob/main/docs/data-factory/activity-overview.md
https://community.fabric.microsoft.com/t5/Pipelines/Automatic-Retry-x-times-in-case-of-error/m-p/4614123
https://fgarofalo56.github.io/Suppercharge_Microsoft_Fabric/best-practices/error-handling-monitoring/
Appreciate if you can "Kudos" and/or "Accept as Solution" if this answered your query.
Post ProdigyGilbertQ , rizalard0684 Thank You Very much
I am only starting, i have no clue of notebooks - do you have an example of the getting the token from api and then use it to do nested looping within one notebook please?
With SSIS i have knowledge but dont i need to have an enterprise SQL license to have SSIS and execute in Fabric? and its only in preview stage correct? I only have F2 and F4 was declined, to have Azure Data factory SSIS i require license, i was hoping that this moved to fabric and SSIS/Datafactory was available within Fabirc? I asked in another thread if Data flow is the new ssis/data factory, i dont have any other licensing, so this means i cant use ssis/azure data factory in fabric correct or is there a local/pc free ssis i can import to fabric?