Forum Discussion
icassiem
Post Prodigy
2 months agoETL API Loop with multiple paramaters
Good day, I asked the ETL API loop recently where the best method is to via the pipeline task use a python noteback pyspark scipt. I have some challenges and i dont have a bronze level to store tem...
- 2 months ago
hey icassiem I'm trying to understand what you're after and hopefully below answers could help.
- if your question is "Where I store and reuse a temporary authentication token during pipeline execution" then possible solutions are combining into one notebook (no passing token across multiple notebooks), use Fabric pipeline variables, use Notebook variable Library or even Key Vault (although this is more for secrets, not transient tokens)
- Re "3-4 loop parameters", yeah it sounds like you're building a nested-loop orchestration, which I doubt that Fabric F2 could handle and potentially you need to increase
- Not much to comment since this is more like API design to me. Does the API support pagination / bulk requests / multiple stores in one call / all products for a client in one call? I'm exploring if we can reduce from 200 calls to much less
- Re "Can't I just use Data Factory like SSIS?", It's still preview but yes Fabric will be able to execute SSIS packages using Fabric data pipeline.
https://community.fabric.microsoft.com/t5/Fabric-Updates-Blog/Invoke-SSIS-Package-Activity-in-Microsoft-Fabric-Preview/ba-p/5172121 - Re setting timeouts and capture api call failures", yes absolutely Fabric data pipeline can configure a timeout so a hung API call doesn't run forever, support Retry Count and Retry Interval settings and alert when failures happen. See below some links to MS learn to explore further.
https://github.com/MicrosoftDocs/fabric-docs/blob/main/docs/data-factory/activity-overview.md
https://community.fabric.microsoft.com/t5/Pipelines/Automatic-Retry-x-times-in-case-of-error/m-p/4614123
https://fgarofalo56.github.io/Suppercharge_Microsoft_Fabric/best-practices/error-handling-monitoring/
Appreciate if you can "Kudos" and/or "Accept as Solution" if this answered your query.
GilbertQ
Super User
2 months agoHi icassiem
From what you are saying it sounds like a notebook is the way to go with regards to this because it can complete all the things you are requesting.
- icassiem2 months ago
Post Prodigy
v-kathullac , GilbertQ , rizalard0684 Thank You so Much for helping, apologines for the delay
I am panicking a little becuase i proposed a new directions "fabric" and not i have an MVP and a POC and this complexity to think of with no python or fabric experience, Hence why i was hoping there was an example or link to share i could learn from fast. Becasue my proosal hangs on a few push backs