Forum Discussion

db042190's avatar
db042190
Impactful Individual
11 days ago

is dataflow gen2 worth the wait

hi, we have about 3 flavors of a rest based orders api feed landing in a lh.   The first is based on a copy job which if memory serves (my head is spinning) cant be extended to pass params to the web api call.   The second is a dataflow gen2 source and dest being the api and lh respectively but it complains when we run it that our gateway isnt the latest and therefore it wont work.  We believe this to be parameter"izable".  The 3rd is a pipeline to which we added a "copy activity" (i think job not data) with a rest source and lh destination.   This runs and we believe it lands the 10 records (we have to prove it but the history log says so) we asked for in a reasonable amount of time and with no complaints.   We can upgrade the gateway (we've had to before) but is the df gen2 a better choice or worse choice than the pipe line with copy activity?   we believe the pipeline is also parameter"izable in that its params can be passed thru to the copy's  source url.

Need Help

5 Replies

  • GilbertQ's avatar
    GilbertQ
    Super User
    11 days ago

    Hi db042190​ 

    If you are specifically getting data from an API, I would highly recommend using A pure Python notebook. This allows you to pass parameters through to the Python as well as there is a lot of support for Python in terms of API when working with rate limits as well as a notebook consume significantly less capacity units than a data flow Gen 2.

  • tayloramy's avatar
    tayloramy
    Super User
    10 days ago

    I second what GilbertQ​ said. 

    Python is the way to go for this to be scalable and maintainable, and for it to not drive capacity usage through the roof. 

  • v-aatheeque's avatar
    v-aatheeque
    Community Support
    6 days ago

    Hi db042190​ 

    Following up to confirm if the earlier responses addressed your query. If not, please share your questions and we’ll assist further.

    • v-aatheeque's avatar
      v-aatheeque
      Community Support
      2 days ago

      Hi db042190​ 

      Have you had a chance to look through the responses shared earlier? If anything is still unclear, we’ll be happy to provide additional support.

  • ShivekMaharaj's avatar
    ShivekMaharaj
    Power Participant
    10 days ago

    Hi db042190​,

    I agree that Python gives you the most control, especially once rate limits or more complex API logic enter the picture, but I would not rule out the pipeline yet.

    Fabric's REST Copy activity supports a relative URL, GET/POST requests, headers and pagination, and Fabric pipeline parameters can be used through dynamic content. So your idea of passing parameters into the REST source URL is supported.

    The fact that your current Copy activity consistently lands 10 rows makes me wonder whether you are simply getting the API's first page. I would check what the orders API returns for pagination and configure the Copy activity's pagination rules accordingly before replacing it with a notebook.

    On the Dataflow Gen2 side, the Web API connector can use no gateway, an on-premises gateway, or a virtual network gateway. So if this is a publicly reachable API, Dataflow Gen2 does not inherently require the on-premises gateway. The gateway warning may instead be related to how that particular connection is currently configured.

    For straightforward REST-to-Lakehouse ingestion, I would personally try the parameterized REST Copy activity with pagination first. I would move to Python if you need custom retry/backoff, irregular pagination, rate-limit handling or more complex response processing.

    AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.