I have several queries in a Dataflow that use API calls:
1. The first query "Customers" - gets a list of customer ids from API call (https://api.somecrm.com/v1/customers)
2. Query "Purchase History" - References Query 1 and adds an "Invoke Custom Function" column. I have more queries like the second that reference the first and use other API cals per customer.
Yesterday when I started to get this error: "429 Too Many Requests" . The CRM has Rate limit of 100 calls only for the first API call.
I think that because of References - the first call is triggered many times instead of only ONCE.
Can Dataflow Gen 2 enable staging help? So the first Query is run, the result are kept somewhere, and then the other queries only reference this physical table with results, without running the first Query again and again.
Please help