Large Volume of data broken up in chunks with multiple API Calls

Are you using Fabric? If so, as a data mesh solution, there is a feature to share the data

To provide datasets to external users, you can utilize the External Data Sharing feature in Microsoft Fabric. This feature allows Fabric users to share data from within their Fabric tenant with users in another Fabric tenant1. The data is shared “in-place” from OneLake storage locations, meaning that no data is actually copied to the other tenant2.

Here’s a high-level overview of how to enable and use this feature:

Enable External Data Sharing: Go to the tenant settings in the consuming tenant and find the “Users can accept external data shares (preview)” setting under export and sharing settings. Enable the toggle and specify who in the tenant can create external data shares3.
Create and Configure Domains: Organize your data into domains and subdomains to manage governance and control access based on business units or departments4.
Assign Workspaces to Domains: Assign workspaces to the relevant domains or subdomains in the admin portal5.
Share Data with External Users: Use the external data sharing feature to share the necessary datasets with users in another Fabric tenant1.
This approach allows for decentralized data ownership and governance, enabling each business unit to define its own rules and restrictions according to its specific business needs45. It’s a powerful way to collaborate across different Fabric tenants while maintaining control over your data. For a detailed guide, you can refer to the resources provided by Microsoft Fabric45213.