Big Data Ingest with Onprem Data Gateway and Fabric

[rayfelipe]

Is there a known limitation to Onpremise Data Gateway (ODG) and Fabric data ingest? My customer requires big-data data ingest into Data Lake Storage using Copy Data in Fabric. Ingesting 125 Million rows of data (12 columns) takes about 40 mins. Which is ok for them. However, they also require downloading historical data, the rest of where the 125M rows come from. This is about 1 Billion rows (utility meter data). When they tried ingesting it, it took about 7 days without completion because the pipeline time out. According to the customer, Fabric ingest with Copy Data shows 15mbps download speed.

Question:

- Any known limitation for ODG given this data size into Fabric? Given the 125M rows in 40 mins. 1B rows should take 40 (mins) x 8, which is 320 mins., which is about 5 hours. But it takes days for them while eventually timing out.

[v-ssriganesh]

Hello @rayfelipe,

Hope everything’s going great on your end. Just checking in has the issue been resolved or are you still running into problems? Sharing an update can really help others facing the same thing.

Thank you.

[v-ssriganesh]

Hello @rayfelipe, 

hope you're doing well. Is the issue resolved or still ongoing? An update would help others in the community facing similar problems.

Thank you.

[v-ssriganesh]

Hello @rayfelipe,

We hope you're doing well. Could you please confirm whether your issue has been resolved or if you're still facing challenges? Your update will be valuable to the community and may assist others with similar concerns.

Thank you.

[v-ssriganesh · Accepted solution]

Hello @rayfelipe,
Thank you for reaching out to the Microsoft Fabric Forum Community. Please consider the below points:

• Increase network bandwidth to reduce transfer time.
• Upgrade the ODG machine’s hardware and consider gateway clustering.
• Break the ingest into smaller batches to avoid timeouts.
• Use non-OAuth2 authentication to bypass refresh limits.
• Optimize the source database query and consider incremental loading for historical data.

Please refer below links:

• https://community.fabric.microsoft.com/t5/Desktop/Gateway-limitations/m-p/234153
• https://learn.microsoft.com/en-us/power-bi/connect-data/service-gateway-onprem-tshoot
• https://learn.microsoft.com/en-us/data-integration/gateway/service-gateway-performance

If issue persist, please share details about the ODG machine specs, the data source type and any specific error messages from the pipeline logs.

Regards,
Ganesh Singamshetty.