Forum Discussion
DM_GWPipeline_Gateway_CancellationOrTimeoutError
My latest partition (last month) have data about 5 million. Starting of the month its about 100K but soon it hits 5-6 million
Super UserHi CSun
It appears that it is getting a timeout error because it is taking too long to pull all the data from the On-Premise source to Azure.
What I would suggest is if it is an Azure Tabular Model is to set it up with an incremental load. So that you are not pulling in all the data each and every time. Which then will reduce the amount of data that you have to transfer to Azure.
Also if you can ensure that you have the fastest connection with the fewest hops to where you are storing your data in Azure. If possible to even ensure that you are using the closest Azure data center.
I have this issue, even when I process a partition with about 5 millin rows. We are in the process of migrating to Azure. in the mean time, is there something i can do?
- GilbertQ
What I could suggest is to partition your data, so that you are only updating the latest data?
In doing it in this way you can also process your data one partition at a time, which will then mean you can process the data into specific chunks.
My latest partition (last month) have data about 5 million. Starting of the month its about 100K but soon it hits 5-6 million