Data Load Times with Teradata

[DanBurgon]

Hi,

We are storing our reporting datasets in Teradata (in the cloud) and then loading them into Power BI, but am finding the load speed is very slow; this isn't too much of a problem for our more aggregated datasets but it is becoming a problem for granular datasets that we would like to work with. As an example, we recently tried to load a transaction (ish) level table of ~150m rows: the load ran for 12 hours, getting to about 120m rows (about 3k rows per second), and then our VPN dropped out. We're finding this speeds up when we load to the service, but not enough for it not to be an issue.

As a comparison, we have been experienting with Amazon Redshift as a data store and the load rates for that on similar data sets is of the order or 100k rows per second. However, Redshift isn't going to be a viable choice for us at the moment, so we're sticking with Teradata. 

My question then is, what is the issue with Teradata? Is it that Power BI just doesn't work well with it, or should we be looking at something else like the network?

Any pointers would be greatly appreciated!

[DanBurgon]

Thanks for your response @lbendlin 

• In the particular example above we didn't try and load it to the service for that reason 🙂
• Incremental for some examples of what we do would work, but for others we need to refresh the the whole dataset as historical records can change.
• I haven't heard of partition bootstrapping - do you have a link to more information?
• Regardless of what we can do with incremental, partion boot strapping etc. should Teradata be that slow? Should we be looking at gateways, network, a new data platform etc? Or do we need to just live with it?

[lbendlin]

Interesting. Your load should have conked out at 300 minutes - that is supposedly a hard limit in the Power BI service.

Have you considered using  incremental refresh and partition bootstrapping?