Forum Discussion

ScottPowellECU's avatar
ScottPowellECU
Icon for Advocate II rankAdvocate II
3 years ago

Copy data for on-prem SQL server

Hi, does anyone have an idea of when the "Copy data" activity might work for on-prem data sources (in our case SQL Server)? I know we can use Gen2 dataflows for this - but they are very slow. Really ...
Need Help
pipelines
Scott_Powell's avatar
Scott_Powell
Icon for Helper V rankHelper V
3 years ago

Anonymous - curious how do you get your data into the datamart performantly? I thought those used dataflows, which is still a huge performance issue for us. I need to move just over a Terabyte nightly. Understand once it's in a datamart it might be fast, but not sure how to get the data in there quickly to start with.

 

Thanks!

Scott

 

Thanks,

Scott

Anonymous's avatar
Anonymous
Not applicable
3 years ago

Hi - We use dataflows to load to datamarts  - they are reasonably fast - do have a limit of 100 GB though.

 

Do you need to load all of that terrabyte of data each day ? - could you not just do a delta load to the datamart ?