Forum Discussion

Argan77's avatar
Argan77
Frequent Visitor
3 years ago

Question On Data Load and Append (Refreshing data) from Teradata

I have a large table I'm trying to pull into Power BI Desktop from Teradata.  Unfortuantly, historical records can change, so I can't do any kind of an incramental download.  The data is coming from a Teradata server that is heavily used by others.  The result is that it takes hours to download 30 million records (4-7 hours depending on time of day).  I've already moved as much of the table into reference tables.  What's left is the unique key, dates, reference IDs and numeric fields.  

 

As a test, I broke up the large table into four smaller tables.  The result is that it simutanously pull in the four tables faster than the one large table (takes a little over an hour).  Since I need the end table to be merged, I created a new table using the Append feature (Append Quiries as new) in Power Query editior.  

 

My hope was that the end result would pull in the four tables then merge them.  It seems like it's downloading the four tables as expected and doing an additional download of each table into the merged table versus merging what was already downloaded resulting in the original issue (hours to download).  

 

  • Is there a way for it to not redownload the table for the merged table (use data already downloaded)?
  • Are there other methods that I can try to get around my problem?

 

Thanks!

Need Help

2 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    3 years ago

    Hi Argan77 ,

    Please check that the four smaller tables are unchecked to enable load.

    Similar thread:
    Solved: Append vs Append as new for performance

    Best Regards,
    Gao
    Community Support Team

     

    If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly. If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

    How to get your questions answered quickly -- How to provide sample data

    • Argan77's avatar
      Argan77
      Frequent Visitor
      3 years ago

      Thank you for your reply!  Oddly enough, it makes little to no difference.  This is why I tried splitting the table up in the first place.  Each table, regardless of how many other tables are being pulled at the same time from this server, refreshes at about the same speed.  The limitation seems to be per connection server side. 

      I've moved away from the append.  It appears like it's doing the union on the server side and then doing the download resulting in the original problem.

       

      I'm now playing with DAX Union(Table1,table2,table3,table4).  This seems to be doing what I want; however, I'm not sure how this will work performance wise once we start adding in measures and other calculations.  It's also increaseing the size of the file.  Is there way to purge the original tables after the union is complete?