Forum Discussion

ebjim's avatar
ebjim
Icon for Resolver I rankResolver I
2 years ago
Solved

Shortening sql server import times

I am looking for any suggestions to shorten the import time of one large table, given that in the near term, I'll be using a single node gateway and importing a binary file is not an option.   The ...
General Question
pipelines
Tips and Tricks
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi ebjim 
    If you are importing this into a semantic model,  you could look at incremental refresh, create partitions, review data types, only include columns needed (used for relationships, analysis, measures), verify if you have auto date/time turned off if not being used (this can potentially create very large hidden date time tables for each date time column they are importing), etc.

    Is this historical or incremental load or truncate and load every time? If this is incremental, copy data to files section as LH (sink) does not support merge operation today. A copy activity followed by notebook or Dataflow gen2 is an option.

    Hope this helps. Please let me know if you have any further queries.

dbrowne's avatar
dbrowne
Icon for Microsoft Employee rankMicrosoft Employee
2 years ago

It's impossible to suggest anything without more information.  Eg what are the gateway VM specs?  what is the CPU, Disk, and Network utilization percentage?  What what is the table DDL?  Is there a bottleneck on SQL Server?

  • ebjim's avatar
    ebjim
    Icon for Resolver I rankResolver I
    2 years ago

    We're using the latest Power BI Gateway version, but I don't have the resource stats available. No bottlenecks with the SQL Server as far as I know. I am more concerned with getting the parallelism and throughput optimization settings correct. As for DDL, I can send it to you separately if you are open to that.