Forum Discussion
Best Practices for Large Data Importing from SQL
You'll still likely want to load in your fact and dimension tables into your model separately as a star schema, rather than merging into a big monster table. One option when working with large tables is to use DirectQuery rather than Import to load your tables.
From your three listed options, #2 is probably the closest to best practice, assuming you don't have to do lots of data manipulation other than some filtering. If you do need lots of data manipulation, then you may want to do that in SQL first and have clean tables (or views) to load into Power BI.
You'll still likely want to load in your fact and dimension tables into your model separately as a star schema, rather than merging into a big monster table. One option when working with large tables is to use DirectQuery rather than Import to load your tables.
From your three listed options, #2 is probably the closest to best practice, assuming you don't have to do lots of data manipulation other than some filtering. If you do need lots of data manipulation, then you may want to do that in SQL first and have clean tables (or views) to load into Power BI.
Thanks for your response! There is a good amount of maniuplation to be done. I tried loading using a long SQL query and the update was awful. I also tried loading multiple tables and filtering best I could, and this still took some good time PRIOR to manipulation.
So if I just created a new table or view in SSMS based on the query I was trying to run, I could then use that to access in Power BI? This makes some sense since it only took about 6 minutes to run in SSMS compared to 4+ hours in Power BI (I canceled it).
Yes. Power Query is powerful but it's often better to push SQL manipulations upstream before trying to load the data into Power BI.
