Forum Discussion
New to BI. Writing SQL queries to parse data. I am obviously missing something.
1) No, that SQL statement box is actually for SQL statements as you would normally write, not for the Power Query language (it's called M. I'm just going to call it that now). It's just not generally recommended to use that feature at all. You don't even have to write M language formulas yourself. The buttons generate them. If you feel ambitious you can write them yourself but it's rarely necessary unless you're doing really crazy stuff.
2) You just connect to your database and pick a table. You can mess with the query in the query editor from there. That's where you would do things like pick specific columns or filter rows.
3) Yes, this is done by merging queries in the query editor.
4) Using the buttons does give you the exact data you need. The order of operations is simply different from what you're used to.
I second Anonymous's view recommendation. I wish I had that option in my current situation.
1) No, that SQL statement box is actually for SQL statements as you would normally write, not for the Power Query language (it's called M. I'm just going to call it that now). It's just not generally recommended to use that feature at all. You don't even have to write M language formulas yourself. The buttons generate them. If you feel ambitious you can write them yourself but it's rarely necessary unless you're doing really crazy stuff.
You are advisng against filtering the data via SQL before I load the data into Power BI? I should filter the data after it is in Power BI via M or the visual querying tools?
alwwebAdvocate II
Personally, as a SQL person first and foremost who is now working in PowerBI, I would have to say, I take the opposite approach and also advise others to take the opposite approach as well if performance is at all importand.
When you do all fo the filtering after you bring in the data, it has a much higher impact on the SQL server every time you refresh the data. Plus, it has to transport everything across the network just to throw it away.
My rules personally would be as follows:
For a one time data pull - not something that will be regularly refreshing OR if you can't write decent SQL code OR if you database is on a strong server and is in the MB or low GB size and not TeraBytes:
1. Connect to SQL as the source from Power BI
2. Take the easy way, click all fo the tables that you want.
3. Click Edit and make and changes that you want via M (either point and click or with programming)
4. Apply and Save
5. If you need - add additional calculated columns or measures via DAX.
6. Design away
If this is a dashboard or report that will be getting refreshed AND can write select statements with SQL
1. Open your favorite query writing tool. You can do it in PowerBI, but no Intellisense, no formatting, no easy debugging, etc.
2. Write the query and play around with it until you are happy.
3. In PowerBI, Use SQL as the source and paste the query that you copied from the other tool. This nice thing is, you can continually refresh and get new data without having to start from scratch.
4. If you need to make changes easier done in M than SQL, make them after importing the query.
5. Continue as above.
Just my thoughts. I feel sorry for the DBAs trying to maintain performance on their servers as we start hitting them with more and more UserBI apps that are pulling everything on earth just to throw it away after.
>You are advisng against filtering the data via SQL before I load the data into Power BI?
Yes I am, but not strongly. You can do it if you're more comfortable in SQL. However if you're going to use SQL I do recommend doing it the way sdjensen demonstrated. That's how I do it too, but only when necessary. Basically I only do it when I've determined that my own SQL code will run significantly more efficiently than what Power BI would run on its own. For me that's mostly cases of really weird seven-table joins on multi-column criteria. Otherwise I usually just click buttons and let it do its thing.
You can write your own M code if you feel ambitious. There are a few things that can only really be done by writing the code, but they're rare and weird. Any regular query tasks can be accomplished by clicking buttons in the editor. You'll have to learn a little bit of M code if you want to start adding custom columns, but it's easy.
Moving from SQL to Power BI's query builder can feel a bit weird because they look very different and the order of operations seems backward. In SQL you tell it to select some columns from a table. In Power Query you tell it to connect to a table and keep the columns you want. The two do the same thing but they're written (or generated via the visual tools) in the opposite order. But that's not really how it runs. In the editor it's just loading a preview for you to play with. As I understand it when the query actually runs it isn't pre-loading an unfiltered table and then filtering it. It actually translates back to a regular SQL select statement. The nice thing about M and the query builder is that you can mashup all sorts of sources, so you can do wacky things like a left outer join of your SQL table with a table off a Wikipedia article. The query is all in one language, but the SQL query still happens behind the scenes.
It's worth learning to do things the regular query builder way even if you're going to embed your own SQL in most cases. It can do things that SQL can't, and learning to use it properly will give you a better handle on what's going on in your Power BI models in general.
- ElizabethTachji
Helper III
Hi,
I would like to add here, in Power BI Desktop, query Editor, you have the option to disable the load query so the query is not refreshed. When you disable the load, you will not slow down the performance of the servers and will improve the performance of Power BI Desktop report.
Cheers
Elizabeth Tachjian
Analytics Realtime
This is exacly what we have been doing with one additional step which may not be applicable for every one.
When we want to refresh/schedule the data and if we are using the first approach to import all the tables, the performance of the SQL server will be impacted.
1. I have written a query to pull all the filtered data from multiple tables with huge number of rows.
2. Schedule a job to retreive all the information and enter into a different table in a different database.
3. Schedule a refresh with in PowerBI to get the updated data.
Reasons to do this: The only SQL server we have is being used by multiple SQL developers to run reports every day. When we try to extract the data with either of these approaches, it impacts the performance of the SQL server. By doing this, the data will be retreived overnight and copied into a different database. We use this database to connect through PowerBI and providing datasets to the Finance team. It has been working so far. I am not saying this solution will work for everyone, but in order to avoid the perf, issues, we thought this might be the best way.
I would be happy to hear if anyone has any other ideas to make it more efficient. Thanks for reading!
