Forum Discussion
Postgres DirectQuery "Requires Data Gateway"
- 6 years ago
Yes Anonymous - if you have data that requires the gateway, like the CSV files, then it can make the issue more complex.
Can you get it working if you remove everything but the Postgre data? Just whack all of that other stuff and save as a new file. Then add data sources back until you hit the one causing the issue. You didn't say where your CSVs were. A gateway is needed for local/network files or SharePoint On Prem. Should not be needed for SharePoint online, Data Lake, and other online resources.
But mixing and matching can still cause issues that are frustrating to trouble shoot. That is why I suggest do 1 data source, then add until it breaks and fix the breaks one at a time.
Given the error you have, something in your report thinks it needs a gateway to access the "data source file"
Query folding is the process Power Query uses to create a SQL statement and sends it to the server. So as an example, I imported the Purchasing table from the Microsoft World Wide Importders sample database on SQL Server using basic transformations in Power Query - removing columns, adding a conditional column. the M code is:
let
Source = Sql.Databases("localhost"),
WideWorldImportersDW = Source{[Name="WideWorldImportersDW"]}[Data],
Dimension_Customer = WideWorldImportersDW{[Schema="Dimension",Item="Customer"]}[Data],
#"Removed Other Columns" = Table.SelectColumns(Dimension_Customer,{"Customer Key", "WWI Customer ID", "Customer", "Bill To Customer", "Category", "Buying Group", "Primary Contact", "Postal Code", "Valid From", "Valid To"}),
#"Added Purchasing Power" = Table.AddColumn(#"Removed Other Columns", "Purchasing Power", each if [Buying Group] = "N/A" then "Standard"
else if [Buying Group] = "Tailspin Toys" then "Standard"
else "Discount", type text)
in
#"Added Purchasing Power"
Power Query though "folds" all of that. It doesn't acutally do any of that in the PQ mashup engine. It analyzes it, and send the following SQL statement to the server:
select [_].[Customer Key] as [Customer Key],
[_].[WWI Customer ID] as [WWI Customer ID],
[_].[Customer] as [Customer],
[_].[Bill To Customer] as [Bill To Customer],
[_].[Category] as [Category],
[_].[Buying Group] as [Buying Group],
[_].[Primary Contact] as [Primary Contact],
[_].[Postal Code] as [Postal Code],
[_].[Valid From] as [Valid From],
[_].[Valid To] as [Valid To],
case
when [_].[Buying Group] = 'N/A'
then 'Standard'
when [_].[Buying Group] = 'Tailspin Toys'
then 'Standard'
else 'Discount'
end as [Purchasing Power]
from
(
select [Customer Key],
[WWI Customer ID],
[Customer],
[Bill To Customer],
[Category],
[Buying Group],
[Primary Contact],
[Postal Code],
[Valid From],
[Valid To]
from [Dimension].[Customer] as [$Table]
) as [_]
The server does all of the work and simply sends back the relevant data. You can do pivots, grouping, merges, appends, etc. Tons of stuff will fold.
For Direct Query to work, Query Folding must be allowed to happen. Direct Query sends SQL statements to the server and returns immediately those results. There is no other work done or allowed in the Power Query mashup engine. SQL to SQL statements, that is it. As soon as you "break folding" Direct Query breaks. Import Queryies will still work. They will do folding to the point the folding breaks, then do the rest internally in the gateway/mashup engine. Direct Query though doesn't allow it. 100% folding, or you have to switch to Import for that query.
I don't have access to a Postgre database to test with, but I would assume in the M code I have posted above for SQL Server, the PostGre connection would have its own code, and you'd need to modify it to enable query folding per that article link. I'm not sure why - SQL Server it just works - there is no flag to set. But there must be something with Postgre that it need to be able to be turned off and on as needed.
And you would need to add it to each query to your server. It is not a global setting but a per-query setting.
Thank you for the thorough answer and clarification edhans!
I got the native query working last night in desktop, enabled the query_folding flag, and it returns exactly what I want.
I checked the native query step and it doesn't appear anything like yours does; it's just a single in-line SQL statement.
However, checking in on the webapp this morning I still see the same gateway error.
What's even more confusing is that a coworker created a small status-view report of some aggregated database values with DirectQuery and his just worked. And he's touched Power BI a total of 2 days. This is why I'd rather not go the data gateway route yet (also because it's another level of hassle).
A difference between my report from his is that I have some csv fact tables related to the direct queries. Would Mixed-Storage mode/Composite-Modeling be the problem?
- edhans6 years agoCommunity Champion
Yes Anonymous - if you have data that requires the gateway, like the CSV files, then it can make the issue more complex.
Can you get it working if you remove everything but the Postgre data? Just whack all of that other stuff and save as a new file. Then add data sources back until you hit the one causing the issue. You didn't say where your CSVs were. A gateway is needed for local/network files or SharePoint On Prem. Should not be needed for SharePoint online, Data Lake, and other online resources.
But mixing and matching can still cause issues that are frustrating to trouble shoot. That is why I suggest do 1 data source, then add until it breaks and fix the breaks one at a time.
Given the error you have, something in your report thinks it needs a gateway to access the "data source file"- Anonymous6 years agoNot applicable
edhans The CSV's are stored in OneDrive, but I set them to import mode (I imported them as CSV's, not as a OneDrive source). So that's definitely the issue then.
I stripped everything back and it's now working! (I'm sad that I had to remove so much though 😞 )
Fortunately it all lives in the server, so I can get it by DQ alone. Are calculated tables allowed as well, though? I had one, but noticed its storage mode was set to import.
Thank you so much for your help!
- edhans6 years agoCommunity Champion
You can have calculated tables in Direct Query. You can even have import tables in DQ. But those will only refresh on the scheduled refresh, not live like the DQ queries will. That would be composite mode.
You can get your OneDrive CSVs as cloud by connecting using SharePoint Folder connector. Get to your OneDrive via the O365 portal page. You'll get a URL like:
https://tenant-my.sharepoint.com/personal/user_name_tenantname_com/_layouts/15/onedrive.aspxJust use the
https://tenant-my.sharepoint.com/personal/user_name_tenantname_com/part of the URL.