Forum Discussion
Option to check Memory usage during refresh to finally decrease the Memory usage
Hi community,
until today, we had no solution, but finally found the problem. But as we have found a solution, I try to explain it and write it down, so if anyone has the same problem, must not search any longer - hopefully.
This problem is only related to when using the Azure Data Explorer (ADX) to read data from.
The ADX connector in Power BI has some issues (or is not meant to be used for >300.000 records/~25 columns), and when instead using the SQL endpoint of ADX, the memory problems on refreshing on the Power BI Service do no longer occur. The ADX connector seems not to be efficient when loading data and thus makes extensive use of the memory.
That said, when connecting to an ADX, it is recommended to:
- Click the down arrow on Get data from the ribbon and select More...
- In the Get Data dialog search for Azure SQL database
- Choose Azure SQL database and click Connect - (!! choose NOT Azure Data Explorer (Kusto))
- Enter the ADX endpoint uri WITHOUT "https://" into Server (e.g. "[your server].[location].kusto.windows.net")
- Choose Import or DirectQuery to your needs
- In the Credentials dialog, choose on the left Microsoft account. (If you have problems on this, you may have to cleanup in the PowerBI Desktop Options&Settings > Data source settings menu the corresponding data source settings)
- Sign in to your account
- Click Connect
The Navigator dialog shows you the databases and tables from which to choose yout data from
This process at least solved out problem on extensive memory usage (error on using >3GB on refresh), although the overall dataset on the service has a size of ~100MB.
Maybe this helps someone else.
Hi community,
until today, we had no solution, but finally found the problem. But as we have found a solution, I try to explain it and write it down, so if anyone has the same problem, must not search any longer - hopefully.
This problem is only related to when using the Azure Data Explorer (ADX) to read data from.
The ADX connector in Power BI has some issues (or is not meant to be used for >300.000 records/~25 columns), and when instead using the SQL endpoint of ADX, the memory problems on refreshing on the Power BI Service do no longer occur. The ADX connector seems not to be efficient when loading data and thus makes extensive use of the memory.
That said, when connecting to an ADX, it is recommended to:
- Click the down arrow on Get data from the ribbon and select More...
- In the Get Data dialog search for Azure SQL database
- Choose Azure SQL database and click Connect - (!! choose NOT Azure Data Explorer (Kusto))
- Enter the ADX endpoint uri WITHOUT "https://" into Server (e.g. "[your server].[location].kusto.windows.net")
- Choose Import or DirectQuery to your needs
- In the Credentials dialog, choose on the left Microsoft account. (If you have problems on this, you may have to cleanup in the PowerBI Desktop Options&Settings > Data source settings menu the corresponding data source settings)
- Sign in to your account
- Click Connect
The Navigator dialog shows you the databases and tables from which to choose yout data from
This process at least solved out problem on extensive memory usage (error on using >3GB on refresh), although the overall dataset on the service has a size of ~100MB.
Maybe this helps someone else.