Custom connector data preview for big data sets

[dgaiduchenko]

Hello

I'm developing custom data connector for Power bi. Dataset that connector operates with is quite and takes several minutes to load.

Currently when i execute my conector it fetches dataset twice : first time when it shows "Data preview" window and second time when i actually press Load button.

Is there a way to determine in connector if data is loaded for data preview or for actual data load ? In that case i could limit the dataset for data preview to a several rows.

Or maybe there is some other way exists to avoid loading whole dataset twice ?

Thanks

[Anonymous]

Probably too late to help the author, but for anyone looking for solution - I was struggling with the same problem, and the best way to solve it seems to be Query Folding. It allows you to pass limit parameter to the api request when powerbi query contains Table.FirstN() function (which it does for previews). 

I find this tutorial very useful: https://docs.microsoft.com/en-us/power-query/samples/trippin/10-tableview1/readme
Some docs: https://docs.microsoft.com/en-us/power-query/handlingqueryfolding

[v-jiascu-msft]

Hi @dgaiduchenko,

Can you mark the proper answer as a solution please?

Best Regards,

Dale

[v-jiascu-msft · Accepted solution]

Hi @dgaiduchenko,

That's two parts of Power BI. In the Query Editor, it's only a preview of data, which isn't the full data. We can shape data here. In the data pane, it's a full data set if we import the data. So we don't load the whole dataset twice. Since it's only a connector, I'm afraid we can't change the behaviour of Query Editor. By the way, we can filter the data source. But that will be applied to the whole dataset. Please refer to DataConnectors#what-you-can-do-with-a-data-connector.

>>>Exposing a limited/filtered view over your data source to improve usability

Best Regards,

Dale