Forum Discussion
Power BI Service (not Desktop) - data source from Kobo Toolbox
Hey dadex78 ,
when using Power Query to fetch data from a given data source, the definition of the query that collects the data is not dynamic. This means that each time the Power Query is exceuted the same query is fetching data, assuming that the amount of data (number of rows) is growing this leads to a growing execution time.
Most of the time this is not a problem, but sometimes it is, especially when there is a real time requirement. This can be solved by using incremental refresh (https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview?WT.mc_id=DP-MVP-5003068). Basically incremental refresh helps to add increments to an existing dataset instead of reading all the available data from the data source all the time. Incremental refresh is simple to configure, as it assumes that the underlying data source is a relational database like MSFT SQL Server, Oracle, or ... If the data soure is not supporting query folding (https://learn.microsoft.com/en-us/power-query/power-query-folding?WT.mc_id=DP-MVP-5003068) then implementing incremental refresh becomes harder.
One of the most challenging aspecs of incremental refresh (at least from my perspective) is the initial load, as the Power BI Service imposes a limit to the query duration. A Power Query can not exceed 5 hours on Power BI Premium, on PRO based licensing it's 2 hours. If the amount of available data (the data that has to be loaded before "incremental refresh" kicks in) is huge, incremental refresh become even harder.Not sure about this "... create similar visualizations independently ..."
Assuming that this means visualizations will be created by business users and not by IT professionals, then the concept of Power BI Shared Datasets is the thing you are looking for (https://learn.microsoft.com/en-us/power-bi/connect-data/service-datasets-share?WT.mc_id=DP-MVP-5003068)
Hopefully, this helps to get things clarified.
Super UserHey dadex78 ,
you have to share your "visualizations" using Power BI Apps.
Nevertheless, web data sources do not support directQuery, meaning it's outside of Power BI to get data in real time from Kobo to Power BI.
When you are using Power BI Prmium you can consider to create a solution leveraging Power BI automate that refreshes the dataset every x-minutes. But this does not mean, it's incremental, by default each refresh reads all available data.
If Kobo does not yell into the wild "Hey, there is new data" and no one is listening, then real time becomes tricky.
Hopefully, this helps somehow to tackle your challenge.
Regards,
Tom
Dear Tom,
Thank you for reaching out; your suggestions are incredibly valuable.
Power BI Automate sounds like a great option. Having the data refresh every x-minutes aligns well with our requirements. However, I am not entirely clear about the issue related to reading all the available data. Could you please provide some clarification on this matter?
I must admit that I am relatively new to Power BI. While I plan to hire an expert to configure the system, I would also like to gain an understanding of the available options and the system's complexity. My goal is to ensure that, in the future, our organization can create similar visualizations independently.
Once again, thank you for sharing your insights; they are invaluable.
Best Regards,
Davide