Forum Discussion
PowerBI Service Dataflow & Datamart & PowerBI Desktop
Hi zhanna27
I am glad you find my tips helpful.
Now, to your new issue. Not sure if I understand it correctly - I think what you are saying is that the dynamic data you receive in .csv file contains only data for the current day. But what happens to the previous files from yesterday, the day before, etc? Is the file replaced with a new file every day, so you don't keep the previous days in separate files? If you don't keep the files from the previous days, then you can't really achieve this in PowerBI as whenever you need to refresh the Dataflow, it will only use the files you currently have. e.g. can't get data which is no longer in the datasource. So, in this case, first, you need to sort out the files from previous days to be kept in the folder, and you only add a new file to the folder every day. This way, you keep all the 'dynamic' data files in a folder, so when the Dataflow is refreshed, it can read data from all files within the folder. Depending on how you get the new file every day, there are different ways to sort it out. In my case, I receive the new file via email everyday, then I have a simple PowerAutomate which gets the attachment from the email and saves it in a SharePoint folder for me (if the file name is the same everyday you can add a date at the end of the file name when saving to the folder). This way, the process doesn't involve any manual handling of the data as it is automatically done for me even if I am away. Once the file is uploaded then the scheduled Dataflow is refreshed and the report updated with the latest data. So, the key is that you need to keep all the files for the 'dynamic' data in a single folder, then every month or couple of months, you can move the files to the 'archived' data folder and then remember to refresh the archive data Dataflow to get it updated and the 'dynamic' data folder is empty again so your daily refreshed Dataflow refreshes nice and quick.
One thing you mentioned is that you have the data on your desktop? Although it may work, I would strongly recommend moving the data into SharePoint (I hope you can create your own site within your company or can be given access to any existing SharePoint). You would need to have two folders - one for the archived data files and one for the dynamic data files, so the Dataflows will load all files from these folders. Which brings me to the next point, how do you load the data from files? Is it just load from Excel or do you use load from folder (SharePoint Folder). The best method in your case is to use Load from SharePoint Folder where you need to provide link to the main SharePoint site then filter to the folder you want to load and when you see all files within the folder, the easiest is to create a simple function to extract data from each file and then combine into one single table. Also, I always use formula =Excel.Workbook([content]) in a new column to get a list of all elements within the workbook, then you can filter to the item which you want to extract e.g. Sheet1 or Table_name to only work with what you need and then apply a function to extract/transform the data for me before expanding the final output into a single table.
Dear Anonymous,
thank you for your quick reply. I appreciate your tips. Again that's a good solution of yours which you propose for dynamic data to be stored in a file in my sharepoint desktop folder. Indeed, I haven't done any storage of this updated data because I thought it could be appended directly to a dataflow dataset and then the updated dataflow dataset could be used by datamarts directly and could be saved in datamarts each day (then the next day, it would upload archived data would delete already present archived data with a "remove_duplicates" step and + would upload just a new date). I understood now that dataflow doesn't save documents but I thought datamarts did so. So my update function here below is supposed to update and append new data into an archived dataset each day but only in dataflows :
- Anonymous2 years agoNot applicable
Hi zhanna27 , ok, sorry I missed the part where you said in the original post that the dynamic data is loaded from an API rather than CSV files. So, is it an API call to a database or something else? Can you query more data through the API than just a single day? I can see in your function you are querying the data based on two dates, so can't you just query the data since the static data ends (mid-Dec 23)? If there is too much data for the API to handle, you could probably multiply the queries and load, e.g. a month at a time. Also, if the data is coming from a Database, can't you just use the DB (e.g. SQL connector in Dataflow, so then you can use normal SQL query to import all the data you need)? This would be probably the simplest solution as an SQL query should handle millions of rows of data, so you could probably do it in a single query.
I assume that from time to time data which you currently loading dynamically are dumped to the csv files (e.g. every year?) so then you would need to amend the dynamic data query date range.
As for the storage options, I don't think there is anything like that in PowerBI itself. I am not sure why specifically you are using Datamarts. In my experience, Dataflows work faster when loading data, and I don't find them very useful apart from one single feature, which gives you an SQL endpoint to query the data from Datamart.
You could use a database to store the data, but first you would still need to extract the data from your API. There are a couple of ways how you could automate this depending on what tools you have access to. One example might be to use PowerAutomate, which can run on a schedule every day and export the data from API and either upload it to a Database, Datamart, or SharePoint List or just save it as a .csv or .xlsx file where from you can load the data into your 'dynamic' dataflow. However, there would be some limits on the amount of data you can handle with PowerAutomate, so if you have millions of rows every day, this may not work. Another more advanced solution would be to use Azure DataFactory (if you have access to Azure). In the ADF, you can create a pipeline which will run every day, export the data from API and load it into, for example, the Azure SQL database.
- zhanna272 years agoFrequent Visitor
Hi Anonymous,
thank you for your suggestions and your feedback. I don't have any storage/database for my current data (archived/.csv + updated daily/API) except of my desktop which is connected to onedrive and sharepoint with my company account or temporary storage on PowerBi service. One person suggested me to save a static dataset into a semantic model of datamarts and update dailly this dataset with new data which will be appended and will not replace the earlier updated data. I don't know how to do that and I am not sure that datamarts can save a document and just append new information and save this information because each new day it would take the same old source and will add a new data. I was asked for the solution to have storage in a semantic model of powerBi service rather than saving on my computer, etc. I have read about powerautomate and it could work I suppose because I don't have a lot of data on a daily basis (7K rows) but if datamarts/semantic model can be refreshed each day and append the new data to a final dataset, then we don't need powerautomate or ADF pipeline. What do you think?