Expand columns without reloading data

[mverwil]

I have a large dataset in PowerBI Service, that was fetched from a web API. It is separated in multiple partitions (one per month) and there is an incremental refresh setup once per day. It only refresh one day's worth of data. It contains about 2 years of data which took a long time to import.

I have a column which contains records that I have not expanded. Is there a way to expand those records without having to start over the whole data import? Is the data stored somewhere and I can modify the structure to make it appear or is it discarded?

Thanks for your help!

[Martin_D · Accepted solution]

Hi @mverwil ,

What is your expectation regarding the newly expanded columns and the past partitons that are already loaded? Do you plan to leave the newly expanded columns empty for past partitions and only load expanded columns starting today? Or do plan to fill these columns for past partitions as well, and if so, what is your strategy to extract the missing data from the source without reprocessing all past data?

If you want to update the past partions then you need to reprocess the past data anyway. And what makes this slow is probably not the loading of the data into the datamodel, but the extracting of the data from the API and the extracting of the values from the API responses (JSON?).

If you want to leave the old partions as they are, then, in Premium workspaces only, you can try deploying the model changes without processing the old partitons using the "Do Not Process" option Deploy Analysis Services tabular models from Visual Studio | Microsoft Learn But then for sure the old partions will contain no data for the newly expanded columns and since they are no longer consitent with the schema the will most likely require reprocessing later anyway.

If you want to speed up implementing your changes in Power BI Desktop then create a parameter to filer the data at a very early step in PowerQuery, then do the changes with only a fraction of the data, then publish, and then change the parameter in the web browser to refresh all data. Maybe you can even use your exisiting incremental refresh parameters in Power BI Desktop to achieve this.

Kind regards,

Martin

[mverwil]

This is what I feared. I had some hope that maybe the datamodel already contained the data and that I could expand the columns of the old partitions, but it seems I will indeed need to reprocess all the data.

That's on me for not thoroughly checking if all the needed fields were present and available.

Thanks for the help! 

[Idrissshatila]

Hello @mverwil ,

you must expand the column then re-publish it again to service.