Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
mverwil
Frequent Visitor

Expand columns without reloading data

I have a large dataset in PowerBI Service, that was fetched from a web API. It is separated in multiple partitions (one per month) and there is an incremental refresh setup once per day. It only refresh one day's worth of data. It contains about 2 years of data which took a long time to import.


I have a column which contains records that I have not expanded. Is there a way to expand those records without having to start over the whole data import? Is the data stored somewhere and I can modify the structure to make it appear or is it discarded?

Thanks for your help!

1 ACCEPTED SOLUTION
Martin_D
Super User
Super User

Hi @mverwil ,

 

What is your expectation regarding the newly expanded columns and the past partitons that are already loaded? Do you plan to leave the newly expanded columns empty for past partitions and only load expanded columns starting today? Or do plan to fill these columns for past partitions as well, and if so, what is your strategy to extract the missing data from the source without reprocessing all past data?

If you want to update the past partions then you need to reprocess the past data anyway. And what makes this slow is probably not the loading of the data into the datamodel, but the extracting of the data from the API and the extracting of the values from the API responses (JSON?).

If you want to leave the old partions as they are, then, in Premium workspaces only, you can try deploying the model changes without processing the old partitons using the "Do Not Process" option Deploy Analysis Services tabular models from Visual Studio | Microsoft Learn But then for sure the old partions will contain no data for the newly expanded columns and since they are no longer consitent with the schema the will most likely require reprocessing later anyway.

 

If you want to speed up implementing your changes in Power BI Desktop then create a parameter to filer the data at a very early step in PowerQuery, then do the changes with only a fraction of the data, then publish, and then change the parameter in the web browser to refresh all data. Maybe you can even use your exisiting incremental refresh parameters in Power BI Desktop to achieve this.

 

Kind regards,

Martin

View solution in original post

3 REPLIES 3
Martin_D
Super User
Super User

Hi @mverwil ,

 

What is your expectation regarding the newly expanded columns and the past partitons that are already loaded? Do you plan to leave the newly expanded columns empty for past partitions and only load expanded columns starting today? Or do plan to fill these columns for past partitions as well, and if so, what is your strategy to extract the missing data from the source without reprocessing all past data?

If you want to update the past partions then you need to reprocess the past data anyway. And what makes this slow is probably not the loading of the data into the datamodel, but the extracting of the data from the API and the extracting of the values from the API responses (JSON?).

If you want to leave the old partions as they are, then, in Premium workspaces only, you can try deploying the model changes without processing the old partitons using the "Do Not Process" option Deploy Analysis Services tabular models from Visual Studio | Microsoft Learn But then for sure the old partions will contain no data for the newly expanded columns and since they are no longer consitent with the schema the will most likely require reprocessing later anyway.

 

If you want to speed up implementing your changes in Power BI Desktop then create a parameter to filer the data at a very early step in PowerQuery, then do the changes with only a fraction of the data, then publish, and then change the parameter in the web browser to refresh all data. Maybe you can even use your exisiting incremental refresh parameters in Power BI Desktop to achieve this.

 

Kind regards,

Martin

This is what I feared. I had some hope that maybe the datamodel already contained the data and that I could expand the columns of the old partitions, but it seems I will indeed need to reprocess all the data.

That's on me for not thoroughly checking if all the needed fields were present and available.


Thanks for the help! 

Idrissshatila
Super User
Super User

Hello @mverwil ,

 

you must expand the column then re-publish it again to service.



Did I answer your question? Mark my post as a solution! Appreciate your Kudos
Follow me on LinkedIn linkedIn
Vote for my Community Mobile App Idea

Proud to be a Super User!




Helpful resources

Announcements
Europe Fabric Conference

Europe’s largest Microsoft Fabric Community Conference

Join the community in Stockholm for expert Microsoft Fabric learning including a very exciting keynote from Arun Ulag, Corporate Vice President, Azure Data.

Power BI Carousel June 2024

Power BI Monthly Update - June 2024

Check out the June 2024 Power BI update to learn about new features.

PBI_Carousel_NL_June

Fabric Community Update - June 2024

Get the latest Fabric updates from Build 2024, key Skills Challenge voucher deadlines, top blogs, forum posts, and product ideas.

Top Solution Authors