Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Grow your Fabric skills and prepare for the DP-600 certification exam by completing the latest Microsoft Fabric challenge.

Reply
mverwil
Frequent Visitor

Expand columns without reloading data

I have a large dataset in PowerBI Service, that was fetched from a web API. It is separated in multiple partitions (one per month) and there is an incremental refresh setup once per day. It only refresh one day's worth of data. It contains about 2 years of data which took a long time to import.


I have a column which contains records that I have not expanded. Is there a way to expand those records without having to start over the whole data import? Is the data stored somewhere and I can modify the structure to make it appear or is it discarded?

Thanks for your help!

1 ACCEPTED SOLUTION
Martin_D
Super User
Super User

Hi @mverwil ,

 

What is your expectation regarding the newly expanded columns and the past partitons that are already loaded? Do you plan to leave the newly expanded columns empty for past partitions and only load expanded columns starting today? Or do plan to fill these columns for past partitions as well, and if so, what is your strategy to extract the missing data from the source without reprocessing all past data?

If you want to update the past partions then you need to reprocess the past data anyway. And what makes this slow is probably not the loading of the data into the datamodel, but the extracting of the data from the API and the extracting of the values from the API responses (JSON?).

If you want to leave the old partions as they are, then, in Premium workspaces only, you can try deploying the model changes without processing the old partitons using the "Do Not Process" option Deploy Analysis Services tabular models from Visual Studio | Microsoft Learn But then for sure the old partions will contain no data for the newly expanded columns and since they are no longer consitent with the schema the will most likely require reprocessing later anyway.

 

If you want to speed up implementing your changes in Power BI Desktop then create a parameter to filer the data at a very early step in PowerQuery, then do the changes with only a fraction of the data, then publish, and then change the parameter in the web browser to refresh all data. Maybe you can even use your exisiting incremental refresh parameters in Power BI Desktop to achieve this.

 

Kind regards,

Martin

View solution in original post

3 REPLIES 3
Martin_D
Super User
Super User

Hi @mverwil ,

 

What is your expectation regarding the newly expanded columns and the past partitons that are already loaded? Do you plan to leave the newly expanded columns empty for past partitions and only load expanded columns starting today? Or do plan to fill these columns for past partitions as well, and if so, what is your strategy to extract the missing data from the source without reprocessing all past data?

If you want to update the past partions then you need to reprocess the past data anyway. And what makes this slow is probably not the loading of the data into the datamodel, but the extracting of the data from the API and the extracting of the values from the API responses (JSON?).

If you want to leave the old partions as they are, then, in Premium workspaces only, you can try deploying the model changes without processing the old partitons using the "Do Not Process" option Deploy Analysis Services tabular models from Visual Studio | Microsoft Learn But then for sure the old partions will contain no data for the newly expanded columns and since they are no longer consitent with the schema the will most likely require reprocessing later anyway.

 

If you want to speed up implementing your changes in Power BI Desktop then create a parameter to filer the data at a very early step in PowerQuery, then do the changes with only a fraction of the data, then publish, and then change the parameter in the web browser to refresh all data. Maybe you can even use your exisiting incremental refresh parameters in Power BI Desktop to achieve this.

 

Kind regards,

Martin

This is what I feared. I had some hope that maybe the datamodel already contained the data and that I could expand the columns of the old partitions, but it seems I will indeed need to reprocess all the data.

That's on me for not thoroughly checking if all the needed fields were present and available.


Thanks for the help! 

Idrissshatila
Super User
Super User

Hello @mverwil ,

 

you must expand the column then re-publish it again to service.



Did I answer your question? Mark my post as a solution! Appreciate your Kudos
Follow me on LinkedIn linkedIn
Vote for my Community Mobile App Idea

Proud to be a Super User!




Helpful resources

Announcements
RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.

MayPowerBICarousel

Power BI Monthly Update - May 2024

Check out the May 2024 Power BI update to learn about new features.

Top Solution Authors