Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

The ultimate Microsoft Fabric, Power BI, Azure AI & SQL learning event! Join us in Las Vegas from March 26-28, 2024. Use code MSCUST for a $100 discount. Register Now

Reply
Alirezam
Helper V
Helper V

Keeping large data in power bi or csv file

Hi, 

I am coming across an issue of very slow refreshing and data transformation. Here is my problem:

- I have a large table (about 4 million rows for the last 3 years) which will be appended to the dynamic table (growing each year). I have two options:

 1. Exporting the old large data to CSV and keep it in on-premise and then refreshing the power bi by reading that every time.

2. keeping this big file in Power Bi and append it to the new data.

 

Which way is more efficient?

I appreciate your time

Alireza

1 ACCEPTED SOLUTION
mahoneypat
Employee
Employee

For #2, you will need to refresh your data to append it anyway.  You could use UNION and a DAX table to avoid the refresh, but I don't recommend that as it will bloat your file size unnecessarily (and it is a bad practice).  Do you have the option to use incremental refresh?  If so, I would do that.  If not, I would store your csv on SharePoint or OneDrive to refresh without need of a gateway.

 

Regards,

Pat





Did I answer your question? Mark my post as a solution! Kudos are also appreciated!

To learn more about Power BI, follow me on Twitter or subscribe on YouTube.


@mahoneypa HoosierBI on YouTube


View solution in original post

1 REPLY 1
mahoneypat
Employee
Employee

For #2, you will need to refresh your data to append it anyway.  You could use UNION and a DAX table to avoid the refresh, but I don't recommend that as it will bloat your file size unnecessarily (and it is a bad practice).  Do you have the option to use incremental refresh?  If so, I would do that.  If not, I would store your csv on SharePoint or OneDrive to refresh without need of a gateway.

 

Regards,

Pat





Did I answer your question? Mark my post as a solution! Kudos are also appreciated!

To learn more about Power BI, follow me on Twitter or subscribe on YouTube.


@mahoneypa HoosierBI on YouTube


Helpful resources

Announcements
Fabric Community Conference

Microsoft Fabric Community Conference

Join us at our first-ever Microsoft Fabric Community Conference, March 26-28, 2024 in Las Vegas with 100+ sessions by community experts and Microsoft engineering.

February 2024 Update Carousel

Power BI Monthly Update - February 2024

Check out the February 2024 Power BI update to learn about new features.

Fabric Career Hub

Microsoft Fabric Career Hub

Explore career paths and learn resources in Fabric.

Fabric Partner Community

Microsoft Fabric Partner Community

Engage with the Fabric engineering team, hear of product updates, business opportunities, and resources in the Fabric Partner Community.