Dataset - storage

[Anonymous]

Hello,
1. i am using the pro licence powerbi user and i am creating the dataset to load the datas. for a specific table i have more than 7crore records and it will be increase 10,000 per day. so in the data set when i transform data then i could not be able to load the table records. i am getting error. so i could not be able to publish and create the report with that table.
2. how to create the incremental refresh in the schedule. (per day one schedule planned)? as because while doing the refresh all 7crore 10,000 - rcecords will not syn and only the changed records will be sync while doing the refresh?

[Anonymous · Accepted solution]

Hi, @Anonymous 

When working with a large dataset of more than 70 million records like yours, you may run into issues due to the size limitations of Power BI Pro. The maximum dataset size for Power BI Pro is 1 GB. Make sure your data model is optimized. Remove unnecessary columns and rows, and consider summarizing data where possible.
Dataflows can help you preprocess and transform your data before loading it into Power BI.
If your dataset exceeds the limits of Pro, consider upgrading to Power BI Premium, which supports larger datasets and provides additional functionality.

You can also use incremental refresh, as rajendraongole1 said. You can check the following links:

Configure incremental refresh and real-time data for Power BI semantic models - Power BI | Microsoft Learn

Troubleshoot Power BI Desktop publishing - Power BI | Microsoft Learn

How to Get Your Question Answered Quickly

Best Regards

Yongkang Hua

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[rajendraongole1]

Hi @Anonymous - you can follow below suggested methods when you are working with huge datasets like 70 millions.

1.Use Query Folding: Ensure transformations are pushed back to the source (query folding) to reduce the load on Power BI.
2.Avoid operations like merging queries, adding custom columns, or applying transformations in Power BI that break query folding.
3.Filter the Data: Apply filters in the query editor to limit the initial dataset to a manageable size during development.
For instance, load only the past year of data during testing.

4.Use aggregations for summary-level data (e.g., monthly totals) and load only detailed records for the latest period.
This reduces the volume of data loaded into Power BI.
Increase Power BI's Resource Limits
5.Power BI Pro supports datasets up to 1 GB in size. If your dataset exceeds this:
Split the dataset into smaller partitions.
6.Upgrade to Power BI Premium if dataset size becomes a recurring limitation.
Switch to DirectQuery Mode
7.Use DirectQuery to connect directly to the source database instead of importing all the data. This approach ensures Power BI queries the data source only when needed.

8.Incremental refresh allows only new or changed data to be refreshed

By using incremental refresh, you'll minimize the load on your system and ensure only the necessary data is processed daily. Let me know if you'd like detailed guidance on any specific step!