Run Time solution for reading different days granularity.

[jaryszek]

I am using big size files parquet ones on fabric. 

And i have days as smallest granularity, months and years after. 

Assume that user want to compare 01.01.2015 with 01.01.2025 day. (only 2 days but time range is huge).  
How can i handle it within power bi ? Thus this is very big data even per 1 day , how to load data from parquets accordingly?

It will be handle by direct query or datalake mode and read on runtime? 

So user will chose 2 dates on slicer but data is not imported into semantind model with import mode (data is too big). How it will work on runtime? Or there are another ways ?

Can anybody explain a mechanism?

Best,
Jacek

[v-veshwara-msft]

Hi @jaryszek ,
Just checking in to see if your query has been resolved and if the responses from the @lbendlin and @DataNinja777 were helpful.

If you need any further assistance, please feel free to reach out.

[DataNinja777 · Accepted solution]

Hi @jaryszek ,

Yes, your scenario is handled very efficiently at runtime without importing the massive dataset. The solution uses Power BI's DirectLake mode, which relies on a powerful technique called partition pruning. This means Power BI will not scan 10 years of data; it will intelligently read only the data for the two specific days you select.

The mechanism works like this: when a user selects 01.01.2015 and 01.01.2025 in a slicer, Power BI generates a query with a filter for those two dates. The Microsoft Fabric engine receives this query and, thanks to partition pruning, it understands that the data is organized into date-based folders. Instead of scanning every file, the engine intelligently navigates directly to the specific folders corresponding to those two days, completely ignoring the terabytes of data in all other folders. It's like knowing exactly which two folders in a massive filing cabinet to open, rather than searching through every single one.

This entire process hinges on one crucial prerequisite: your data must be correctly structured in the lakehouse. Your Parquet files should be saved using Hive-style partitioning, where the folder path itself contains the date information. A correct path for one of your files would look something like this:

YourLakehouse/Tables/YourTable/year=2015/month=1/day=1/data_file_01.parquet
You mentioned Direct Query, and it's important to know you'll be using something even better: DirectLake mode. This is the default for a Fabric Lakehouse and is much faster ⚡. While Direct Query translates your requests into SQL, DirectLake mode allows the Power BI engine to read the Parquet files directly from OneLake without any translation or intermediate steps. Think of it as reading a book in its original language (DirectLake) versus waiting for a translation (Direct Query).

In essence, as long as your Parquet files are partitioned by date in the Fabric Lakehouse, the system is perfectly designed to handle your request. The combination of proper data structure and DirectLake mode ensures that your reports will be fast and responsive, querying only the tiny slivers of data needed at runtime, no matter how large the total dataset is.

Best regards,

[jaryszek]

Wow, 

such an amazing answer! Fully explained, it is rare on this forum! 
Thank you very much,
Best,
Jacek

[lbendlin]

1. Use the filter pane to allow the report user to pick arbitrary dates

2. Use Visual Calculations to compare the selected values