Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
darrmus
New Member

Publishing Large Dataset of Parquet files in Azure Storage Gen 2

Hi,

I need to create a report that queries a folder containing above 300 parquet files with same schema. 1for file per day is added. Data resides in a Gen 2 Azure storage account. The total size is 4GB (equivalent to 20GB if CSV).

I would need to add a filter for serial number and get all data (all coumns required) related to this serial number. 

The problem is as the size is too large, I am getting Load exception in Power Bi desktop and cannot publish the whole dataset. The idea is to cache the dataset in the cloud say every day and users query this dataset, hopefully with a fast response.

 

Can you please guide me the best approach? I was looking for direct querying, but haven't found the connector in Parquet / CSV files

 

Tks!

1 REPLY 1
lbendlin
Super User
Super User

Supposedly the new DirectLake stuff should allow you to query Parquet files at rest.

 

Until then you can consider loading them into a dataset (with incremental refresh if needed), and then point your report to that dataset.

Helpful resources

Announcements
July 2024 Power BI Update

Power BI Monthly Update - July 2024

Check out the July 2024 Power BI update to learn about new features.

July Newsletter

Fabric Community Update - July 2024

Find out what's new and trending in the Fabric Community.