Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
jeanrozin
New Member

BIG DATA load time

Hello friends.

I have the following situation:

I imported some .XML files into PowerQuery and transformed the data into the layout I need. I did this procedure with around a thousand files.

However, I have a folder with 70GB of .XML files that I need to import, transform and cross-reference with another table.

As you may have already noticed, this is a very high volume of files.

Yesterday I did a test with around 1 million files and after 10 hours of loading, the procedure still hadn't finished.

How can I work with this volume of data? I don't understand anything about MySQL or PostgreSQL.

1 ACCEPTED SOLUTION
Anonymous
Not applicable

Hi @jeanrozin ,

Please try to filter and aggregate the data before loading it into Power Query, This will help you reduce the size of the dataset and improve performance.

 

Best regards,
Community Support Team_Binbin Yu
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

View solution in original post

1 REPLY 1
Anonymous
Not applicable

Hi @jeanrozin ,

Please try to filter and aggregate the data before loading it into Power Query, This will help you reduce the size of the dataset and improve performance.

 

Best regards,
Community Support Team_Binbin Yu
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Helpful resources

Announcements
Europe Fabric Conference

Europe’s largest Microsoft Fabric Community Conference

Join the community in Stockholm for expert Microsoft Fabric learning including a very exciting keynote from Arun Ulag, Corporate Vice President, Azure Data.

AugPowerBI_Carousel

Power BI Monthly Update - August 2024

Check out the August 2024 Power BI update to learn about new features.

August Carousel

Fabric Community Update - August 2024

Find out what's new and trending in the Fabric Community.