cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
fengel
New Member

Flatten a giant Json DB.

Hello there,

Before everything a big thanks, for any help received.
Well, I'm not a BI developer, I'm actually a C# game developer adventuring here.

I have an extensive data set, from one of the games I'm developing, registering some data about the gameplay for each match for each user.

With a small data set, I could flatten all the matches, as I needed, with the code:

WyBf0XCTmF.png

 my issues appear applying this to the real production data, which has more than 26k users (in this code, the total rows there, and then each user could have N number of matches recorded).

The query runs(is running) for more than 1hour, and I don't know the end of it.
Is there a way to optimize this first query, just flattening the records into one big table that I'll use later to generate insights ?

Thanks in advance

1 REPLY 1
otravers
Super User
Super User

There's probably a way to optimize your M code, but in your situation where you're in charge of the app, I'd flatten the JSON into one or more tables (depending on data structure) in a staging database which I'd use as the source for Power BI. Power Query is convenient but it's not fast. Also you won't be able to do incremental refresh with a file source so things will only get worse as your dataset grows over time.

------------------------------------------------
1. How to get your question answered quickly - good questions get good answers!
2. Learning how to fish > being spoon-fed without active thinking.
3. Please accept as a solution posts that resolve your questions.
------------------------------------------------
BI Blog: Datamarts | RLS/OLS | Dev Tools | Languages | Aggregations | XMLA/APIs | Field Parameters | Custom Visuals

Helpful resources

Announcements
PBI Sept Update Carousel

Power BI September 2023 Update

Take a look at the September 2023 Power BI update to learn more.

Learn Live

Learn Live: Event Series

Join Microsoft Reactor and learn from developers.

Dashboard in a day with date

Exclusive opportunity for Women!

Join us for a free, hands-on Microsoft workshop led by women trainers for women where you will learn how to build a Dashboard in a Day!

MPPC 2023 PBI Carousel

Power Platform Conference-Power BI and Fabric Sessions

Join us Oct 1 - 6 in Las Vegas for the Microsoft Power Platform Conference.

Top Solution Authors