Fabric Capacity Estimation

[sun-sboyanapall]

We are planning on migrating from DOMO to Fabric F64 SKU, we have around 100 - 200 datasets ranging from 1k rows to 1mil rows at max

one of the sample dataset is consuming around 1000CUs each time it is running, and with 50 ETL's I would expect these to consume 5000CUs per ETL, frequency of updates we are thinking every 1hr for 50% and some will run daily and some will run weekly. 

with around 100 users using our BI reports. is F64 viable in our case. Am I missing anything?

Thanks,

[v-karpurapud]

Hello @sun-sboyanapall 
Hope you are doing well!

I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.

Thank you.

[v-karpurapud]

Hi @sun-sboyanapall 

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.

[v-karpurapud]

Hi @sun-sboyanapall 

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[v-karpurapud]

Hi @sun-sboyanapall 

Thank you for reaching out to the Microsoft Fabric Community Forum.

I appreciate the insights shared by both @FabianSchut  and @nilendraFabric . In addition to that , here are few points to optimize your workload and capacity.

Optimizing ETL Execution:

• If all ETLs are scheduled to run at fixed intervals, try staggering the schedules to distribute CU consumption more evenly over time. Running multiple ETLs simultaneously can cause peak load congestion.
• Identify ETLs that can be batched or refreshed incrementally instead of performing full dataset refreshes, which will help reduce CU consumption.
• While the average CU consumption is well within the F64 capacity, ensure that peak loads (e.g., multiple ETLs running at the same time) do not exceed the capacity.

Additionally, refer to the documents below for guidance on effectively planning, optimizing, and managing your Microsoft Fabric capacity and ETL processes.

• Plan your capacity size - Microsoft Fabric | Microsoft Learn
• Capacity and SKUs in Power BI embedded analytics - Power BI | Microsoft Learn

By following these steps, you can confidently proceed with the F64 SKU while ensuring scalability and cost efficiency.

If my response has resolved your query, please mark it as the Accepted Solution to help others. Additionally, I would appreciate a 'Kudos' if you found my response helpful.

Thank you!
 

[nilendraFabric · Accepted solution]

Hello @sun-sboyanapall 

@FabianSchut  shared a great guidance. 
let me put my 2 cents here as well

Workload Estimation
Your workload involves:
• ETLs: 50 ETLs with varying frequencies:
• 50% (25 ETLs) run hourly.
• 30% (15 ETLs) run daily.
• 20% (10 ETLs) run weekly.
• CU Consumption per ETL: Each ETL consumes approximately 1000 CUs per run.
From the calculations:
• Hourly CU Consumption: CUs.
• Daily CU Consumption: CUs.
• Weekly CU Consumption: CUs. Averaged per day: CUs.
The total daily CU consumption:

25000*24 +360000+ 240000=1,200,000 cu

Monthly consumption:

12000000 * 30 =360,000,000 cU/month

F64 Capacity
The F64 SKU provides:
• Capacity Units (CUs):

64*60*60*24=5,529,600 cu

Monthly capacity 

5529600* 30 =165,888,000 cu

Your estimated workload (36M CUs/month) is well within the F64 capacity (165.9M CUs/month). This leaves significant headroom for additional workloads or unexpected spikes.

User Load and BI Reports
With around 100 users accessing BI reports:
• The F64 SKU supports up to 50 concurrent DirectQuery connections and a maximum of 30 live connections per second, which should be sufficient for your user base.
• The maximum memory per query is 10 GB, and the total memory for semantic models is 25 GB, which aligns with typical BI workloads.

Cost Optimization: Reserved pricing for F64 is recommended if the capacity will be utilized consistently.
• Feature Requirements: F64 supports advanced features like AI Skills and Copilot, which may be beneficial depending on your use case.

Thanks

[FabianSchut]

Your potential F64 SKU has 64 (total number of CU) * 3600 (seconds in an hour) = 230.400 CU second available. If you are running all your 50 ETL's every hour (max consumption), you would consume 50 * 1.000 = 50.000 CU second every hour. That means that you have 230.400 - 50.000 = 180.400 CU second left every hour to run all other tasks in Fabric, such as hosting the Power BI report. It depends on the sizes of your Power BI reports, but averagely taken, that should be enough. I think the Fabric F64 SKU is indeed viable for your scenario. Of course this is based on assumptions, like the sample dataset is an average of all ETL tasks. However, you have enough slack to safely say that it will be viable.