Cost Optimization and Cross-Workspace connect for Power BI and ETL Workloads (Separate Capacities)

[virajjayant]

Hi

I want to understand the billing options and cost optimization considerations for workloads where we wish to “decouple” Power BI and Lake House / ETL to separate “capacities”.

From my understanding, I am planning to follow the below mentioned steps:

1. Have ETL / Spark workloads in one workspace A with Capacity A (say, capacity = F4, F8)
2. Have Power BI workloads in another workspace B with Capacity B (say, capacity = A1)
3. Foster a connection b/w workspace A and workspace B so that I can save costs as much as possible (with or without One-Lake Shortcuts)
1. Spark workloads are on-demand, used for limited period of time per day
2. Power BI workloads are on-demand as well 
4. Note - When the capacity is not used, it must be paused to save costs

Referring to the example shared in the following link Fabric Capacity and One Lake Consumption, I want to understand how I can:

1. Foster cross-workspace connectivities
2. Save costs as much as possible (leveraging pause-resume capabilities for Fabric)
3. What does it mean when it says - “storage is billed to the capacity” like the above example states “the data is still stored in Capacity1, the data stored is billed to Capacity1.”?

Also, need a clarification for the following line (from the above link) “As Capacity2 accesses the data, the compute or transaction cost for this read operation consumes CU from Capacity2. The storage continues to be billed to Capacity1.”

1. Will the storage billing be at the rate Fabric Capacity consumes CU per hour, or is it normal ADLS / blobs storage pricing per month (which is nominal)

Thanks in advance!

[danish33]

Certainly! Let's break down your questions and concerns regarding cost optimization and cross-workspace connectivity for Power BI and ETL workloads in Azure:

### Billing Options and Cost Optimization Considerations

1. **Separate Capacities for Workspaces**:
- You plan to have ETL/Spark workloads in Workspace A with Capacity A (e.g., F4, F8).
- Power BI workloads will be in Workspace B with Capacity B (e.g., A1).

2. **Connection Between Workspaces**:
- To optimize costs, you want to establish a connection between Workspace A and Workspace B. This is typically achieved through dataflows or datasets published from one workspace to another, leveraging One-Lake Shortcuts for efficient data transfer.

3. **Cost Optimization Strategies**:
- **Pause-Resume Capability**: Azure allows you to pause capacities (like Azure Synapse or Power BI Premium capacities) when they are not in use. This can significantly reduce costs, especially for workloads that are not 24/7 operational.
- **Storage Billing**: When data is stored in a capacity (e.g., Azure Synapse Analytics or dedicated Power BI capacities), the storage costs are billed to that capacity. This means you are billed based on the storage capacity tier you have provisioned (e.g., F4, F8, A1) rather than standard ADLS/Blob storage rates.

4. **Example Clarification**:
- The statement "the data stored is billed to Capacity1" means that the storage costs for data residing in Capacity1 (Workspace A in your case) will be charged based on the capacity's storage pricing model. This is separate from standard Azure storage (ADLS/Blob) pricing.
- When Capacity2 (Workspace B) accesses data stored in Capacity1, the compute or transaction costs (measured in Compute Units, CU) for read operations are consumed from Capacity2. However, the storage billing continues to be attributed to Capacity1.

5. **Storage Billing Rates**:
- The storage billing within Azure capacities (e.g., Azure Synapse Analytics, dedicated Power BI capacities) is specific to those services and not based on standard Azure Data Lake Storage (ADLS) or Blob Storage rates. Azure capacities often have a higher cost structure per unit of storage compared to raw ADLS/Blob storage due to additional features and capabilities provided by these services.

### Recommendations for Implementation:

- **Use Pause-Resume**: Implement automatic pause-resume schedules based on workload patterns to minimize costs during idle periods.
- **Efficient Data Transfer**: Utilize One-Lake Shortcuts or other data integration methods between workspaces to optimize data movement and reduce latency.
- **Monitor and Adjust**: Regularly monitor usage patterns and adjust capacity sizes (e.g., F4, F8, A1) based on actual workload demands to avoid over-provisioning.

By following these strategies, you can effectively manage and optimize costs while maintaining flexibility and performance for your Power BI and ETL workloads across separate capacities and workspaces in Azure.

[Anonymous]

Hi @virajjayant ,

Thanks for using Fabric Community.
The best place to get an answer for billing related queries is here.

Please feel free to ask your query in Microsoft Fabric Pricing portal. 
Allow some time, so team can check and provide a resolution.

In case if you got a resolution please do share with the community as it can be helpful to others .

Hope this is helpful. Do let me know in case of further queries.

[Anonymous]

Hello @virajjayant ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.

[virajjayant]

Hi @Anonymous,

As of now, I haven't found the suitable resolution - I will perhaps reach out to the Fabrics Sales / Support Team for pricing related clarifications.

Meanwhile, it'd be helpful if you can share your inputs (if any) related to the above prescribed cross-workspace architecture and its feasibility.

Thanks a ton!

[Anonymous]

Hi @virajjayant ,

The best course of action is to open a support ticket and have our support team take a closer look at it. Link 

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Hope this helps. Please let us know if you have any other queries.

[Anonymous]

Hi @virajjayant ,

We haven’t heard from you on the last response and was just checking back to got a chance to open a support ticket.
After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thanks