Hello everyone,I recently transitioned to using Microsoft Fabric products, specifically Notebooks (PySpark) and Lakehouse, running on F2 capacity. My current setup involves retrieving data from an API using Notebooks and storing it in a Lakehouse. While this workflow functions well, I’ve noticed significant costs due to high CU usage during Notebook execution. Here's a summary of my use case and the challenges I’m facing: I have two Notebooks that process data for two customers:Notebook for Customer 2:Frequency: Runs 5 times per day.Compute: Uses 8 driver cores, runtime ~2 minutes and 10 seconds per run.Data Processed:Rows: 3,310,767 rows processed per run.Data Read: 21,078 KB.Data Written: 800 KB.CU Usage (per run): 424 CU.Notebook for Customer 1:Frequency: Runs 1 time per day.Compute: Uses 8 driver cores, runtime ~2 minutes and 20 seconds per run.Data Processed:Rows: 1,888,114 rows processed per run.Data Read: 14,544 KB.Data Written: 1,952 KB.CU Usage (per run): 999.9555 CU.Additionally, storage costs are relatively low, but I’m wondering if using a Warehouse instead of a Lakehouse for storage would be more cost-efficient in the long term. Main points:High CU Usage in Notebooks:The Notebooks use 8 cores, but I’m unsure if the code is optimized. Each API call fetches data from multiple endpoints, and I suspect this could be contributing to the high CU usage. Could pipelines be a better alternative for this type of process? And does somebody know if batching reduces the CU'sStorage Costs: Lakehouse vs. Warehouse:I store the ingested data in a Lakehouse. Would moving this to a Warehouse result in better performance or cost savings, especially for Customer 2, which has frequent data queries?Capacity Management Issues:To save costs, I pause the capacity for several hours each day. However, after resuming the capacity, I observe massive CU usage spikes in the Fabric Metrics app—sometimes exceeding 32216%. Why is this happening, and does it negate the benefits of pausing the capacity? Here’s a summary of CU and runtime statistics across my Fabric workspace for reference:WorkspaceItem KindItem NameCU (s)Duration (s)UsersBilling TypeFabric workspaceSynapseNotebookNotebook Customer 22,117.9995529.49301BillableFabric workspaceSynapseNotebookNotebook Customer 1999.9555249.98501BillableFabric workspaceLakehouseLakehouse Customer 2872.79687.22101BillableFabric workspaceWarehouseWarehouse Customer 2742.17801,660.09602BillableFabric workspaceLakehouseLakehouse Customer 1405.68383.12301BillableFabric workspaceWarehouseWarehouse Customer 1214.3460340.08702BillableFabric workspacePipelinePipeline Customer 2100.8000587.72701BillableFabric workspacePipelinePipeline Customer 140.3200573.16801BillableTotal 5,494.07963,950.9000 Billable Thanks guys in advance

Hi CReportify Yes, it is possible to reduce the driver cores when executing a notebook in Microsoft Fabric. You can configure the driver cores using the `%%configure` magic command at the beginning of your notebook. Here’s an example:%%configure -f{"driverMemory": "56g","driverCores": 32,"executorMemory": "28g","executorCores": 4}Microsoft Fabric does not currently offer a built-in way to manually trigger or schedule specific background tasks like metadata processing during off-peak hours. The platform automatically manages certain background processes, and there’s no direct control over when these tasks execute after capacity is resumed. Microsoft Fabric supports autoscaling for Spark pools. When you set up a Spark pool, you can enable autoscaling by configuring a range for the number of nodes. The system will automatically add or remove nodes based on workload demand.I have tried to answer all your queries here. please accept this as solution if this resolved your query. thanks

Optimizing CU Usage in Microsoft Fabric | Microsoft Fabric Community

Forum Discussion

5 Replies

Related content