Enhanced Raw Capacity Metrics Extraction for In-depth Analytics and Reporting

[joshua_wallace]

There's an emerging need for comprehensive raw usage data extraction within Microsoft Fabric to support detailed trending analysis, feature usage tracking, and accurate chargeback calculations based on both item counts and detailed resource consumption (CU/Memory usage). The current challenge lies in the limited data retention policy of Microsoft's Metrics App, which only preserves data for the last 14 days and presents it in a pre-aggregated format, complicating historical analysis and long-term storage for future chargebacks, which may occur six months or more post-event.

Our current workaround involves deploying custom Python scripts to interact with Power BI APIs, executing DAX queries to extract the necessary data. This process is cumbersome, especially when considering access limitations related to the account permissions associated with the Metrics App setup. The need to manage multiple Apps/Workspaces to gather comprehensive capacity data, followed by a complex process of merging this data into Parquet files for storage in ADLS Gen2, further complicates the workflow. Ultimately, this data feeds into a Fabric Workspace for analysis, an unnecessarily convoluted process that can be streamlined.

We propose the development of a ReadOnly Admin API designed specifically for the extraction of raw capacity metrics. This API would include optional filters for date, Capacity Id, WorkspaceId, UserId, or ItemId, facilitating tailored data retrieval. Such an API would vastly simplify the process of data extraction, storage, and analysis, allowing for more efficient and accurate reporting and chargeback processes without the need for intricate workarounds.

[oaladekomo]

Thanks for providing the detailed suggestion on how we can improve Capacity Metrics. We are working on a solution for customers that will let you subscribe to the same unsampled data that is used by Capacity Metrics and retain it as needed for your business needs using the Fabric Real-time Intelligence feature. In addition to simplifying the consumption model, this approach will also improve latency of data allowing you near-real-time access to the same data used by the capacity platform. We look forward to hearing your feedback on the feature as we land it and adapt it based on customer feedback.