Enhanced Raw Capacity Metrics Extraction for In-depth Analytics and Reporting

There's an emerging need for comprehensive raw usage data extraction within Microsoft Fabric to support detailed trending analysis, feature usage tracking, and accurate chargeback calculations based on both item counts and detailed resource consumption (CU/Memory usage). The current challenge lies in the limited data retention policy of Microsoft's Metrics App, which only preserves data for the last 14 days and presents it in a pre-aggregated format, complicating historical analysis and long-term storage for future chargebacks, which may occur six months or more post-event.

Our current workaround involves deploying custom Python scripts to interact with Power BI APIs, executing DAX queries to extract the necessary data. This process is cumbersome, especially when considering access limitations related to the account permissions associated with the Metrics App setup. The need to manage multiple Apps/Workspaces to gather comprehensive capacity data, followed by a complex process of merging this data into Parquet files for storage in ADLS Gen2, further complicates the workflow. Ultimately, this data feeds into a Fabric Workspace for analysis, an unnecessarily convoluted process that can be streamlined.

We propose the development of a ReadOnly Admin API designed specifically for the extraction of raw capacity metrics. This API would include optional filters for date, Capacity Id, WorkspaceId, UserId, or ItemId, facilitating tailored data retrieval. Such an API would vastly simplify the process of data extraction, storage, and analysis, allowing for more efficient and accurate reporting and chargeback processes without the need for intricate workarounds.