Expose Last DeltaLake Table Update Timestamp in Refresh SQL Endpoint Metadata API

[frithjof_v]

Currently, the Refresh SQL Endpoint Metadata API (/refreshMetadata) returns information such as lastSuccessfulSyncDateTime, status, and startDateTime / endDateTime for each table. However, it does not expose when the underlying DeltaLake table was last updated.

This makes it difficult to automate sanity checks. For example:

• NotRun status could mean the table truly hasn’t changed, or it could indicate unexpected behavior.
• To validate, users currently need to manually query the DeltaLake table for the last modification time, which adds complexity to pipelines and monitoring.
  
  

Proposed Solution:
Add a field like lastDeltaLakeUpdateTime to the API response for each table. This timestamp would represent the most recent modification to the underlying DeltaLake table.

Benefits:

1. Improved observability: Users can directly see whether a DeltaLake table was updated since the last sync.
2. Reliable automation: Pipelines can programmatically verify that a NotRun status is expected and that the DeltaLake table contains recent data.
3. Easier debugging: Reduces the need to query DeltaLake tables manually or maintain additional tracking logic.

Example of proposed API response snippet:

{ 
"tableName": "Table 1", 
"startDateTime": "2025-02-04T22:29:12.4400865Z", 
"endDateTime": "2025-02-04T22:29:12.4869641Z", 
"status": "NotRun", 
"lastSuccessfulSyncDateTime": "2024-07-23T14:28:23.1864319Z",
"lastDeltaLakeUpdateTime": "2024-07-23T14:28:12.1234567Z" 
}

This addition would improve the Refresh SQL Endpoint Metadata APIs usability for monitoring and automated validation in Fabric pipelines.