Forum Discussion
Event or Timestamp for Shortcut Update
Hi houckm_amhs,
Thank you for reaching out to the Microsoft fabric community forum and thank you, suparnababu8 for the valuable insights.- You can absolutely retrieve metadata for shortcut tables, whether they’re in ADLS, OneLake, S3 or other supported storage. Shortcuts act as pointers to the source data, so you can use standard Fabric tools to access metadata without scanning row-level data. For Delta table shortcuts, check the delta log for the latest version or timestamp. For non-Delta shortcuts use file system metadata like file modification times. These methods work regardless of the storage type, if the shortcut is properly configured with access credentials.
- Yes, standard Fabric functions (e.g: Spark SQL, PySpark or SQL analytics endpoint) work on shortcuts just like regular Lakehouse tables. For Delta shortcuts, use DESCRIBE HISTORY to get updated timestamps or versions. For non-Delta, file system APIs can list file timestamps. Note that Warehouse shortcuts may have a ~5-10-minute cache latency, so you might need a REFRESH command in SQL for freshness.
For Live Data, your 15–30-minute triggers, schedule a lightweight pipeline to check metadata. For lower latency, consider Fabric Eventstream for streaming data or Azure Event Grid for ADLS/S3 file events, if applicable and Test with a few shortcuts in a dev Lakehouse to validate metadata retrieval.
Please consider below offical documentations:
Best regards,
Ganesh Singamshetty.- You can absolutely retrieve metadata for shortcut tables, whether they’re in ADLS, OneLake, S3 or other supported storage. Shortcuts act as pointers to the source data, so you can use standard Fabric tools to access metadata without scanning row-level data. For Delta table shortcuts, check the delta log for the latest version or timestamp. For non-Delta shortcuts use file system metadata like file modification times. These methods work regardless of the storage type, if the shortcut is properly configured with access credentials.
Hi suparnababu8 - When you say "Try to create a metadata table like refresh_log table having columns shuch as shortcutname, timestampmax, lastcheckin time" - we do have those tables for the data that we're loading directly, but how can we find out the timestamps and metadata for shortcut tables? Some of the data shortcutted may not be ADLS.
Are there standard functions that can be called on shortcuts to pull out that metadata? Will the same functions that work on regular lakehouse tables work? I haven't done extensive testing there yet.
Community SupportHi houckm_amhs,
Thank you for reaching out to the Microsoft fabric community forum and thank you, suparnababu8 for the valuable insights.
- You can absolutely retrieve metadata for shortcut tables, whether they’re in ADLS, OneLake, S3 or other supported storage. Shortcuts act as pointers to the source data, so you can use standard Fabric tools to access metadata without scanning row-level data. For Delta table shortcuts, check the delta log for the latest version or timestamp. For non-Delta shortcuts use file system metadata like file modification times. These methods work regardless of the storage type, if the shortcut is properly configured with access credentials.
- Yes, standard Fabric functions (e.g: Spark SQL, PySpark or SQL analytics endpoint) work on shortcuts just like regular Lakehouse tables. For Delta shortcuts, use DESCRIBE HISTORY to get updated timestamps or versions. For non-Delta, file system APIs can list file timestamps. Note that Warehouse shortcuts may have a ~5-10-minute cache latency, so you might need a REFRESH command in SQL for freshness.
For Live Data, your 15–30-minute triggers, schedule a lightweight pipeline to check metadata. For lower latency, consider Fabric Eventstream for streaming data or Azure Event Grid for ADLS/S3 file events, if applicable and Test with a few shortcuts in a dev Lakehouse to validate metadata retrieval.
Please consider below offical documentations:
Best regards,
Ganesh Singamshetty.