Fetch the File modification time

Hi mahid721 ,

Yes, you can definitely get the file modification time from a Lakehouse file, even though mssparkutils.fs doesn't expose this directly.

The simplest approach is to use the Spark API with the Hadoop FileSystem. Here's a code snippet you can try in your notebook:

from pyspark.sql import SparkSession
from pyspark.sql.functions import *

# Get the Spark session
spark = SparkSession.builder.getOrCreate()

# Path to your file in the lakehouse
file_path = "Files/your_folder/your_file.csv"  # adjust this path to your file

# Get the Hadoop FileSystem
fs = spark._jvm.org.apache.hadoop.fs.FileSystem.get(spark._jsc.hadoopConfiguration())

# Get file status which contains modification time
file_status = fs.getFileStatus(spark._jvm.org.apache.hadoop.fs.Path(file_path))

# Get the modification time as a timestamp (in milliseconds)
mod_time_ms = file_status.getModificationTime()

# Convert to a readable datetime format
import datetime
mod_time = datetime.datetime.fromtimestamp(mod_time_ms/1000).strftime('%Y-%m-%d %H:%M:%S')

print(f"Last modified time: {mod_time}")

Alternatively, if you want this in a dataframe format (maybe for multiple files), you can do:

# List the files in a directory with their details
files_info = spark.sql(f"CALL lakehouse.system.files('Files/your_folder/')")
files_info.select("name", "size", "modificationTime").show()

Hope this helps with your problem!

If my response resolved your query, kindly mark it as the Accepted Solution to assist others. Additionally, I would be grateful for a 'Kudos' if you found my response helpful.