Forum Discussion
Solved
How calculating the difference between two datetimes in minutes in pyspark SQL
Hi, in a my lakehouse table I've a timestamp or datetime data. I need to calculate the difference between this timestamp and the current timestamp, in minutes. I can determine the current timestam...
Hi, thanks for your reply.
I've solved with:
sql_sample_table = """ SELECT * FROM mytable WHERE cast(((unix_timestamp()-unix_timestamp(mytimestamp)) / 60) as int) > 30 """ spark.sql(sql_sample_table).show()unix_timestamp() returns the current timestamp in seconds.
The unix_timestamp statement in Spark SQL returns the number of seconds that have passed since January 1, 1970 (epoch time) until the specified date and time.
Thanks
Hi, thanks for your reply.
I've solved with:
sql_sample_table = """
SELECT * FROM mytable
WHERE cast(((unix_timestamp()-unix_timestamp(mytimestamp)) / 60) as int) > 30
"""
spark.sql(sql_sample_table).show()unix_timestamp() returns the current timestamp in seconds.
The unix_timestamp statement in Spark SQL returns the number of seconds that have passed since January 1, 1970 (epoch time) until the specified date and time.
Thanks
HI pmscorca,
I'm glad to hear you get the resolution and share it here, they will help others who has the similar requirement.
Regards,
Xiaoxin Sheng