topic Shortcut shows different data in Data Engineering

Shortcut shows different data

SergioPerdiz — Thu, 11 Jul 2024 14:07:01 GMT

I have a shortcut in the silver lakehouse to a table in the bronze lakehouse.

However it seems like the data is not updated.

When I look at the SQL endpoint of the bronze lakehouse I get the below result for one specific example, which has the correct EndDate (2024-06-30)

However when I use the shortcut in the silver lakehouse to the same table from the bronze lakehouse it shows a EndDate of (2024-07-30).

This post, Datalake shortcut is not showing the same data (i.... - Microsoft Fabric Community, seems to be the same issue, but there isn't a clear solution.

Re: Shortcut shows different data

Anonymous — Fri, 12 Jul 2024 03:13:54 GMT

Hi @SergioPerdiz,

I think this should be related to the setting which define the ‘data time zone’, by default datetime values will be stored with UTC format.

For the first one , your SQL statement get the records from table. For the second one that run in notebook, you have set the time zone offset which may affect the datetime values.

Regards,

Xiaoxin Sheng

Re: Shortcut shows different data

SergioPerdiz — Fri, 12 Jul 2024 08:12:58 GMT

Hi @Anonymous,

Thanks, that was indeed the issue!

Then I have another question how can I handle the following.

The notebook is executed around 01:00 AM GMT +2, which is 11:00 PM UTC, however for some calculations I use the below code.

# Get today's date today_date = current_date().cast("string")

So that means that for last night's run it would return 2024/07/11, but it should be 2024/07/12.

A solution could be to change the code to the below, but I'm wondering if this would be the correct/best way to do it.

# Get today's date today_date = (current_date() +1).cast("string")

Re: Shortcut shows different data

Anonymous — Mon, 15 Jul 2024 07:02:33 GMT

HI @SergioPerdiz,

If you had to calculate with specific time zone datetime, you can add a new column to the table to calculate with the offset between UTC and local time zone. Than you can use the new column to work as local datetime.

Regards,

Xiaoxin Sheng