Expression.Error: We cannot apply operator < to types Time and Number

You have qualified Start Time with Time.Hour which is gives a number output which then you are compring with time and this gives the error. Both side should be having same unit, hence either they should have time on both side or number on both side.

Approach 1 - When time on both side

= if [Start Time] >=Time.FromText("00:00:00am") and [Start Time]<Time.FromText("09:00:00am")
and [End Time]<Time.FromText("09:00:00am")
then if [End Time] > [Start Time] then [End Time] - [Start Time] else 1-[Start Time] + [End Time]
else null

Approach 2 - When number on both side

= if Time.Hour([Start Time]) >=0 and Time.Hour([Start Time])<9 and Time.Hour([End Time])<9
then if [End Time] > [Start Time] then [End Time] - [Start Time] else 1-[Start Time] + [End Time]
else null