topic Re: QUESTION::LAKEHOUSE::BINARY COLUMN in Data Engineering

QUESTION::LAKEHOUSE::BINARY COLUMN

Element115 — Tue, 23 Apr 2024 19:54:14 GMT

Is this the expected display behavior in a LH view pane for a column of type binary?

Re: QUESTION::LAKEHOUSE::BINARY COLUMN

Anonymous — Wed, 24 Apr 2024 06:10:54 GMT

Hi @Element115
Thanks for using Fabric Commnity.
This is an expected behavior in Fabric Lakehouse.

System.Byte[] you encounter in programming is a data type specifically used to represent raw binary data. It's essentially an array that stores a collection of individual bytes.

Here's a breakdown of what it means:

System.Byte: This refers to the built-in Byte data type, which represents a single unsigned 8-bit integer. This means each value in the array can range from 0 to 255.
[]: The square brackets indicate it's an array, which is a collection of elements of the same data type. In this case, the array holds multiple Byte values.

Hope this helps. Please let me know if you have any further questions.

Re: QUESTION::LAKEHOUSE::BINARY COLUMN

Element115 — Wed, 24 Apr 2024 20:25:56 GMT

@Anonymous Thanks, but I've done enough software development in a previous life to know what System.Byte[] means.

That was not my question. The question is:

Is this really a good UX for all those users who do not have my software dev background? In other words, is the GUI supposed to show the content of a binary column as 'System.Byte[]'?

Couple days ago that's not how the column was displayed. That's not how it is displayed in a SQL Server DB client such as DBeaver. And that's not how it is displayed in PBI Power Query Editor. Aren't these enough hints to suggest that this is not the way to display a snapshot for type binary?

Further, why not auto detect if the binary represent an image and automatically transform it to type text during the pipeline Copy data process since PBI Desktop can only handle such data when it is of type text, ie show the picture data in a report???

And finally, as a former GUI developer myself, this is horrible UX. I trust I do not have to explain why as it is so self-evident.

Re: QUESTION::LAKEHOUSE::BINARY COLUMN

Element115 — Wed, 24 Apr 2024 21:00:18 GMT

Besides, the lakehouse SQL endpoint does show the content of the same binary column as an hexadecimal string. So having the LH data preview pane show System.Byte[] is total UX nonsense.

Re: QUESTION::LAKEHOUSE::BINARY COLUMN

Anonymous — Thu, 25 Apr 2024 17:38:19 GMT

Hi @Element115
Thanks for the feedback.

Appreciate if you could share the feedback on our feedback channel, which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Feedback Link: https://ideas.fabric.microsoft.com/

Please let me know if you have any other questions.

Re: QUESTION::LAKEHOUSE::BINARY COLUMN

Anonymous — Mon, 29 Apr 2024 10:21:06 GMT

Hi @Element115
We haven’t heard from you on the last response and was just checking back to see if you could provide the feedback. Otherwise, will respond back with the more details and we will try to help.
Thanks

Re: QUESTION::LAKEHOUSE::BINARY COLUMN

Element115 — Mon, 29 Apr 2024 14:37:36 GMT

I posted the permalink of this topic on https://ideas.fabric.microsoft.com/ . Now we wait, I guess.

Re: QUESTION::LAKEHOUSE::BINARY COLUMN

arlindTrystar — Sat, 20 Jul 2024 22:02:03 GMT

@Element115 Have you found the correct way of readin this type of data back in pyspark notebook?

Re: QUESTION::LAKEHOUSE::BINARY COLUMN

Element115 — Mon, 22 Jul 2024 17:45:37 GMT

I have not tried.

Re: QUESTION::LAKEHOUSE::BINARY COLUMN

arlindTrystar — Tue, 23 Jul 2024 13:02:32 GMT

for anyone interested:

from pyspark.sql import functions as F df = spark.read.table('tableName') df = df.withColumn('columnName', F.hex('columnName'))

assuming that those bytes represent a hexadecimal value, which was my case.