8000 Character limit SQL Endpoint For Lakehouse Fabric

[r_Gao]

Hi!

We saw this blog post. 

https://blog.fabric.microsoft.com/en-us/blog/working-with-large-data-types-in-fabric-warehouse/

It mentioned expanding capabiltiy of fabric warehouse to handle varchar(max). And it said 

The VARCHAR(MAX) type will open new scenarios by enabling you to store semi-structured data formatted as JSON without worrying about potential parsing errors due to truncation. JSON formatted text commonly exceeds 8KB in length, and with the new 1MB storage size, most JSON documents will fit into the warehouse columns. Also, the introduction of VARCHAR(MAX) and VARBINARY(MAX) opens the possibility to enhance SQL endpoints for Lakehouse and mirrored databases, as the string or binary data will no longer be truncated to 8KB.

Does this mean after this update we should be able to use the sql endpoint for fabric Lakehouse on fields > 8000 characters? Or is this a future update

[AlexanderPowBI]

Hello, found this old post so I better write here then start a new. 

I have columns with data of length > 8k in my lakehouse. Querying them in notebook using pyspark, there is no issues. However, when I query this using SQL endpoint, it truncates the data. Is this expected behavior or not? 

[r_Gao]

This is what I have experienced. They are aware and there is no date planned for when text wont be truncated when querying from sql endpoint

[AlexanderDoerr]

I am also experiencing the same problem. I did a complet new upload into a new lakehouse and everything over 8000 was cut of. 

Does the extension only apply to warehouses or also lakehouses ? 
Or is it still in the rollout phase?

[r_Gao]

according to support this feature is not yet released, no date announced for when it will be

[lsg_1987]

Hi! I don't see the VARCHAR(max) in effect on the SQL endpoint - it still limits to VARCHAR(8000). Is there anything that one should do in order for it to take effect?

[r_Gao]

I created a ticket. It is not in effect - but is planned to be released (no date)

[r_Gao]

Hi! I just tested it out and it appears that the string still gets truncated (it's supposed to say this is a test of mass proportions done by the great richard but got cut off at "a"). Is there something i need to do to activate this update?

[richbenmintz]

Assuming you have changed the table data type and reloaded the data, could it be that the export to excel is what is truncating the data? Have you tried to query the last occurance for the last word in the string?

[r_Gao]

I have been told by support this feature is not availble yet

[Anonymous]

Hi @r_Gao ,

This update allows you to use Fabric Lakehouse's SQL endpoint on fields larger than 8,000 characters.The introduction of the VARCHAR(MAX) and VARBINARY(MAX) types allows for storing larger data, such as text in JSON format, without truncation issues.

However, there may still be some ongoing tweaks and improvements after the update, so please feel free to contact community support if you need help.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

[prabhatnath]

Thanks @v-huijiey-msft for your response.
I am also exploring similar solution. I am not sure how do I save more than 8000 chars inside a table column? Are there VARCHAR(MAX) and VARBINARY(MAX)  column types supported in Lakehouse table. Can you please point me to some reference / documentation please?

For now I have stored my large text (about 9000 chars) data in a String column but unable to pull the same using the Endpoint as the endpoint only gets me 8000 chars. Can you help on this please?

Thanks,
Prabhat