Starting December 3, join live sessions with database experts and the Microsoft product team to learn just how easy it is to get started
Learn moreShape the future of the Fabric Community! Your insights matter. That’s why we created a quick survey to learn about your experience finding answers to technical questions. Take survey.
I am using a dataflow Gen2 to fetch a long, JSON-formatted table from an API.
This is being pulled into a Lakehouse so that it can be accumulatively loaded and incrementally refreshed.
Preferably, we would be pulling the JSON in as a large binary object, but the data destination button greys out if the query is not a table, and gives an error during refresh if it is a binary object in a table.
Therefore, we have to have the JSON as text in a table and then push the table through to the Lakehouse.
However, it seems there is a limit of 8,000 characters on a text field when being used with a Delta table or Lakehouse.
This makes the JSON text cut off and we cannot decode it later.
Due to there being an SQL endpoint and ABFS path, it seems we may be able to give commands to the Lakehouse externally.
We want to see if we can turn the field into a binary after it has already been saved inside the Lakehouse.
In the documentation, it says:
"To modify data in lakehouse delta tables, you have to switch to lakehouse mode and use Apache Spark."
Looking through the rest of the documentation, I see nothing about lakehouse mode.
I am also fairly new to Spark, Delta lakes, tables, and Azure in general.
If there is a way to either bypass this 8,000 character limit or send commands to change the text field to a binary, please give me some advice.
Thanks in advance.
I am facing the same issue.
I also tried to save into Warehouse, not lakehouse, I hit the same limit of around 7600 characters.
In Dataflow I have Text field (pulled from JSON) that contains close to a million characters.
I need to save it in Lake/Warehouse, how?
User | Count |
---|---|
35 | |
32 | |
21 | |
11 | |
8 |
User | Count |
---|---|
54 | |
43 | |
28 | |
13 | |
11 |