I am using a dataflow Gen2 to fetch a long, JSON-formatted table from an API.
This is being pulled into a Lakehouse so that it can be accumulatively loaded and incrementally refreshed.
Preferably, we would be pulling the JSON in as a large binary object, but the data destination button greys out if the query is not a table, and gives an error during refresh if it is a binary object in a table.
Therefore, we have to have the JSON as text in a table and then push the table through to the Lakehouse.
However, it seems there is a limit of 8,000 characters on a text field when being used with a Delta table or Lakehouse.
This makes the JSON text cut off and we cannot decode it later.
Due to there being an SQL endpoint and ABFS path, it seems we may be able to give commands to the Lakehouse externally.
We want to see if we can turn the field into a binary after it has already been saved inside the Lakehouse.
In the documentation, it says:
"To modify data in lakehouse delta tables, you have to switch to lakehouse mode and use Apache Spark."
Looking through the rest of the documentation, I see nothing about lakehouse mode.
I am also fairly new to Spark, Delta lakes, tables, and Azure in general.
If there is a way to either bypass this 8,000 character limit or send commands to change the text field to a binary, please give me some advice.
Thanks in advance.
