Forum Discussion

AlexanderPowBI's avatar
AlexanderPowBI
Icon for Resolver I rankResolver I
2 years ago
Solved

Ingesting Varchar(MAX) to datawarehouse failure

Hello,   I am trying to build a pipeline for a table we have in our azure sql database into Fabric. One of the columns has varchar(MAX) and can be quite large (intentionally).  When I try to use t...
data engineering
Data Integration
data warehouse
  • AlexanderPowBI's avatar
    AlexanderPowBI
    2 years ago

    Hi,

    So I found it possible to save records larger than the 8000 limit into a delta table in lakehouse, but as you wrote, the SQL analytic endpoint will truncate it. 

    I have went on and remodelled the column in a notebook to continue my work. Thank you for your answers! 

AlexanderPowBI's avatar
AlexanderPowBI
Icon for Resolver I rankResolver I
2 years ago

Hi,

Thank you for your response. As I cannot change how the data is set up in the app I will need to handle it with one of your options. However, I thoguht of another option that might work and that is to bring the data into a lakehouse delta table. Or does the lakehouse have such limitaiton aswell? 

 

thanks,

Alexander

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi AlexanderPowBI ,

Unfortunately both are having the same limitation -


But I suggest you to please give a try. If that doesn't work then you can follow any of the above solutions.

Hope this answers your query. Please let me know in case of further queries.

  • AlexanderPowBI's avatar
    AlexanderPowBI
    Icon for Resolver I rankResolver I
    2 years ago

    Hi,

    So I found it possible to save records larger than the 8000 limit into a delta table in lakehouse, but as you wrote, the SQL analytic endpoint will truncate it. 

    I have went on and remodelled the column in a notebook to continue my work. Thank you for your answers! 

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi AlexanderPowBI ,

      Thanks for you response. I hope you got some insights while using Fabric Community.
      Please continue using Fabric Commnity on your further queries.

    • Element115's avatar
      Element115
      Icon for Memorable Member rankMemorable Member
      2 years ago

      Could you please elaborate on what you mean by 'remodelled the column in a notebook' with a snapshot of code from your Notebook?  Would help us learn something new.

    • Element115's avatar
      Element115
      Icon for Memorable Member rankMemorable Member
      2 years ago

      Also, would switching to CosmoDB have taken care of this issue?

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi AlexanderPowBI ,

    We haven’t heard from you on the last response and was just checking back to see if your query was answered.
    Otherwise, will respond back with the more details and we will try to help .

    Thanks