Forum Discussion

Scott_Powell's avatar
Scott_Powell
Icon for Helper V rankHelper V
2 years ago

Limitations on Lakehouse table columns in SQL endpoint?

Hi, does anyone know if there's a limitation on table columns in the SQL endpoints of a Lakehouse? I've got a table which tracks county statistics that has 720 columns. I was able to load this table ...
data engineering
one lake
Scott_Powell's avatar
Scott_Powell
Icon for Helper V rankHelper V
2 years ago

Hi v-cboorla-msft , it's definitely a managed table. The only thing "odd" about it is that it has a large number of columns - 700 or so. I hadn't seen this on any other table, so I'm kinda assuming the large # of columns is what did it.

 

I'm happy to open a support ticket, or take a Teams call from anyone that needs more details.

 

Thanks,
Scott

v-cboorla-msft's avatar
v-cboorla-msft
Icon for Microsoft Employee rankMicrosoft Employee
2 years ago

Hi Scott_Powell 

 

Thanks for your response.

We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Icon for Microsoft Employee rankMicrosoft Employee
    2 years ago

    Hi Scott_Powell 

     

    Apologies for the delay in response.
    Follow up to see if you have a resolution yet on your issue.
    In case if you have any resolution please do share that same with the community as it can be helpful to others. 
    In case if you didn't get any resolution.

    Please go ahead and raise a support ticket to reach our support team: https://support.fabric.microsoft.com/support

    Please provide the ticket number here as we can keep an eye on it.

     

    Thank you.

    • Scott_Powell's avatar
      Scott_Powell
      Icon for Helper V rankHelper V
      2 years ago

      Hi v-cboorla-msft - I actually don't think this is an issue with "too many columns". Instead, I think it's an issue with combining data from multiple .csv input files, where many of them don't have all the columns. I think something is going wrong during the data load to the table (via a notebook using dataframes), but no error is being shown. 

       

      I suspect this because I loaded "only" the latest file - and everything worked fine.

       

      Trying now to figure out how to load all of the years .csv files, plugging in NULLs for columns that don't exist. But I'm not at all fluent in PySpark so really struggling.

      Long and short though I don't believe it's a limit on number of columns - everything works fine even with 700 columns if I stick to just a single year.

       

      Thanks,

      Scott