Forum Discussion

Scott_Powell's avatar
Scott_Powell
Icon for Helper V rankHelper V
2 years ago

Limitations on Lakehouse table columns in SQL endpoint?

Hi, does anyone know if there's a limitation on table columns in the SQL endpoints of a Lakehouse? I've got a table which tracks county statistics that has 720 columns. I was able to load this table ...
data engineering
one lake
v-cboorla-msft's avatar
v-cboorla-msft
Icon for Microsoft Employee rankMicrosoft Employee
2 years ago

Hi Scott_Powell 

 

Apologies for the delay in response.
Follow up to see if you have a resolution yet on your issue.
In case if you have any resolution please do share that same with the community as it can be helpful to others. 
In case if you didn't get any resolution.

Please go ahead and raise a support ticket to reach our support team: https://support.fabric.microsoft.com/support

Please provide the ticket number here as we can keep an eye on it.

 

Thank you.

Scott_Powell's avatar
Scott_Powell
Icon for Helper V rankHelper V
2 years ago

Hi v-cboorla-msft - I actually don't think this is an issue with "too many columns". Instead, I think it's an issue with combining data from multiple .csv input files, where many of them don't have all the columns. I think something is going wrong during the data load to the table (via a notebook using dataframes), but no error is being shown. 

 

I suspect this because I loaded "only" the latest file - and everything worked fine.

 

Trying now to figure out how to load all of the years .csv files, plugging in NULLs for columns that don't exist. But I'm not at all fluent in PySpark so really struggling.

Long and short though I don't believe it's a limit on number of columns - everything works fine even with 700 columns if I stick to just a single year.

 

Thanks,

Scott