Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
3 years ago

Lakehouse Add or Remove columns from table

Not sure if this is the right forum or not - but here is the issue.   We are loading data to a Lakehouse using gen 2 data flows (for now they are just pointing at exisitn gen 1 dataflows then doi...
Dataflows
asittrivedi's avatar
asittrivedi
Regular Visitor
3 years ago

I have used a Python notebook to add a column to an existing table and that works just fine. One can use spark dataframe or pyspark.pandas dataframe to get the desired outcome. 

  • coolie's avatar
    coolie
    Icon for Helper I rankHelper I
    2 years ago

    could you share how to do this in python?

    • asittrivedi's avatar
      asittrivedi
      Regular Visitor
      2 years ago
      import pyspark.pandas as ps
      import pandas as pd
      import numpy as np
      from pyspark.sql import *
       
      psdf = ps.read_delta('path_to_table')
      psdf.head(10)
       
      psdf['new_col'] = ''
      psdf.head(10)
       
      sdf = psdf.to_spark()
       
      sdf.write.mode('overwrite').saveAsTable('existing_table')
       
      Once can use a spark dataframe in lieu of pandas.
       
      How ever the easiest option now is to use a SQL notebook and add a column. Please refer
  • murray-matthew's avatar
    murray-matthew
    Frequent Visitor
    2 years ago

    Would really appreciate any additional insight or links to resources that could be provided on this subject.