Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago
Solved

Loading delta tables into a lakehouse

Hi All,   Please HELP, I am new to fabric and struggling to load my files to delta tables in a lakehouse. once uploading my data file (in CSV.) format and then Load to tables it gives an error reg...
data engineering
  • Anonymous's avatar
    Anonymous
    2 years ago

    Thanks for the help, how do I ensure that my headers are UTF-8 encoded Unicode word characters? I am so lost, thank you for the explanation above.

  • BartStens's avatar
    BartStens
    2 years ago

    Dear Anonymous ,

     

    I was also struggling with exact same error msg. My headers followed requirements exactly as UTF-8 demands, but still got the error msg.

     

    Turns out I had to specify my column-seperator a.k.a. delimiter in the prompt when selecting the three dots near the file > "Load file in new table" then enter de correct delimiter used in your .csv-file, in my case a comma (,) was pre-filled in the prompt, but my .csv-file used (;) so had to adjust that one and load was succesfull!

     

    See below screencap (sorry it's in Dutch-UI)

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Thanks for the help, how do I ensure that my headers are UTF-8 encoded Unicode word characters? I am so lost, thank you for the explanation above.

MartinMason's avatar
MartinMason
Icon for Resolver I rankResolver I
2 years ago

I'm pretty sure your issue is that your column names contain spaces, i.e. Customer Name. You'll need to go through the list of columns and replace the spaces, either CustomerName or Customer_Name are both valid.

  • swapnasam's avatar
    swapnasam
    New Member
    1 year ago

    Hi, I had the same problem and its just when you try to load the dataset to Delta Table from UI, its forcing us to use a proper format for column headers with no spaces or special character. 

     

    You can solve this problem by using a notebook

    Load the csv into a pandas df and write it with delta.columnMapping.mode", "name" which lets the df to write into a Table with spaces in column headers.

     

    ``` 

    import pandas as pd
    # Load data into pandas DataFrame from "/lakehouse/default/Files/metadata.csv"
    df = pd.read_csv("/lakehouse/default/Files/metadata.csv")
    display(df)
    spark_df = spark.createDataFrame(df)
    spark_df.write.format("delta").option("delta.columnMapping.mode", "name").saveAsTable("dbo.Sample")
    ```
     
    Hope this helps.
    Thanks