Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago
Solved

Loading delta tables into a lakehouse

Hi All,   Please HELP, I am new to fabric and struggling to load my files to delta tables in a lakehouse. once uploading my data file (in CSV.) format and then Load to tables it gives an error reg...
data engineering
  • Anonymous's avatar
    Anonymous
    2 years ago

    Thanks for the help, how do I ensure that my headers are UTF-8 encoded Unicode word characters? I am so lost, thank you for the explanation above.

  • BartStens's avatar
    BartStens
    2 years ago

    Dear Anonymous ,

     

    I was also struggling with exact same error msg. My headers followed requirements exactly as UTF-8 demands, but still got the error msg.

     

    Turns out I had to specify my column-seperator a.k.a. delimiter in the prompt when selecting the three dots near the file > "Load file in new table" then enter de correct delimiter used in your .csv-file, in my case a comma (,) was pre-filled in the prompt, but my .csv-file used (;) so had to adjust that one and load was succesfull!

     

    See below screencap (sorry it's in Dutch-UI)

v-cboorla-msft's avatar
v-cboorla-msft
Icon for Microsoft Employee rankMicrosoft Employee
2 years ago

Hi Anonymous 

 

Thanks for using Microsoft Fabric Community.

 

The error message you received indicates that there are invalid column names in your CSV file. Specifically, the column names listed include characters that do not meet the requirements for Delta tables.

Delta tables expect column names to be UTF-8 encoded Unicode word characters, which means they can include letters (of any case), nonspacing marks, punctuation connectors (like underscores), and decimal digits. However, spaces are not allowed. To resolve this, ensure that your CSV file’s column headers adhere to these rules.

Table names must contain alphanumeric characters and underscores. Column names allow any English letters (upper or lower case), underscores, and characters from other languages (such as Chinese) in UTF, up to 32 characters. If a proper column name cannot be achieved during validation, the load action will fail.

 

I hope this information helps. Please do let us know if you have any further queries.

 

Thank you.

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Thanks for the help, how do I ensure that my headers are UTF-8 encoded Unicode word characters? I am so lost, thank you for the explanation above.

  • MartinMason's avatar
    MartinMason
    Icon for Resolver I rankResolver I
    2 years ago

    I'm pretty sure your issue is that your column names contain spaces, i.e. Customer Name. You'll need to go through the list of columns and replace the spaces, either CustomerName or Customer_Name are both valid.

    • swapnasam's avatar
      swapnasam
      New Member
      1 year ago

      Hi, I had the same problem and its just when you try to load the dataset to Delta Table from UI, its forcing us to use a proper format for column headers with no spaces or special character. 

       

      You can solve this problem by using a notebook

      Load the csv into a pandas df and write it with delta.columnMapping.mode", "name" which lets the df to write into a Table with spaces in column headers.

       

      ``` 

      import pandas as pd
      # Load data into pandas DataFrame from "/lakehouse/default/Files/metadata.csv"
      df = pd.read_csv("/lakehouse/default/Files/metadata.csv")
      display(df)
      spark_df = spark.createDataFrame(df)
      spark_df.write.format("delta").option("delta.columnMapping.mode", "name").saveAsTable("dbo.Sample")
      ```
       
      Hope this helps.
      Thanks