Forum Discussion

DebbieE's avatar
DebbieE
Community Champion
2 years ago
Solved

Notebook. New Errors when loding a parquet file into a df

I create a parquet file (transformed data) which I then use to create dims and facts.   Its been working fine. However I changed the code from "overwrite" to append the other day and now this code ...
data engineering
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi DebbieE ,

    I found this hint from the error message you provided:

    snappy.parquet. Column: [ID], Expected: int, Found: BINARY.

    You can try to check the data format of the ID column. If it does not help, you can provide me with your code so that I can find out if you are having problems.

     

    Best Regards,

    Ada Wang

    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi DebbieE ,

I found this hint from the error message you provided:

snappy.parquet. Column: [ID], Expected: int, Found: BINARY.

You can try to check the data format of the ID column. If it does not help, you can provide me with your code so that I can find out if you are having problems.

 

Best Regards,

Ada Wang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

  • DebbieE's avatar
    DebbieE
    Community Champion
    2 years ago

    What I have done is removed everything and started again and it seems to have resolved the problem

     

    I went from full load overwrite code to append code and this seemed to have caused the isues. I suspect, like you say above, its because i created the Dataframe before adding the data into it.

     

    Thanks for that