Forum Discussion
Bad Data in manifest.json
Hi all,
I am using a Data Pipeline to copy data from ADLS into a fabric data warehouse. I have about 11 snappy parquet files and one manifest.json file in the folder.When I'm trying to connect to the data source, it's throwing me this error;
Bad data is found at line 2 in source 444c16d6-4ad4-440b-a6de-157c2310f1e6/manifest.json.
You can ignore bad data by setting BadDataFound to null.
IReader state:
ColumnCount: 2
CurrentIndex: 0
HeaderRecord:
IParser state:
ByteCount: 0
CharCount: 39
Row: 2
RawRow: 2
Count: 2
RawRecord:
Hidden because ExceptionMessagesContainRawData is false.
How can I resolve this? Attached screenshot for reference
From that screenshot it look like you're trying to copy comma-delimited text and both the parquet files *and* the json file won't be comma-delimited.
If you are just after a straight copy of the files into a lakehouse, then if you select the option at the top right of your screenshot "Schema agnostic (binary copy)", then the files will get copied across without attempting to decode the files.
As a separate note, it is also possible to 'shortcut' the ADLS location such that the files appear to be in the lakehouse and can be used, but the actual data remains in the ADLS location.
2 Replies
From that screenshot it look like you're trying to copy comma-delimited text and both the parquet files *and* the json file won't be comma-delimited.
If you are just after a straight copy of the files into a lakehouse, then if you select the option at the top right of your screenshot "Schema agnostic (binary copy)", then the files will get copied across without attempting to decode the files.
As a separate note, it is also possible to 'shortcut' the ADLS location such that the files appear to be in the lakehouse and can be used, but the actual data remains in the ADLS location.