Error when loading Json using Dataflow Gen2

[edchuy]

I'm getting the following error when trying to do load a relative large (~1 GB) Json file using Dataflow Gen2. Please help!

[Ehren]

Is the JSON file a single monolithic object? If so, that could explain the error.

[edchuy]

Would that JSON be a one line one with about a billion characters in it? How can you fix that?

[Ehren · Accepted solution]

You'd have to try to break it up into smaller json objects.

For example, if you control the source file format, you could try using the json lines format instead.

Or if you can't change the file format, you could pre-process it using some of our text functions that stream the input, break it up into the individual nested child objects, and process each of those using Json.Document.

A condensed/redacted sample of what the file looks like might help me provide more concrete suggestions.

[edchuy]

Thanks for the suggestions! I guess they probably do need to fix it so that those types of workarounds aren't required.

Is there any external tools you could suggest I could use to do the processing you mentioned?

[Ehren]

I can suggest ways to process the current format in PQ, but I unfortunately can't tell you how to change the file format itself. That likely depends on how the file is currently being produced.