topic Re: Error when loading Json using Dataflow Gen2 in Dataflow

Error when loading Json using Dataflow Gen2

edchuy — Wed, 09 Aug 2023 16:57:05 GMT

I'm getting the following error when trying to do load a relative large (~1 GB) Json file using Dataflow Gen2. Please help!

Re: Error when loading Json using Dataflow Gen2

Ehren — Thu, 10 Aug 2023 20:15:56 GMT

Is the JSON file a single monolithic object? If so, that could explain the error.

Re: Error when loading Json using Dataflow Gen2

edchuy — Thu, 10 Aug 2023 23:07:22 GMT

Would that JSON be a one line one with about a billion characters in it? How can you fix that?

Re: Error when loading Json using Dataflow Gen2

Ehren — Fri, 11 Aug 2023 17:17:38 GMT

You'd have to try to break it up into smaller json objects.

For example, if you control the source file format, you could try using the json lines format instead.

Or if you can't change the file format, you could pre-process it using some of our text functions that stream the input, break it up into the individual nested child objects, and process each of those using Json.Document.

A condensed/redacted sample of what the file looks like might help me provide more concrete suggestions.

Re: Error when loading Json using Dataflow Gen2

edchuy — Mon, 14 Aug 2023 21:42:03 GMT

Thanks for the suggestions! I guess they probably do need to fix it so that those types of workarounds aren't required.

Is there any external tools you could suggest I could use to do the processing you mentioned?

Re: Error when loading Json using Dataflow Gen2

Ehren — Mon, 14 Aug 2023 22:01:42 GMT

I can suggest ways to process the current format in PQ, but I unfortunately can't tell you how to change the file format itself. That likely depends on how the file is currently being produced.