topic Re: Unexpected error Region Canada when trying to read a file from OneLake in Dataflow

Unexpected error Region Canada when trying to read a file from OneLake

mrojze — Fri, 18 Aug 2023 14:31:26 GMT

This is my situation:

I used ADF to get a JSON file from an API.

I placed the file in my new Lakehouse files section.

Now I am trying to read the JSON file so I can start getting the tables I wanted and place them in the Tables section.

Unfortunately, I am getting an Unexpected error and I can't find the way around it.

Am I doing something wrong here?

Thanks!

Martin

Re: Unexpected error Region Canada when trying to read a file from OneLake

DennesTorres — Sat, 19 Aug 2023 20:06:01 GMT

Hi,

This image is the connection window from a Dataflow Gen 2, right?

Could you try this using a pipeline in the same workspace as the data lake?

The connection process is easier and it can get the data from "Files" to "Tables".

About the exactly error you are getting, I don't know the root cause, but if the pipeline solves your problem, you don't need to face this error.

Kind Regards,

Dennes

P.S: The other time I faced this error was related to an outdated data gateway. An updated solved. But I know this makes no sense in your scenario.

Re: Unexpected error Region Canada when trying to read a file from OneLake

mrojze — Mon, 21 Aug 2023 16:32:03 GMT

Thanks Dennes.

It looks like you can't use Dataflows in the files section. I was hoping to use it since it'd be easier to read all the sections of the JSON file at once.

Now I have to do a copy per section.

Thanks!

Martin

Re: Unexpected error Region Canada when trying to read a file from OneLake

DennesTorres — Mon, 21 Aug 2023 16:48:20 GMT

Hi,

Yes, in this case I belive you are facing a current limitation.

If the copy behaviour/flatten hierarchy doesn't help, maybe you can consider the possibility of using an azure data factory dataflow. I believe it will be able to connect to the files area of a lakehouse, although I haven't tested.

Kind Regards,

Dennes

Re: Unexpected error Region Canada when trying to read a file from OneLake

miguel — Mon, 21 Aug 2023 17:10:20 GMT

Hey!

Have you tried using the Lakehouse connector and the section to connect to the files inside of it?

If not, please give it a try and let us know if that works for you.

Re: Unexpected error Region Canada when trying to read a file from OneLake

miquelladeboer — Mon, 21 Aug 2023 17:11:43 GMT

Hi Martin,

You can definetly use dataflows gen 2 to connect to your files in the Lakehouse and no need to use Pipelines instead.

If you use the Lakehouse connector in dataflows, you can simply browse the file part of your Lakehouse and select the files you want to connect to.

If you keep experiences an unexpeted error, please open a ticket so we can invetsigate.

Re: Unexpected error Region Canada when trying to read a file from OneLake

DennesTorres — Mon, 21 Aug 2023 18:03:40 GMT

Hi,

You are right. I double-checked and I noticed the access to the files is "hidden" in such a way I believed it was not possible.

This is the image from the connector:

The way the "Files" folder is inside a "database" icon and closed by default make it seems like it's not possible to access it. But it is.

But there is another important detail (which made me miss it): The "Files" is only acessible as a source. When considered as a destination, it's not acessible. The image below illustrates the lakehouse as destination, only with the "Tables" area available, no "Files" folder:

Kind Regards,

Dennes