Dataflow Gen 2 struggling to use Lakehouse as source

PaulCH — Tue, 25 Jul 2023 14:24:13 GMT

Hi,

We're trying to build a data warehouse in Fabric and struggling with data ingestion. Pipelines for an on-prem SQL gateway isn't currently available and we're having the same problems as others writing to a lakehouse using Dataflow Gen2. However, we're also seeing lots of other apparently sporadic performance related issues, where things will work sometimes and other times not at all, or jobs will take either 20 mins or time out after 8 hours. One issue that we're able to replicate consistently is an inability to read from our lakehouse using Dataflows; see attached error message which follows after about 5-10 minutes of hanging.

The data in the lakehouse isn't huge; 4 tables - pretty chunky ones, but we're able to access them via pipelines, SQL endpoints and Notebooks seemingly without issue. Does anyone know what the problem might be, or how we might work around it?

Thanks,

Paul.

Re: Dataflow Gen 2 struggling to use Lakehouse as source

SidJay — Tue, 25 Jul 2023 17:56:20 GMT

Hi PaulCH,

There is an onoing Lakehouse issue that results in these "A time out occurred while waiting to optimize the query." errors. Engineers are investigating with priority.

Thanks

Re: Dataflow Gen 2 struggling to use Lakehouse as source

PaulCH — Tue, 25 Jul 2023 18:47:18 GMT

Okay, thanks. Fingers crossed, I guess.

topic Re: Dataflow Gen 2 struggling to use Lakehouse as source in Dataflow

Dataflow Gen 2 struggling to use Lakehouse as source

Re: Dataflow Gen 2 struggling to use Lakehouse as source

Re: Dataflow Gen 2 struggling to use Lakehouse as source