Dataflow Gen 2 struggling to use Lakehouse as source

[PaulCH]

Hi,

We're trying to build a data warehouse in Fabric and struggling with data ingestion. Pipelines for an on-prem SQL gateway isn't currently available and we're having the same problems as others writing to a lakehouse using Dataflow Gen2. However, we're also seeing lots of other apparently sporadic performance related issues, where things will work sometimes and other times not at all, or jobs will take either 20 mins or time out after 8 hours. One issue that we're able to replicate consistently is an inability to read from our lakehouse using Dataflows; see attached error message which follows after about 5-10 minutes of hanging.

The data in the lakehouse isn't huge; 4 tables - pretty chunky ones, but we're able to access them via pipelines, SQL endpoints and Notebooks seemingly without issue. Does anyone know what the problem might be, or how we might work around it?

Thanks,

Paul.

[PaulCH]

Okay, thanks. Fingers crossed, I guess.

[SidJay · Accepted solution]

Hi PaulCH,

There is an onoing Lakehouse issue that results in these "A time out occurred while waiting to optimize the query." errors. Engineers are investigating with priority. 

Thanks