Reply
PaulCH
Frequent Visitor

Dataflow Gen 2 struggling to use Lakehouse as source

Hi,

We're trying to build a data warehouse in Fabric and struggling with data ingestion. Pipelines for an on-prem SQL gateway isn't currently available and we're having the same problems as others writing to a lakehouse using Dataflow Gen2. However, we're also seeing lots of other apparently sporadic performance related issues, where things will work sometimes and other times not at all, or jobs will take either 20 mins or time out after 8 hours. One issue that we're able to replicate consistently is an inability to read from our lakehouse using Dataflows; see attached error message which follows after about 5-10 minutes of hanging.

PaulCH_0-1690294857719.png

The data in the lakehouse isn't huge; 4 tables - pretty chunky ones, but we're able to access them via pipelines, SQL endpoints and Notebooks seemingly without issue. Does anyone know what the problem might be, or how we might work around it?

Thanks,

Paul.

1 ACCEPTED SOLUTION
SidJay
Microsoft Employee
Microsoft Employee

Hi PaulCH,

 

There is an onoing Lakehouse issue that results in these "A time out occurred while waiting to optimize the query." errors. Engineers are investigating with priority. 

 

Thanks

View solution in original post

2 REPLIES 2
PaulCH
Frequent Visitor

Okay, thanks. Fingers crossed, I guess.

SidJay
Microsoft Employee
Microsoft Employee

Hi PaulCH,

 

There is an onoing Lakehouse issue that results in these "A time out occurred while waiting to optimize the query." errors. Engineers are investigating with priority. 

 

Thanks

avatar user

Helpful resources

Announcements
MarchFBCvideo - carousel

Fabric Monthly Update - March 2025

Check out the March 2025 Fabric update to learn about new features.

March2025 Carousel

Fabric Community Update - March 2025

Find out what's new and trending in the Fabric community.

Top Solution Authors (Last Month)
Top Kudoed Authors (Last Month)