- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Dataflow Gen 2 struggling to use Lakehouse as source
Hi,
We're trying to build a data warehouse in Fabric and struggling with data ingestion. Pipelines for an on-prem SQL gateway isn't currently available and we're having the same problems as others writing to a lakehouse using Dataflow Gen2. However, we're also seeing lots of other apparently sporadic performance related issues, where things will work sometimes and other times not at all, or jobs will take either 20 mins or time out after 8 hours. One issue that we're able to replicate consistently is an inability to read from our lakehouse using Dataflows; see attached error message which follows after about 5-10 minutes of hanging.
The data in the lakehouse isn't huge; 4 tables - pretty chunky ones, but we're able to access them via pipelines, SQL endpoints and Notebooks seemingly without issue. Does anyone know what the problem might be, or how we might work around it?
Thanks,
Paul.
Solved! Go to Solution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi PaulCH,
There is an onoing Lakehouse issue that results in these "A time out occurred while waiting to optimize the query." errors. Engineers are investigating with priority.
Thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Okay, thanks. Fingers crossed, I guess.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi PaulCH,
There is an onoing Lakehouse issue that results in these "A time out occurred while waiting to optimize the query." errors. Engineers are investigating with priority.
Thanks

Helpful resources
Subject | Author | Posted | |
---|---|---|---|
09-03-2024 06:29 AM | |||
Anonymous
| 08-08-2024 11:55 PM | ||
03-13-2024 02:41 AM | |||
09-13-2024 04:44 AM | |||
07-10-2024 12:40 AM |