Awaiting Futures not blocking when present in loops or functions

[anawast]

As per this documentation: https://docs.scala-lang.org/overviews/core/futures.html
Await.results() should block until all futures have completed. This works as expected when called in an individual cell in a Fabric Notebook. However, when called within a loop or if present in a function and called through the function- the same doesn't work. 

As you can see below things work fine if called within a cell: 

However if called within a function (or in a loop)- the same doesn't work.

[anawast]

Can you suggest what you would use instead of the below code, in that case?

def parallelNotebooks(notebookMetadata):Future[Seq[String]]=

{

//runs notebooks in parallel

}

breakable {

    while (true) {

        val now: LocalDateTime = LocalDateTime.now()

        println(now)

        val duration = java.time.Duration.between(start, now)

        val executedMinutes = duration.toHours * 60 + duration.toMinutes

        if (executedMinutes > threshold_executionMinutes) {

            break()

        }

        val df = // Get data from metadata

        if (df.count() == 0) {

            break()

        }

        val notebookMetadata = //get seq from df

        val res = parallelNotebooks(notebookMetadata)

        Await.result(res, scala.concurrent.duration.Duration.Inf)

    }

}

[v-menakakota]

Hi @anawast ,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[v-menakakota]

Hi @anawast ,

I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.

Thank you.

[v-menakakota · Accepted solution]

Hi @anawast ,

I have successfully reproduced your scenario and observed the behavior of Await.result() inside loops. Below is the output I received when executing the modified version of your code: 

Key Findings: 

• Await.result() works correctly in a single cell and inside a function. 

• Inside a loop, execution may continue unexpectedly due to Fabric Notebook's execution model. 

• To ensure proper blocking, an updated approach is required. 

Solution & Workaround: 

To ensure Await.result() fully blocks execution inside a loop, you can use the following approach: 

Code: 

import scala.util.control.Breaks._ 

import java.time.LocalDateTime 

import java.time.Duration 

import scala.concurrent.{Future, Await} 

import scala.concurrent.ExecutionContext.Implicits.global 

import scala.concurrent.duration._ 

val start = LocalDateTime.now() 

val threshold_executionMinutes = 1  // Adjust as needed 

breakable { 

    while (true) { 

        val now: LocalDateTime = LocalDateTime.now() 

        val duration = Duration.between(start, now) 

        val executedMinutes = duration.toMinutes 

        if (executedMinutes >= threshold_executionMinutes) { 

            println("Execution threshold reached. Exiting loop.") 

            break() 

        } 

        val notebooks = Seq("Notebook_X", "Notebook_Y", "Notebook_Z") 

        println(" Awaiting notebook execution...") 

        val result = Await.result(Future.sequence(notebooks.map { notebook => 

            Future { 

                println(s"Running notebook: $notebook") 

                Thread.sleep(5000)  // Simulate notebook execution 

                s"Completed: $notebook" 

            } 

        }), scala.concurrent.duration.Duration.Inf)  // Ensure full blocking 

        println(s"Iteration complete. Result: $result") 

        println(" Sleeping for 10 seconds before next iteration...\n") 

        Thread.sleep(10000)  // Ensure loop doesn’t execute too fast 

    } 

} 

Expected Behavior: 

• Each iteration waits until all notebooks finish execution before proceeding. 

• The loop terminates after the specified time threshold (1 minute in this example). 

• The execution order may vary slightly due to parallel execution of Future tasks. 
   

Please try this solution in your Fabric Notebook and inform me if you continue to encounter issues. If this resolves your problem, kindly Accept it as a solution and give a "Kudos" to assist other members in finding it more easily. 
I hope this works for you. 

Thank you and Regards,
Menaka.

[v-menakakota]

Hi @anawast ,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a Kudos so other members can easily find it.

Thank you and Regards,
Menaka

[v-menakakota]

Hi @anawast ,

Thank you for reaching out to us on the Microsoft Fabric Community Forum

Await.result blocks the thread until the Future completes. When used within a loop or a function, this blocking behavior can lead to inefficiencies and potential deadlocks, especially if multiple threads are involved.I suggest using Avoid Blocking in Loops/Functions, Instead of using Await.result within a loop or a function.

If this post was helpful, please give us Kudos and consider marking Accept as solution to assist other members in finding it more easily.

Regards,
Menaka.