Experiments and parallel processing going wrong

nielsvdc, 

Thanks for reaching out to MS Fabric community support.

Your code is mostly on track, but there are a few key changes needed to ensure the MLflow experiments and runs are correctly isolated when using ThreadPoolExecutor. Specifically, you need to make sure that:

1. Each location's prediction process is contained within a separate MLflow experiment.
2. The nested runs (for each iteration) are correctly managed and don't interfere with other locations.
3. No shared state is accessed concurrently in a way that could cause the wrong experiment or run to be affected.

Please follow below improvements:

1. Set Experiment Properly: The call to mlflow.set_experiment(location_name) inside run_prediction_task is correct for setting a separate experiment per location. However, we need to ensure that mlflow.start_run() is executed within the context of each thread and location.

2. Unique Run Names: You're using run_name=f"Prediction_{run_timestamp}" to uniquely identify the main run. That’s great! This will ensure each location has its own main run.

3. Manage Iteration Runs: You should avoid using nested=True in the mlflow.start_run() if you want fully independent runs (and if the iteration runs don't need to be nested within the main run). If nested runs are necessary, you can keep nested=True, but the parent-child relationship between runs can sometimes lead to issues when parallelizing execution.

4. DataFrame Management: Ensure that your df_with_predictions isn't being modified by multiple threads simultaneously. Each thread should work with its own version of the data.

5. MLflow Context in Threads: Make sure that each thread correctly creates and manages its own experiment and run context.

You can find more information on managing MLflow experiments and runs in the official documentation:

• MLflow Tracking API
• Logging runs and metrics

Thanks,

Prashanth Are

MS Fabric community support