Forum Discussion
Machine Learning Prophet Issues
Good afternoon. I am learning how to use the ML models in Fabric Notebooks but am having issues with Prophet. When I run an expirement using AutoML, it tests multiple models and generally comes back with Prophet as the best. But when I save the model and run it, it fails because it doesn't have all of the regressors that were generated in the expirement I think. When I run other models (non-prophet) it works fine, but I cannot for the life of me run a Prophet model outside of an expirement.
HELP!
-Alex
Hi Anonymous , Thank you for reaching out to the Microsoft Community Forum.
You're getting a KeyError because Prophet in Microsoft Fabric’s AutoML expects a set of engineered features that were automatically created during training but aren’t regenerated at prediction time. Since Fabric doesn’t expose the exact featurization logic, you’ll need to manually recreate those features based on the error message and what the model expects.
First, use MLflow to inspect the required input columns for your model. Example:
import mlflow
model = mlflow.pyfunc.load_model("models:/your_model_name/1")
required_cols = model.metadata.get_input_schema().input_names()
print("Required columns:", required_cols)
Then, generate the missing features based on your ds datetime column. Example:
import pandas as pd
import numpy as np
df_future = pd.read_csv("your_inference_data.csv")
df_future['ds'] = pd.to_datetime(df_future['ds'])
df_future['hour'] = df_future['ds'].dt.hour
df_future['dayofweek'] = df_future['ds'].dt.dayofweek
df_future['dayofyear'] = df_future['ds'].dt.dayofyear
df_future['month'] = df_future['ds'].dt.month
df_future['quarter'] = df_future['ds'].dt.quarter
df_future['minute'] = df_future['ds'].dt.minute
df_future['second'] = df_future['ds'].dt.second
def add_cyclic_features(df, col, max_val, prefix):
df[f'{prefix}_sin'] = np.sin(2 * np.pi * df[col] / max_val)
df[f'{prefix}_cos'] = np.cos(2 * np.pi * df[col] / max_val)
return df
df_future = add_cyclic_features(df_future, 'hour', 24, 'ds_hour')
df_future = add_cyclic_features(df_future, 'dayofweek', 7, 'ds_dayofweek')
df_future = add_cyclic_features(df_future, 'dayofyear', 365, 'ds_dayofyear')
df_future = add_cyclic_features(df_future, 'month', 12, 'ds_month')
df_future = add_cyclic_features(df_future, 'quarter', 4, 'ds_quarter')
df_future = add_cyclic_features(df_future, 'minute', 60, 'ds_minute')
df_future = add_cyclic_features(df_future, 'second', 60, 'ds_second')
for i in range(1, 5):
df_future[f'ds_sin{i}'] = np.sin(i * 2 * np.pi * df_future['ds'].dt.dayofyear / 365)
df_future[f'ds_cos{i}'] = np.cos(i * 2 * np.pi * df_future['ds'].dt.dayofyear / 365)
Validate that all required columns are present before predicting. Example:
missing_cols = [col for col in required_cols if col not in df_future.columns]
if missing_cols:
raise ValueError(f"Missing columns: {missing_cols}")
Then make predictions. Example:
forecast = model.predict(df_future[required_cols])
print(forecast[['ds', 'yhat', 'yhat_lower', 'yhat_upper']])
If this helped solve the issue, please consider marking it “Accept as Solution” and giving a ‘Kudos’ so others with similar queries may find it more easily. If not, please share the details, always happy to help.
Thank you.
6 Replies
Hi Anonymous , Thank you for reaching out to the Microsoft Community Forum.
Prophet, unlike most other models, requires all the regressors used during training to be passed in again during prediction, with the same column names and types. Microsoft Fabric AutoML adds these regressors automatically through a process called auto_featurize, but when you export and run the model outside the AutoML experiment, those features aren't automatically recreated.
To fix this, go to Data Science -> Experiments in your Fabric workspace, open the AutoML run where Prophet was selected and check the artifacts, look for best_run_notebook.ipynb or model_summary.json. These should show the additional regressors added during training. If that's unclear, you can also inspect the model directly. Example:
import mlflow
model = mlflow.pyfunc.load_model("models:/your_model_name/1")
print(model.metadata.get_input_schema())
Then, make sure your prediction data includes all those same columns. For example:
import pandas as pd
df_future = pd.read_csv("your_inference_data.csv")
df_future['is_holiday'] = df_future['ds'].apply(lambda x: 1 if x in holiday_dates else 0)
df_future['day_of_week'] = pd.to_datetime(df_future['ds']).dt.dayofweek
# Add other regressors used during training
Be careful, even a small mismatch in column names or missing values will cause the model to fail. Column names are case-sensitive and types must match. If you're unsure, print df_future.dtypes and compare it against what the model expects. Once your data is ready, load the model and make predictions. Example:
model = mlflow.pyfunc.load_model("models:/your_model_name/1")
forecast = model.predict(df_future)
print(forecast[['ds', 'yhat', 'yhat_lower', 'yhat_upper']])
To debug faster, test on a small set of known-good data, Make sure your environment matches the one used during training. Check the requirements.txt in your MLflow run and install exact versions. If all else fails, or if the AutoML artifacts are unclear, training Prophet manually gives you full control and avoids surprises.
If this helped solve the issue, please consider marking it “Accept as Solution” and giving a ‘Kudos’ so others with similar queries may find it more easily. If not, please share the details, always happy to help.
Thank you.The issue seems to be that the featurization that is happening during the AutoML just isn't transparent so I can't figure out what I need to add. I get an error like this, but there is no easy way to call the features from the expirement. KeyError: "['ds_sin2', 'ds_hour_cos', 'ds_hour_sin', 'ds_cos1', 'ds_month_cos', 'ds_second_cos', 'ds_dayofyear_sin', 'ds_month_sin', 'ds_quarter_sin', 'ds_sin4', 'ds_quarter_cos', 'ds_dayofweek_cos', 'ds_sin3', 'ds_cos2', 'ds_cos4', 'ds_dayofyear_cos', 'ds_second_sin', 'ds_sin1', 'ds_minute_cos', 'ds_cos3', 'ds_minute_sin', 'ds_dayofweek_sin', 'ds'] not in index"
Hi Anonymous , Thank you for reaching out to the Microsoft Community Forum.
You're getting a KeyError because Prophet in Microsoft Fabric’s AutoML expects a set of engineered features that were automatically created during training but aren’t regenerated at prediction time. Since Fabric doesn’t expose the exact featurization logic, you’ll need to manually recreate those features based on the error message and what the model expects.
First, use MLflow to inspect the required input columns for your model. Example:
import mlflow
model = mlflow.pyfunc.load_model("models:/your_model_name/1")
required_cols = model.metadata.get_input_schema().input_names()
print("Required columns:", required_cols)
Then, generate the missing features based on your ds datetime column. Example:
import pandas as pd
import numpy as np
df_future = pd.read_csv("your_inference_data.csv")
df_future['ds'] = pd.to_datetime(df_future['ds'])
df_future['hour'] = df_future['ds'].dt.hour
df_future['dayofweek'] = df_future['ds'].dt.dayofweek
df_future['dayofyear'] = df_future['ds'].dt.dayofyear
df_future['month'] = df_future['ds'].dt.month
df_future['quarter'] = df_future['ds'].dt.quarter
df_future['minute'] = df_future['ds'].dt.minute
df_future['second'] = df_future['ds'].dt.second
def add_cyclic_features(df, col, max_val, prefix):
df[f'{prefix}_sin'] = np.sin(2 * np.pi * df[col] / max_val)
df[f'{prefix}_cos'] = np.cos(2 * np.pi * df[col] / max_val)
return df
df_future = add_cyclic_features(df_future, 'hour', 24, 'ds_hour')
df_future = add_cyclic_features(df_future, 'dayofweek', 7, 'ds_dayofweek')
df_future = add_cyclic_features(df_future, 'dayofyear', 365, 'ds_dayofyear')
df_future = add_cyclic_features(df_future, 'month', 12, 'ds_month')
df_future = add_cyclic_features(df_future, 'quarter', 4, 'ds_quarter')
df_future = add_cyclic_features(df_future, 'minute', 60, 'ds_minute')
df_future = add_cyclic_features(df_future, 'second', 60, 'ds_second')
for i in range(1, 5):
df_future[f'ds_sin{i}'] = np.sin(i * 2 * np.pi * df_future['ds'].dt.dayofyear / 365)
df_future[f'ds_cos{i}'] = np.cos(i * 2 * np.pi * df_future['ds'].dt.dayofyear / 365)
Validate that all required columns are present before predicting. Example:
missing_cols = [col for col in required_cols if col not in df_future.columns]
if missing_cols:
raise ValueError(f"Missing columns: {missing_cols}")
Then make predictions. Example:
forecast = model.predict(df_future[required_cols])
print(forecast[['ds', 'yhat', 'yhat_lower', 'yhat_upper']])
If this helped solve the issue, please consider marking it “Accept as Solution” and giving a ‘Kudos’ so others with similar queries may find it more easily. If not, please share the details, always happy to help.
Thank you.
Hi Anonymous ,
I wanted to follow up and see if you’ve had a chance to review the information provided here.
If any of the responses helped solve your issue, please consider marking it "Accept as Solution" and giving it a 'Kudos' to help others easily find it.
Let me know if you have any further questions!Hi Anonymous ,
I hope the information shared was helpful. If you have any additional questions or would like to explore the topic further, feel free to reach out. If any of the responses resolved your issue, please mark it "Accept as solution" and give it a 'Kudos' to support other members in the community.
Thank you!Hi Anonymous , Just checking in—were you able to resolve the issue?
If one of the replies helped, please consider marking it as "Accept as Solution" and giving a 'Kudos'. Doing so can assist other community members in finding answers more quickly.
Thank you!