Forum Discussion
How to do the Machine Model application in Fabric?
Dear All,
is there any link or tutorial to do machine learning and deploy the models to production using fabric?
1. My datasets are stored in Lakehouse
2. We Trained the model and save it in Fabric
But when applying the script to load the new data coming the lakehouse. It says the
RuntimeError: Unable to get model info: Registered Model with name=component_classification_v3 not found
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, udf
from pyspark.sql.types import ArrayType, IntegerType
import pandas as pd
from transformers import BertTokenizer
import tensorflow as tf
import mlflow
from synapse.ml.predict import MLFlowTransformer
# Initialize Spark session
spark = SparkSession.builder.getOrCreate()
# Load data from your Spark SQL environment or DataFrame
df = spark.sql("SELECT removal_reasons, reliability_tracked FROM lakehouse1.part_removal")
# Initialize the tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
# Tokenization function
def tokenize_text(text):
tokens = tokenizer(text, padding="max_length", truncation=True, max_length=128, return_tensors="np")
return tokens['input_ids'][0].tolist(), tokens['attention_mask'][0].tolist()
# Register UDF for tokenization
@udf(ArrayType(IntegerType()))
def udf_tokenize_input_ids(text):
return tokenize_text(text)[0]
@udf(ArrayType(IntegerType()))
def udf_tokenize_attention_mask(text):
return tokenize_text(text)[1]
# Apply the UDF to add tokenized columns
df = df.withColumn("input_ids", udf_tokenize_input_ids(col("removal_reasons")))
df = df.withColumn("attention_mask", udf_tokenize_attention_mask(col("removal_reasons")))
# Ensure the DataFrame has the correct format
df = df.select("input_ids", "attention_mask", "reliability_tracked")
# Configure MLflow
mlflow.set_tracking_uri("UNKNOWN") # Your MLflow tracking URI here
mlflow.set_experiment("Notebook-1") # Your experiment name here
# Load the model
model = MLFlowTransformer(
inputCols=["input_ids", "attention_mask"], # Your input columns here
outputCol="predictions", # Your new column name here
modelName="component_classification_v3", # Your model name here
modelVersion=1 # Your model version here
)
# Transform the data using the model
predicted_df = model.transform(df)
# Write the predictions to Delta Lake
predicted_df.write.format('delta').mode("overwrite").save("predicted_amos_part_removal") # Your output table filepath here
Regards,
King
7 Replies
- AnonymousNot applicable
Hi kirah2128 ,
Thanks for using Fabric Community.
Did you got any chance to look into this doc - Machine learning model - Microsoft Fabric | Microsoft Learn
Hope this is helpful. Do let me know incase of further queries.- kirah2128Resolver I
Hi, the link is not helpful.
What I want to achieve is to deploy now the model. There's a wizard option but that's not gonna work if the input is converted to other data types. in my Case its TensorFlow data type.; I attached the picture below for your ref.- AnonymousNot applicable
Hi kirah2128 ,
Can you please check your input data and also the version configuration from your end once?
Do let me know incase of further queries.