Forum Discussion

kirah2128's avatar
kirah2128
Resolver I
2 years ago

How to do the Machine Model application in Fabric?

Dear All,

 

is there any link or tutorial to do machine learning and deploy the models to production using fabric?

 

1. My datasets are stored in Lakehouse

2. We Trained the model and save it in Fabric 

But when applying the script to load the new data coming the lakehouse. It says the 

RuntimeError: Unable to get model info: Registered Model with name=component_classification_v3 not found

 

 

 

 

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, udf
from pyspark.sql.types import ArrayType, IntegerType
import pandas as pd
from transformers import BertTokenizer
import tensorflow as tf
import mlflow
from synapse.ml.predict import MLFlowTransformer

# Initialize Spark session
spark = SparkSession.builder.getOrCreate()

# Load data from your Spark SQL environment or DataFrame
df = spark.sql("SELECT removal_reasons, reliability_tracked FROM lakehouse1.part_removal")

# Initialize the tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Tokenization function
def tokenize_text(text):
    tokens = tokenizer(text, padding="max_length", truncation=True, max_length=128, return_tensors="np")
    return tokens['input_ids'][0].tolist(), tokens['attention_mask'][0].tolist()

# Register UDF for tokenization
@udf(ArrayType(IntegerType()))
def udf_tokenize_input_ids(text):
    return tokenize_text(text)[0]

@udf(ArrayType(IntegerType()))
def udf_tokenize_attention_mask(text):
    return tokenize_text(text)[1]

# Apply the UDF to add tokenized columns
df = df.withColumn("input_ids", udf_tokenize_input_ids(col("removal_reasons")))
df = df.withColumn("attention_mask", udf_tokenize_attention_mask(col("removal_reasons")))

# Ensure the DataFrame has the correct format
df = df.select("input_ids", "attention_mask", "reliability_tracked")

# Configure MLflow
mlflow.set_tracking_uri("UNKNOWN")  # Your MLflow tracking URI here
mlflow.set_experiment("Notebook-1")  # Your experiment name here

# Load the model
model = MLFlowTransformer(
    inputCols=["input_ids", "attention_mask"],  # Your input columns here
    outputCol="predictions",  # Your new column name here
    modelName="component_classification_v3",  # Your model name here
    modelVersion=1  # Your model version here
)

# Transform the data using the model
predicted_df = model.transform(df)

# Write the predictions to Delta Lake
predicted_df.write.format('delta').mode("overwrite").save("predicted_amos_part_removal")  # Your output table filepath here

 

 

 

 

Regards,

King

 

7 Replies

    • kirah2128's avatar
      kirah2128
      Resolver I

      Hi, the link is not helpful.


      What I want to achieve is to deploy now the model. There's a wizard option but that's not gonna work if the input is converted to other data types. in my Case its TensorFlow data type.; I attached the picture below for your ref. 

       

       

      • Anonymous's avatar
        Anonymous
        Not applicable

        Hi kirah2128 ,

        Can you please check your input data and also the version configuration from your end once?

        Do let me know incase of further queries.