Forum Discussion

kirah2128's avatar
kirah2128
Resolver I
2 years ago

How to do the Machine Model application in Fabric?

Dear All,   is there any link or tutorial to do machine learning and deploy the models to production using fabric?   1. My datasets are stored in Lakehouse 2. We Trained the model and save it in...
data engineering
data science
  • kirah2128's avatar
    kirah2128
    Resolver I
    2 years ago

    Hi, the link is not helpful.


    What I want to achieve is to deploy now the model. There's a wizard option but that's not gonna work if the input is converted to other data types. in my Case its TensorFlow data type.; I attached the picture below for your ref. 

     

     

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi kirah2128 ,

      Can you please check your input data and also the version configuration from your end once?

      Do let me know incase of further queries.

      • kirah2128's avatar
        kirah2128
        Resolver I
        2 years ago

        This is the model

         

         

        here is how I supply the ML with the new inputs from lakehouse source.

        # Initialize Spark session
spark = SparkSession.builder.getOrCreate()

# Load data from your Spark SQL environment or DataFrame
df = spark.sql("SELECT removal_reasons, reliability_tracked FROM lakehouse1.part_removal")

# Initialize the tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Tokenization function
def tokenize_text(text):
    tokens = tokenizer(text, padding="max_length", truncation=True, max_length=128, return_tensors="np")
    return tokens['input_ids'][0].tolist(), tokens['attention_mask'][0].tolist()

# Register UDF for tokenization
@udf(ArrayType(IntegerType()))
def udf_tokenize_input_ids(text):
    return tokenize_text(text)[0]

@udf(ArrayType(IntegerType()))
def udf_tokenize_attention_mask(text):
    return tokenize_text(text)[1]

# Apply the UDF to add tokenized columns
df = df.withColumn("input_ids", udf_tokenize_input_ids(col("removal_reasons")))
df = df.withColumn("attention_mask", udf_tokenize_attention_mask(col("removal_reasons")))

# Ensure the DataFrame has the correct format
df = df.select("input_ids", "attention_mask", "reliability_tracked")