Power BI vs Qlik Sense
Power BI vs Qlik In this session, we will cover the following topics: Perspectives: Developers End Users Domains: Data ETL Dataset Storage Data Modeling Calculated Expressions Visuals Real Samples to Compare RBC Group Prebuilt Solutions PBI Report - Qlik Sense Application What Qlik Sense Solutions we couldn't realize in Power BI What Power BI Features we have not in Qlik Sense Visual's Performance Agenda: Introductions (5 mins) Presentation (55 mins) Q&A session (15 mins) About Artsem Zharkou: Artsem Zharkou is a BI Consultant in Azerbaijan, certified Microsoft Power BI expert & Qlik Sense Business Analyst, and working with IT Tools for over 22 years. Skills: Development of loading scripts, associative models and user interfaces; data quality management; development of technical documentation and video instructions, user training and coaching; business advisory, business process modeling, change management; pmbok\agile project management. Project’s industries: Retail and wholesale, manufacturing, fast food, pharmaceutics, machinery maintenance and engineering. Experience in developing and implementing applications: Sales analysis, Sales receipts analysis, Stock (inventories) analysis, Financial statement analysis, Statistical sales forecasting, Cost analysis, Sales teams performance analysis. Experience in loading data from systems: 1С, MS SQL, BelMag, R-Keeper, Firebird, IIKO, DBF, REST-api, SharePoint, Google spreadsheets, Google analytics, Yandex metrika, RPM Global, SSAS, SSIS. Additional experience: BI-applications for agent sales analysis, supply planning, financial analysis, customer loyalty in retail (Qlik Sense). ERP system development (MS Visual FoxPro). VBA scripts development for process automation (MS Excel). Website’s support (html, php, css). Implementation of PMIS (MS Project, ELMA). Implementation of CRM (AmoCRM, Bitrix24). Stock quotes flow analysis scripts development (QUIK\LUA). BI-application for sales real time monitoring (MS Power BI). Maintenance of BI-report of the medical representatives’ activities vs sales for pharmaceutical company (MS Power BI Report Server). Strong managerial and business-analysis skills. This event is online. You may need to install Zoom: https://zoom.us/download Space is limited so reserve your seat. Contact with Artsem: YouTube: https://www.youtube.com/channel/UCOHjtmlemSpGENtGfqOfHYg WWW: https://www.rbcgrp.com/ LinkedIn: https://www.linkedin.com/in/tomsharkbel/ Members participating in Meetup Live events must be respectful in interactions with people and in content posted on the platform.
Predicting Loan Eligibility with Data Science in Microsoft Fabric
Join us for an insightful session on ‘Predicting Loan Eligibility with Data Science in Microsoft Fabric’. This event will delve into the application of data science and machine learning techniques in the financial sector. We will explore how predictive models can be built to assess loan eligibility, thereby aiding financial institutions in their decision-making process. The session will specifically focus on the use of Microsoft Fabric, a powerful tool for data analysis and machine learning. Participants will gain knowledge on how to leverage this platform for financial analysis and risk assessment. Whether you’re a data science enthusiast, a financial analyst, or someone interested in the intersection of these fields, this event will equip you with valuable insights and practical skills. Don’t miss out on this opportunity to learn, network, and enhance your understanding of data science in the financial industry.Azure ML Model Training with Responsible AI
Take your model interpretability and transparency to the next level with the Responsible AI Dashboard in Azure Machine Learning. The Responsible AI Dashboard can help you and others to gain trust in your Machine Learning model and provide transparent decision-making support. Jeremy Schieblon will show how to train a ML model and set up the Dashboard in the Azure Machine Learning Studio and dive into it (the Dashboard) live to walk through its model debugging, interpretability, and decision-making capabilities. You will learn how to: Train an ML model in Azure Machine Learning End-to-End Set up the Responsible AI Dashboard for your model Use the Dashboard for Model Debugging Utilize the Dashboard to make transparent decisionsFabric Deep Dive: Traditional ML & Purview for Data Governance
Fabric Deep Dive: Traditional ML & Purview for Data Governance Join us for our next Microsoft Fabric Meetup event on 12.06.2025. Topics include Traditional ML capabilities in and with Fabric, as well as Data Governance with Purview and Fabric. Don't miss this opportunity to enhance your Microsoft Fabric expertise, exchange ideas, and network with the community. We look forward to seeing you there! Agenda 18:00 – Arrive, grab a drink, and find your spot 18:15 – Traditional ML in the Age of AI: Exploring Data Science Capabilities in Fabric and Beyond In an era dominated by generative and agentic AI, traditional machine learning remains a vital part of the data science toolkit. This session explores how Microsoft Fabric empowers data professionals with tools for data preparation, model training, AutoML, and scoring. We’ll highlight how Fabric simplifies ML workflows while also showing where Azure Machine Learning adds value for more complex scenarios. Learn how to combine both platforms to build scalable, efficient ML solutions. Discover how Fabric can serve as a robust foundation for operationalizing models in Azure ML. Speaker: Maria Mohr - Data & AI Engineer @ Obungi 19:00 – Short Break 19:15 – Microsoft Fabric and Microsoft Purview - How is that connected? In this session we will give you an introduction into how those services need to be set up. Afterwards we try to tackle tasks like data governance topics like data asset management, access management and data lineage in Microsoft Fabric via Microsoft Purview. What tasks can be fulfilled with built-in functionality and where would customers need to add clue code? Lets find out together! Speaker: Christian Wolff - Cloud Solution Architect @ Microsoft 20:00 – Discussion & Drinks Engage with our speakers and participants in discussions while enjoying some
Enrich Power BI reports with machine learning in Microsoft Fabric
Many organizations want to go beyond descriptive Power BI reports and start answering forward‑looking questions with machine learning—identifying emerging trends, at-risk accounts, and where to focus effort to maximize impact. In practice, this is hard. Adding machine learning to Power BI often means moving data out of semantic models, rebuilding logic, managing separate storage and security, and assembling custom pipelines. As teams cross these boundaries, definitions drift, refreshes break, and insights rarely return to the tools business users rely on. Microsoft Fabric takes a different approach. Fabric unifies data engineering, data science, and business intelligence on a single, unified platform built on OneLake. Instead of breaking apart your BI stack, Fabric lets you extend it—reusing governed semantic models, training models where the data lives, and operationalize predictions in the same flow without duplicating logic or rebuilding pipelines. This post shows an end‑to‑end pattern for enriching a Power BI report with machine learning in Fabric. We start with a governed semantic model, train a churn-prediction model, and operationalize predictions with batch and real‑time scoring. The result is predictive insight aligned with business logic, reliably refreshed, and surfaced directly in Power BI for monitoring, sharing, and action. Scenario: Predicting bank customer churn Across industries, teams use Power BI to understand what has already happened. Dashboards show trends, highlight performance, and keep organizations aligned around a shared view of the business. But leaders are asking new questions—not just what happened, but what is likely next and how outcomes might change if they act. They want insights that help teams prioritize, intervene earlier, and focus effort where it matters. This is why many organizations look to enrich Power BI reports with machine learning. This challenge is especially common in financial services. Consider a bank that uses Power BI to track customer activity, balances, and service usage. Historical analysis shows that around 20% of customers churn, with churn tied to factors such as customer tenure, product usage, service interactions, and balance changes. Figure 1 Bank customer churn overview in Power BI At this point, descriptive reporting is no longer enough. The real questions become: Which customers are most likely to churn next? How confident are those predictions? Can churn risk update automatically as customer data changes? Can accounts teams monitor high‑risk customers and act early? Microsoft Fabric makes it possible to answer those questions with a fully integrated, end-to-end workflow. Architecture Figure 2 Architecture to enrich Power BI reports with ML Model First, let's look briefly at the architecture: Semantic model defines business logic and metrics. Semantic Link to access that model directly from notebooks. Fabric ML experiment to train and evaluate a churn prediction model. Batch scoring applies to the model at scale and saves predictions to OneLake. Real-time scoring endpoints support low-latency inference where needed. Dataflow Gen2 calls the real-time scoring endpoint to enrich data during ingestion. Power BI Report visualizing churn risk and driving action. All components live natively in Fabric and share the same security, governance, and storage layer. Prerequisites Before starting, you should have: A Fabric-enabled workspace. A “Bank Customer Churn Analysis” semantic model published in the workspace. You can find a sample semantic model and notebooks in this repository. Step 1: Explore the semantic model with Semantic Link A common challenge in analytics organizations is logic duplication. Business concepts like active customer, average monthly balance, or churned user are first defined in DAX for reporting, then redefined again in SQL or Spark for pipelines, and rebuilt once more in Python during feature engineering. Over time, those definitions drift, numbers diverge, and teams spend more time reconciling metrics than using them. In Fabric, the Power BI semantic model becomes the contract. Instead of exporting data and reimplementing logic, Semantic Link lets you query the semantic model directly from a Fabric notebook. Measures, filters, and calculated columns behave exactly as they do in Power BI. There is no separate authentication step, and no risk of logic drifting over time—the Fabric security context applies automatically. With one line of Python code, you can explore all semantic models, tables, and columns. # Semantic Link uses your Fabric identity — no separate credential handling. import sempy.fabric as fabric # Discover semantic models (datasets) you can access df_datasets = fabric.list_datasets() display(df_datasets) # Inspect tables in a specific semantic model dataset_name = "Bank Customer Churn Analysis" df_tables = fabric.list_tables(dataset_name) display(df_tables) # Inspect columns and metadata (useful for feature planning) df_columns = fabric.list_columns(dataset_name) display(df_columns) Figure 3 Semantic model discovery When you query a Power BI semantic model through Semantic Link, you work with the same logic that already powers your reports, not a parallel copy for data science. The result is a FabricDataFrame, which looks and feels like a pandas DataFrame but carries much more value. It preserves semantic context—such as measures, calculations, and lineage—directly from the semantic model into the data science environment. Because it subclasses pandas, you can use it with familiar libraries and workflows, while also calling Power BI measures directly from your notebook. # Measures are the business logic. Keeping them centralized prevents drift. df_measures = fabric.list_measures(dataset_name) display(df_measures) Figure 4 Inspect measure definitions # Pull governed data from the semantic model. # This keeps your ML features aligned with what Power BI shows. df_raw = fabric.read_table(dataset=dataset_name, table="churn") display(df_raw) Figure 5 Pull Semantic model table as a SparkDataFrame This lets you build features, explore data, and train models on the same business definitions your stakeholders trust in Power BI—without rewriting logic, managing credentials, or worrying about drift over time. Once the data is available in the notebook, you can perform exploratory analysis with familiar tools such as pandas, seaborn, or matplotlib. Any additional features you derive build on centrally governed definitions instead of replacing them. That foundation makes everything that follows safe, repeatable, and trustworthy. import seaborn as sns sns.set_theme(style="whitegrid", palette="tab10", rc = {'figure.figsize':(9,6)}) import matplotlib.pyplot as plt import matplotlib.ticker as mticker from matplotlib import rc, rcParams import numpy as np import pandas as pd import itertools Drop the duplicated rows, rows with missing data and drop the columns that you do not need. def clean_data(df): # Drop rows with missing data across all columns df = df.dropna() # Drop duplicate rows in columns: 'CustomerId', 'RowNumber' df = df.drop_duplicates(subset=['CustomerId', 'RowNumber']) # Drop columns: 'RowNumber', 'Surname' return df df_clean = clean_data(df_raw.copy()) df_clean.head() Figure 6 Cleaned training dataset Data exploration Display some summaries and visualizations of the cleaned data. Use this code to determine categorical, numerical, and target attributes. # Determine the dependent (target) attribute dependent_variable_name = "Exited" print(dependent_variable_name) # Determine the categorical attributes categorical_variables = [col for col in df_clean.columns if col in "O" or df_clean[col].nunique() <=5 and col not in "Exited"] print(categorical_variables) # Determine the numerical attributes numeric_variables = [col for col in df_clean.columns if df_clean[col].dtype != "object" and df_clean[col].nunique() >5 and col not in "CustomerId"] print(numeric_variables) Show the five-number summary (the minimum score, first quartile, median, third quartile, the maximum score) for the numerical attributes, using box plots. df_num_cols = df_raw[numeric_variables] sns.set(font_scale = 0.7) fig, axes = plt.subplots(nrows = 2, ncols = 3, gridspec_kw = dict(hspace=0.3), figsize = (17,8)) fig.tight_layout() for ax,col in zip(axes.flatten(), df_num_cols.columns): sns.boxplot(x = df_num_cols[col], color='green', ax = ax) fig.delaxes(axes[1,2]) Figure 7 Five-number summary for numeric attributes Show the distribution of exited versus non-exited customers across the categorical attributes. df_raw['Exited'] = df_raw['Exited'].astype(str) attr_list = ['Geography', 'Gender', 'HasCrCard', 'IsActiveMember', 'NumOfProducts', 'Tenure'] df_raw['Exited'] = df_raw['Exited'].astype(str) fig, axarr = plt.subplots(2, 3, figsize=(15, 4)) for ind, item in enumerate (attr_list): print(ind, item) sns.countplot(x = item, hue = 'Exited', data = df_raw, ax = axarr[ind%2][ind//2]) fig.subplots_adjust(hspace=0.7) Figure 8 Distribution of exited versus non-exited customers across the categorical attributes Show the frequency distribution of numerical attributes using histogram. columns = df_num_cols.columns[: len(df_num_cols.columns)] fig = plt.figure() fig.set_size_inches(18, 8) length = len(columns) for i,j in itertools.zip_longest(columns, range(length)): plt.subplot((length // 2), 3, j+1) plt.subplots_adjust(wspace = 0.2, hspace = 0.5) df_num_cols[i].hist(bins = 20, edgecolor = 'black') plt.title(i) plt.show() Figure 9 Frequency distribution of numerical attributes The exploratory analysis highlights a few clear patterns. Most customers are based in France, while Spain—despite having fewer customers—shows the lowest churn rate compared to France and Germany. Product adoption is limited, with very few customers using more than two bank products. Customer activity stands out as a strong signal: inactive customers are significantly more likely to churn. In contrast, gender and tenure length show little influence on whether a customer decides to close their account. Feature engineering Now let us perform feature engineering to generate new attributes based on current attributes: df_clean['Tenure'] = df_clean['Tenure'].astype(int) df_clean["NewTenure"] = df_clean["Tenure"]/df_clean["Age"] df_clean["NewCreditsScore"] = pd.qcut(df_clean['CreditScore'], 6, labels = [1, 2, 3, 4, 5, 6]) df_clean["NewAgeScore"] = pd.qcut(df_clean['Age'], 8, labels = [1, 2, 3, 4, 5, 6, 7, 8]) df_clean["NewBalanceScore"] = pd.qcut(df_clean['Balance'].rank(method="first"), 5, labels = [1, 2, 3, 4, 5]) df_clean["NewEstSalaryScore"] = pd.qcut(df_clean['EstimatedSalary'], 10, labels = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]) Perform one-hot encoding for geography and gender columns: import pandas as pd def clean_data(df_clean): # One-hot encode columns: 'Geography', 'Gender' df_clean = pd.get_dummies(df_clean, columns=['Geography', 'Gender']) return df_clean df_clean_1 = clean_data(df_clean.copy()) df_clean_1.head() Create a delta table for the cleaned data to get ready for model training. table_name = "df_clean" # Create Spark DataFrame from pandas sparkDF=spark.createDataFrame(df_clean_1) sparkDF.write.mode("overwrite").option("overwriteSchema", "true").format("delta").save(f"Tables/{table_name}") print(f"Spark dataframe saved to delta table: {table_name}") Step 2: Build a machine learning model in Fabric With governed data in hand, the focus shifts from alignment to prediction: learning which customers are most likely to leave. Model training To predict churn, we train a LightGBM classifier, a strong fit for large tabular datasets like customer records. LightGBM captures non‑linear patterns with minimal tuning and integrates natively with Fabric’s MLflow experience, making experiments easy to track and models easy to operationalize. Because churn data is naturally imbalanced, we apply SMOTE during training to ensure the model learns equally from both churned and retained customers. This combination keeps the notebook focused on insight and outcomes—building a reliable model that can be registered once and reused consistently across batch and real‑time scoring—rather than on managing infrastructure or custom training code. # Install imblearn for SMOTE using pip %pip install imblearn Prior to training any machine learning model, you need to load the delta table from the Lakehouse to read the cleaned data you created in the previous step. import pandas as pd SEED = 12345 df_clean = spark.read.format("delta").load("Tables/df_clean").toPandas() Now let us generate experiments for tracking and set the experiment. import mlflow # Setup experiment name EXPERIMENT_NAME = "bank-churn-experiment" mlflow.set_experiment(EXPERIMENT_NAME) Import the required libraries for model training from sklearn.model_selection import train_test_split from lightgbm import LGBMClassifier from sklearn.metrics import accuracy_score, f1_score, precision_score, confusion_matrix, recall_score, roc_auc_score, classification_report Use the train_test_split function from scikit-learn to split the data into training, validation, and test sets. #Split the dataset to 60%, 20%, 20% for training, validation, and test datasets y = df_clean["Exited"] X = df_clean.drop("Exited",axis=1) #Train-Test Separation X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=SEED) #Train-Validation Separation X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.25, random_state=SEED) And save the test data to a delta table # Save the test data to a delta table table_name = "df_test" df_test=spark.createDataFrame(X_test) df_test.write.mode("overwrite").option("overwriteSchema", "true").format("delta").save(f"Tables/{table_name}") print(f"Spark test DataFrame saved to delta table: {table_name}") The data exploration in Step 1 showed that of 10,000 customers, only 2,037 customers (around 20%) left the bank, indicating a highly imbalanced dataset. With so few examples of the minority class, a model struggles to learn the decision boundary. SMOTE is the most widely used approach for synthesizing new minority-class samples. Apply SMOTE to the training data to synthesize new samples for the minority class. from collections import Counter from imblearn.over_sampling import SMOTE sm = SMOTE(random_state=1234) X_res, y_res = sm.fit_resample(X_train, y_train) new_train = pd.concat([X_res, y_res], axis=1) Train the model using LightGBM and register the trained model as a ML model artifact. At the time of this writing, Fabric Real-Time Scoring Endpoint did not support tensor-based models, so we created a non-tensor-based output schema for the model. import mlflow.pyfunc import pandas as pd import numpy as np class Predictor(mlflow.pyfunc.PythonModel): def __init__(self, base_model): self.base_model = base_modeldef predict(self, context, model_input: pd.DataFrame) -> pd.DataFrame: # prediction pred = self.base_model.predict(model_input) pred = np.asarray(pred, dtype=np.int64) # probability (positive class) proba = self.base_model.predict_proba(model_input) proba = np.asarray(proba[:, 1], dtype=np.float64) return pd.DataFrame({ "prediction": pred, "probability": proba }) import mlflow from mlflow.models.signature import infer_signature import numpy as np import pandas as pd from lightgbm import LGBMClassifier # Define the LightGMB model lgbm_sm_model = LGBMClassifier( learning_rate=0.07, max_delta_step=2, n_estimators=100, max_depth=10, eval_metric="logloss", objective='binary', random_state=42 ) with mlflow.start_run(run_name="lgbm_sm_non_tensor") as run: lgbm_non_tensor_sm_run_id = run.info.run_id # Train on balanced data lgbm_sm_model.fit(X_res, y_res.ravel()) # Validation predictions (convert to pandas for non-tensor signature) y_pred = lgbm_sm_model.predict(X_val) # Optionally include probability for positive class as a tabular output example for signature y_proba = lgbm_sm_model.predict_proba(X_val)[:, 1] y_proba_series = pd.Series(y_proba.astype(float), name="probability") # Build output example for signature (TWO cols) pred_ex = np.asarray(lgbm_sm_model.predict(X_val), dtype=np.int64) proba_ex = np.asarray(lgbm_sm_model.predict_proba(X_val)[:, 1], dtype=np.float64) y_example = pd.DataFrame({"prediction": pred_ex, "probability": proba_ex}) signature = infer_signature(X_val, y_example) # Compute metrics accuracy = accuracy_score(y_val, y_pred) cr_lgbm_sm = classification_report(y_val, y_pred) cm_lgbm_sm = confusion_matrix(y_val, y_pred) roc_auc_lgbm_sm = roc_auc_score(y_res, lgbm_sm_model.predict_proba(X_res)[:, 1]) # Log metrics mlflow.log_metric("val_accuracy", accuracy) mlflow.log_metric("train_roc_auc", roc_auc_lgbm_sm) # Classification report mlflow.log_text(cr_lgbm_sm, "classification_report.txt") # Confusion matrix cm_df = pd.DataFrame( cm_lgbm_sm, index=["Actual_0", "Actual_1"], columns=["Pred_0", "Pred_1"]) mlflow.log_table(cm_df, "confusion_matrix.json") # Log model mlflow.pyfunc.log_model( artifact_path="model", python_model=Predictor(lgbm_sm_model), signature=signature, input_example=X_val.head(5), registered_model_name="lgbm_sm_non_tensor" ) View the machine learning experiment The experiment runs are automatically saved in the experiment artifact that can be found from the workspace. You can select the link from the cell output and open the experiment view. Parameters, metrics, and artifacts are all captured in the experiment, making every run reproducible and auditable. Figure 10 Fabric machine learning experiment view Validate the model performance Once machine learning model training is complete, you can assess its performance. Open the saved experiment, load the machine learning models, and evaluate them on the validation dataset. import mlflow.pyfunc # Fetch the model load_model_lgbm1_sm = mlflow.pyfunc.load_model(f"runs:/{lgbm_non_tensor_sm_run_id}/model") # Assess the performance of the loaded model on validation dataset ypred_lgbm1_sm_v1 = load_model_lgbm1_sm.predict(X_val) Next, you will develop a script to plot the confusion matrix to evaluate the accuracy of the classification using the validation dataset. import seaborn as sns sns.set_theme(style="whitegrid", palette="tab10", rc = {'figure.figsize':(9,6)}) import matplotlib.pyplot as plt import matplotlib.ticker as mticker from matplotlib import rc, rcParams import numpy as np import itertools def plot_confusion_matrix(cm, classes, normalize=False, title='Confusion matrix', cmap=plt.cm.Blues): print(cm) plt.figure(figsize=(4,4)) plt.rcParams.update({'font.size': 10}) plt.imshow(cm, interpolation='nearest', cmap=cmap) plt.title(title) plt.colorbar() tick_marks = np.arange(len(classes)) plt.xticks(tick_marks, classes, rotation=45, color="blue") plt.yticks(tick_marks, classes, color="blue") fmt = '.2f' if normalize else 'd' thresh = cm.max() / 2. for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])): plt.text(j, i, format(cm[i, j], fmt), horizontalalignment="center", color="red" if cm[i, j] > thresh else "black") plt.tight_layout() plt.ylabel('True label') plt.xlabel('Predicted label') cfm = confusion_matrix(y_val, y_pred=ypred_lgbm1_sm_v1["prediction"]) plot_confusion_matrix(cfm, classes=['Non Churn','Churn'], title='LightGBM-non-tensor') tn, fp, fn, tp = cfm.ravel() Figure 11 Confusion matrix for the churn prediction model Once performance is validated, the model is registered in the MLflow Model Registry. Moving forward, the model becomes a managed Fabric asset with a stable name and version, ready to be consumed by downstream pipelines. Step 3: Batch scoring on active customers Once the model is registered, the fastest way to make it usable is to score data at Spark scale and persist the results in OneLake—in our case, scoring active customers to identify who is likely to churn next. Fabric’s PREDICT capability is built for this: you point to a named model version in the Fabric registry and run inference where the data lives. That model name and version keeps scoring repeatable, letting you rerun the job with the same model and upgrade only when ready. Start by loading the latest snapshot of active customers from OneLake and applying the same feature-engineering logic to the active users. # Load the active users active_users = spark.read.format("delta").load("Tables/active") display(active_users) from pyspark.sql import functions as F from pyspark.sql.window import Window # Start with Spark DataFrame df = active_users.drop("RowNumber", "Surname") # Tenure as integer df = df.withColumn("Tenure", F.col("Tenure").cast("int")) # NewTenure = Tenure / Age df = df.withColumn("NewTenure", F.col("Tenure") / F.col("Age")) # CreditScore qcut into 6 bins credit_window = Window.orderBy("CreditScore") df = df.withColumn("CreditIdx", F.row_number().over(credit_window)) credit_count = df.count() df = df.withColumn("NewCreditsScore", F.ceil(F.lit(6) * F.col("CreditIdx") / F.lit(credit_count))) # Age qcut into 8 bins age_window = Window.orderBy("Age") df = df.withColumn("AgeIdx", F.row_number().over(age_window)) age_count = df.count() df = df.withColumn("NewAgeScore", F.ceil(F.lit(8) * F.col("AgeIdx") / F.lit(age_count))) # Balance qcut with rank(method="first") into 5 bins balance_window = Window.orderBy("Balance") df = df.withColumn("BalanceRank", F.row_number().over(balance_window)) balance_count = df.count() df = df.withColumn("NewBalanceScore", F.ceil(F.lit(5) * F.col("BalanceRank") / F.lit(balance_count))) # EstimatedSalary qcut into 10 bins salary_window = Window.orderBy("EstimatedSalary") df = df.withColumn("SalaryIdx", F.row_number().over(salary_window)) salary_count = df.count() df = df.withColumn("NewEstSalaryScore", F.ceil(F.lit(10) * F.col("SalaryIdx") / F.lit(salary_count))) # Cleanup temporary columns df_clean = df.drop("CreditIdx","AgeIdx","BalanceRank","SalaryIdx") from pyspark.sql import functions as F def clean_data_spark(df): df = ( df # One-hot Geography .withColumn("Geography_France", F.when(F.col("Geography") == "France", 1).otherwise(0)) .withColumn("Geography_Germany", F.when(F.col("Geography") == "Germany", 1).otherwise(0)) .withColumn("Geography_Spain", F.when(F.col("Geography") == "Spain", 1).otherwise(0)) # One-hot Gender .withColumn("Gender_Male", F.when(F.col("Gender") == "Male", 1).otherwise(0)) .withColumn("Gender_Female", F.when(F.col("Gender") == "Female", 1).otherwise(0)) # Drop original categorical columns .drop("Geography", "Gender") ) return df df_clean_1 = clean_data_spark(df_clean) display(df_clean_1) In practice, you have a few equally supported ways to invoke PREDICT, depending on how you like to work. If you prefer a notebook‑first workflow, the Transformer API gives you a clean wrapper around your registered MLflow model. You create an MLFlowTransformer, tell it which input columns to use, and choose an output column name (for example, predictions). from synapse.ml.predict import MLFlowTransformer model = MLFlowTransformer( inputCols=list(df_clean_1.columns), outputCol='predictions', modelName='lgbm_sm_non_tensor', modelVersion=1 ) From there, scoring is a standard Spark transform: you call transform() and you get back a DataFrame that includes predicted labels and probabilities—ready to save as a Delta table. The payoff is simple: the scoring step feels like any other Spark pipeline step, and it scales with your data. import pandas from pyspark.sql.functions import col predictions = model.transform(df_clean_1) predictions = predictions.select( "*", col("predictions.prediction").alias("prediction"), col("predictions.probability").alias("probability") ).drop("predictions") ay(predictions) If your team prefers more declarative pipelines—or you want scoring that reads like a query—Fabric also supports calling PREDICT through Spark SQL. That is useful when you want inference to sit alongside familiar SQLstyle transformations, or when you are handing off a scoring step to engineers who live in SQL. style transformations, or from pyspark.ml.feature import SQLTransformer # Substitute "model_name", "model_version", and "features" below with values for your own model name, model version, and feature columns model_name = 'lgbm_sm_non_tensor' model_version = 1 features = df_clean_1.columns sqlt = SQLTransformer().setStatement( f"SELECT PREDICT('{model_name}/{model_version}', {','.join(features)}) as predictions FROM __THIS__") # Substitute "X_test" below with your own test dataset display(sqlt.transform(df_clean_1)) And if you need maximum flexibility inside PySpark, you can convert the registered model into a PySpark UDF and call it like any other function over columns. This is handy when scoring needs to be embedded inside more customized DataFrame logic, while keeping the model reference centralized in the registry. from pyspark.sql.functions import col, pandas_udf, udf, lit # Substitute "model" and "features" below with values for your own model name and feature columns my_udf = model.to_udf() features = df_clean_1.columns display(df_clean_1.withColumn("predictions", my_udf(*[col(f) for f in features]))) Whatever you choose, the goal is the same: write predictions back to OneLake as a durable table. At that point, churn risk stops being “ML output in a notebook” and becomes governed data you can join, track over time, and surface directly in Power BI. # Save predictions to lakehouse to be used for generating a Power BI report table_name = "active_predictions" predictions.write.format('delta').mode("overwrite").save(f"Tables/{table_name}") print(f"Spark DataFrame saved to delta table: {table_name}") Step 4: Enable real-time scoring Batch scoring supports many reporting scenarios, but some decisions need predictions as soon as data changes—for example, a customer updating their profile, opening a support ticket, or onboarding. Waiting for the next scheduled refresh is often too late. Microsoft Fabric addresses this with real-time scoring endpoints. The same model used for batch scoring can be activated as an online endpoint without rewriting or redeploying it. ‑time scoring endpoints‑deploying An endpoint loads a specific model version once and serves low latency predictions through a secure REST interface. It uses the same feature schema and versioning model as batch inference, keeping training and serving cleanly separated. You can iterate on models without disrupting live scoring and always know which version is serving predictions.‑latency predictions through a secure REST interface. It uses the same feature schema and versioning model as batch inference, Fabric makes this simple. Endpoints are a built-in property of ML models. You can enable them directly from the Fabric UI with a low code experience, preview sample predictions immediately, and then integrate them wherever needed—inside Fabric or from external systems—through a scalable, managed API.‑in property of ML models. You can enable them directly from the Fabric UI with a low‑code experience, preview sample predictions immediately, and then integrate them wherever Figure 12 Activate real-time scoring endpoint for a ML model version Step 5: Enrich data during ingestion with Dataflow Gen2 Realtime scoring becomes most powerful when it is embedded in the data ingestion pipeline. Instead of scoring after the fact, predictions are generated as data arrives or changes.‑time scoring becomes most powerful when it is embedded With Dataflow Gen2, you can call the real-time scoring endpoint as part of a transformation step. New or updated customer records are sent to the model, enriched with churn prediction and probability, and written back to OneLake as part of the ingestion flow.‑time scoring endpoint as part of a transformation step. New or updated customer records are sent to the model, enriched with churn prediction and probability, and written back to OneLake as part of the ingestion flow. This keeps your data continuously ML enriched without custom orchestration or brittle glue code. Downstream consumers—reports, dashboards, and analysts—always see the latest predictions, and the scoring logic stays centralized around the managed model, not scattered across pipelines. Below is the sample M code for calling the ML endpoint from Dataflow Gen2. Use a service principal as credentials, and ensure it has the necessary permissions on the workspace hosting your ML model endpoint. To maintain workflow reliability, turn off the auto-sleep feature for endpoints. (input as table) as table => let // Feature engineering features = fnFeatureEngineering(input), // Build JSON request body jsonBody = "{""inputs"":" & Text.FromBinary( Json.FromValue( List.Transform( Table.ToRows(features), each _ ) ) ) & "}", // OAuth parameters TenantId = "<Your Tenant ID>", ClientId = "<Your Client ID>", ClientSecret = "<Your Client Secret>", Scope = "https://api.fabric.microsoft.com/.default", TokenUrl = "https://login.microsoftonline.com/" & TenantId & "/oauth2/v2.0/token", FormEncode = (fields as record) as text => Text.Combine( List.Transform( Record.FieldNames(fields), (k) => Uri.EscapeDataString(k) & "=" & Uri.EscapeDataString(Text.From(Record.Field(fields, k))) ), "&" ), TokenBodyText = FormEncode([ client_id = ClientId, client_secret = ClientSecret, grant_type = "client_credentials", scope = Scope ]), TokenResponse = Json.Document( Web.Contents( TokenUrl, [ Headers = [#"Content-Type" = "application/x-www-form-urlencoded"], Content = Text.ToBinary(TokenBodyText) ] ) ), AccessToken = TokenResponse[access_token], // Call ML model endpoint ScoreUrl = "<Your ML model endpoint url>", response = Json.Document( Web.Contents( ScoreUrl, [ Headers = [ #"Content-Type" = "application/json", #"Authorization" = "Bearer " & AccessToken ], Content = Text.ToBinary(jsonBody) ] ) ), // Parse orientation=values output PredictionsRaw = response[predictions], PredictionValues = List.Transform(PredictionsRaw, each _{0}), ProbabilityValues = List.Transform(PredictionsRaw, each _{1}), // Append columns to table Result = Table.FromColumns( Table.ToColumns(features) & { PredictionValues, ProbabilityValues }, Table.ColumnNames(features) & { "prediction", "probability" } ) in Result Step 6: Visualize and act in Power BI Once predictions are stored in OneLake, they behave like any other curated dataset in Fabric. You bring them into the semantic model, join them with customer dimensions, and shape them using the same governance patterns you already apply to business data. You can enrich your Power BI report by adding a dedicated view that highlights customers with a high risk of churn. Figure 13 Customers with high churn risk From the Power BI user’s perspective, churn risk becomes a clear, refreshed metric that can be filtered, segmented, and tracked over time. Account teams can monitor high-risk customers, prioritize outreach, and act before customers leave. Under the hood, this experience is powered by a full machine learning pipeline. On the surface, it feels like standard BI—reliable, explainable, and easy to consume. Closing thoughts The outcome of this pattern isn’t just a trained model—it’s machine learning insight that behaves like data. Predictions are written to OneLake as governed; refreshable tables aligned to the Power BI semantic model and surfaced directly in reports. As data changes, insights update. As definitions evolve, logic stays consistent. For business teams, churn risk appears as a trusted metric—filterable, explainable, and ready to act on. Microsoft Fabric makes this practical by keeping data, models, and BI on the same platform. There’s no need to export data, rebuild business logic, or maintain custom scoring pipelines. Teams can move from descriptive reporting to predictive insight without adding operational complexity. The same approach extends naturally to forecasting, anomaly detection, and propensity modeling. The principle stays the same: start from governed semantics, train models in place, and operationalize predictions where the business already works. In Fabric, machine learning isn’t adjacent to BI—it’s embedded in the analytics experience. Try it out Explore the sample repository and try it out. If you’re already using Power BI and Fabric, you can apply this pattern directly to your own data—starting with a single semantic model, one report, and one prediction, then expand as value becomes clear. Check out Fabric data science document for more information. If you’d like to learn more or discuss how this fits your organization, you can book time with author.
Report Server functionalities and licenses
Hi there, I would have the following questions regarding Power BI Report Server: 1. Would a Power BI Premium license make sense with PBI RS? I mean, the insights for advance AI would be available for the user besides being in RS? Or does only Power BI Pro make sense in a PBI RS environment just to publish the reports on the server? 2. Regarding the following link (see column "Power BI Report Server") it says "No" to "R scripts and visual" and "Python scripts and visuals". Besides that, on the Power BI Desktop edition for RS, there are available both icons, whereas on the "Notes" column, it says "you can't save Power BI reports with Python scripts or visuals to Power BI Report Server". Does this mean that you can use it meanwhile you are in the Desktop but the functionality does not exist on the server? 3. I would be interested in machine learning/AI capabilities in PB RS. I would be grateful if anyone can provide any advice. Thanks in advance and hope my questions make senseCost-Benefit Analysis using machine learning in Power BI
Imagine you are the owner of a local restaurant and you’ve had modest success with a marketing coupon campaign in the past. You sent out some coupons through the local mail and ended up noticing an uptick in business over the next few weeks. Now you’re wondering which customers you should target next. Additionally, you’d like to know if there’s anything you can do to analyze the coupon campaign with the goal of maximizing benefits and revenue while minimizing costs. Cost-benefit analysis, further referred to as CBA, is one such analytical tool. Now when building binary prediction models using AutoML in Power BI, the resulting report allows you to use CBA with your data.
Building a Regression Model with zero code in PowerBI
I love using PowerBI for analysis! Recently PowerBI added support for AutoML (Supervised Automated machine Learning) and I wanted to demonstrate the new ML capabilities with an end to end example. In this article, I use AutoML to create and apply a regression model. Regression is a machine learning technique used for understanding the relationship between input and output variables (input is what is known in your data and output is generally the effect of input data and is sometimes unknown). Regression technique gives us more insight into data by making it easier to see relation between dependent and independent variables. These are the five steps I followed in this article to create the Regression model: Refer this tutorial for a detailed walk through of each of these steps, and this tutorial for ones specific to regression models. Creating a dataflow with the input data I created a dataflow using the House Sales prediction dataset from Kaggle. It is also available, at the following link: house sales prediction for purposes of this article. It contains sale prices, number of bedrooms, square footage, number of floors, latitude, longitude etc. of houses in King County (which includes Seattle) sold between May 2014 and May 2015. The Price attribute indicating the price of the house is the numeric field used as label for the Regression model. The dataset was split into two entities HousePrices2014 and HousePrices2015 based on the year attribute. Note that though most of the attribute data types in this dataset are numeric, these may be interpreted as strings by Power Query. So, most of these fields, including the label, were converted to numeric. Training a machine learning model I chose the HousePrices2014 as the entity to apply ML models. Power BI analyzed the Price field and suggested Regression as the type of machine learning model that can be created to predict that field. It took around an impressive 7 minutes to train 12 algorithms on 14 thousand data points. Model validation report After the training refresh completed, I checked out the training report. Key Influencers shows that Latitude, Sqft_living, Grade are the top predictors for the model. To verify this, I plotted a ft_living (square footage of the home) and price. The plot does indeed show a relationship indicating that that higher square feet living indicates higher prices. As shown in the sqft_living breakdown above, 2.7k to 2.9k sqfeet houses has the highest share of range 1 priced houses (72k to 1M). Likewise, a scatterplot between grade (overall grade given to the housing unit, based on King County grading system) and price shows houses with higher grades does have higher prices. The model is 88% performant, which is a pretty great value for this dataset. As you can see in the chart below, most of the test samples lie in the diagonal showing the low error in prediction. The Training Details page shows the performance of the 25 different models, how features were extracted from the inputs, and the hyperparameters for the final model used. AutoML chose an ensemble, Pre fitted Soft Voting Regressor as the best model for this dataset. Applying the model As I was satisfied with the training results, I proceeded to apply the model to the HousePrices2015 entity. A HousePrices2015 enriched RegressionModel was created, which includes the predicted output from the model. Overall, it took only around 2 minutes to score nearly 7 thousand rows! Applying a Regression model added two columns with the Regression outcome, and the top record-specific influencers for each prediction. The HousePrices2015 enriched Regression entity with explanation, predicted prices is shown below along with the actual prices in PowerBI Desktop. The local explanations contains break down of how much each feature moved the specific instance away from the average price of houses. For categorical features, it shows how much price would have changed for different values of the categories. In the highlighted example, you can see that the house price was predicted to be $554174. The explanation says that, "condition", caused the price to increase by 44k from the average price of houses, and that the "latitude" caused the price to fall down by 26k. I established a connection to the dataflow from PBI Desktop, to incorporate these predictions from the model in PBI reports so that I can visualize these results. I plotted a heat map of predicted prices using latitude, longitude. Prices were represented by a red (100% gradient stop) through green (50% gradient stop), to yellow(0% gradient stop) gradient. We can see that, for example, Downtown Seattle and Downtown Bellevue have red regions as expected, owing to high prices. I tried a scikit-learn Random Forest on this same dataset. It took 1 min using 3-fold cross validation, searching across 30 different combinations, using all available cores to give an R2 score of 0.85. Summary Even though AutoML took longer, I am impressed to have gotten a better R2 score of 0.88 by trying not just Random Forest but 25 different models, in addition to getting instance-level explanations. Moreover, AutoML automatically does preprocessing like dropping features like ID with no useful information and generating additional features for datetime like month, year etc. On the other hand, in scikit-learn one must encode string or date-time like features or drop it. Overall, I’m very excited about AutoML, because I did not have to learn the intricacies of the ML models nor worry about the implementation and still get the same results. I have done these in Python and R and it took me considerably longer to code up and setup the hyperparameters, whereas in AutoML I am just hitting a bunch of buttons and voila I am done! Given how much easier it is to get started with lesser effort, I encourage business analysts and even data scientists/analysts to try out AutoML!
