rss.livelink.threads@place:occasion

FabCon 2026 Barcelona: Beyond the Announcements

michaelolafusi — Tue, 20 Oct 2026 00:23:39 GMT

Data Engineers in Toronto October 2026 Semimonthly Meeting

Topic: FabCon 2026 Barcelona - Beyond the Announcements

Abstract:
What was it like to attend Microsoft's flagship Fabric conference in Barcelona? Which announcements generated the most excitement? And more importantly, which ones are likely to matter in real-world customer projects?

Join me for an informal recap of FabCon 2026 as I share photos, conference highlights, conversations with Microsoft engineers and community members, and a curated selection of Fabric announcements that caught my attention.

Speaker: Alexander Turlov, Principal Consultant at Alta Computing

Speaker Profile:
Alexander Turlov is a professional consultant, specializing in cloud data architecture and Microsoft Fabric solutions. He helps organizations design scalable data platforms and integrate modern tools like GitHub Copilot and VS Code to accelerate development workflows.

Alexander is a certified Microsoft Fabric Data Engineer and an active community contributor, sharing practical insights through presentations and workshops on AI-assisted data engineering and analytics.

The meeting is over Microsoft Teams, and the joining link is https://vip.dataengineersintoronto.org/webinar

See you at the meeting!

Agentic Loops in the Data Stack: From Pipeline Failure to Auto-Remediation

michaelolafusi — Fri, 04 Sep 2026 06:01:27 GMT

Data Engineers in Toronto January 2027 Semimonthly Meeting

Topic: Agentic Loops in the Data Stack: From Pipeline Failure to Auto-Remediation

Abstract:
Every data engineer knows the 2 AM pipeline failure, the one nobody notices until Friday's report is wrong.

In this session, we break down five AI agents that are changing how data teams operate: from monitoring pipelines 24/7 and catching schema drift at ingestion, to closing the gap between a production failure and its root cause in minutes. We'll walk through real implementation patterns, including a baseline-learning monitoring agent and a tool-use driven incident response loop, and discuss what the shift to agentic data engineering actually means for the way teams are built and how engineers grow. Whether you're evaluating agents for your platform or already running them in production, you'll leave with concrete patterns you can apply immediately.

Speaker: Varun Joshi, Senior Data Engineer at AWS

Speaker Profile:
Highly motivated and results-oriented Data Engineer with 12+ years of experience in designing, building,and optimizing scalable data pipelines and architectures.Proven expertise in data warehousing, ETL/ELT processes, and cloud platforms. Passionate about leveraging Artificial Intelligence (AI) and Machine Learning (ML).

Designed and deployed AI-driven Data solutions, integrating LLM-powered coding assistants into Data Engineering to produce AI solutions for customers. Focused on leveraging LLMs and advanced engineering to build scalable, secure, and trustworthy platforms, resulting in significant efficiency gains, reduced on-call burden,and improved customer trust.

Driving AI adoption across teams to enhance productivity, streamline deployments, and improve end-user experience.

The meeting is over Microsoft Teams, and the joining link is https://vip.dataengineersintoronto.org/webinar

See you at the meeting!

City of Toronto Open Data Portal

michaelolafusi — Fri, 04 Sep 2026 05:59:53 GMT

Data Engineers in Toronto December 2026 Semimonthly Meeting

Topic: City of Toronto Open Data Portal

Abstract:
The City of Toronto launched its first Open Data Portal in the fall of 2009 to meet growing demand for open data. Since then, the open data movement has gained increasing momentum within Toronto’s digital and civic spaces.

Come hear the city staff that manage the open data portal (https://open.toronto.ca) pull back the curtain and discuss how it works behind the scenes and where it contributes to the city.

Speaker: Mac Nichols; Supervisor, City of Toronto Open Data

Speaker Profile:
Mac is the supervisor of the city of Toronto's open data portal. He's worked on the portal since 2021, before which he worked in the private sector as a data engineer and analyst
The meeting is over Microsoft Teams, and the joining link is https://vip.dataengineersintoronto.org/webinar

See you at the meeting!

The Integration Blueprint: Choosing the Right Path in Microsoft Fabric

michaelolafusi — Fri, 04 Sep 2026 05:40:59 GMT

Data Engineers in Toronto November 2026 Semimonthly Meeting

Topic: The Integration Blueprint: Choosing the Right Path in Microsoft Fabric

Abstract:
In a world of multiple tools and endless data sources, how do you design the right integration strategy in Microsoft Fabric?

Join Javier and Gaston, two Microsoft MVPs in Data Platform for a live demo-driven session that breaks down the real-world patterns, use cases, and best practices for landing, transforming, and orchestrating data in Microsoft Fabric.

Whether you’re working with a Lakehouse, Data Warehouse, Eventhouse, or SQL Database, this session gives you the playbook to select and combine the right tools for the job.

We’ll walk through:
• Landing data using Data Pipelines, OneLake shortcuts, and external connections
• Transforming and cleaning data using Notebooks (Spark) for scale and flexibility
• Business logic and last-mile transformation using Dataflows Gen2
• How to orchestrate it all using Fabric Data Pipelines for repeatable, maintainable workflows
• When and why to land your data into Lakehouses, Data Warehouses, Eventhouses, or SQL endpoints.

You’ll walk away with:
• A clear understanding of which tools to use when across the Fabric integration stack
• A working knowledge of how to combine multiple integration techniques in one solution
• Architecture patterns and decision criteria for building scalable data landing zones
• Live demos you can replicate to accelerate your own Fabric journey

Speaker: Javier Villegas, IT Director & Microsoft MVP Data Platform

Speaker Profile:
Javier Villegas is IT Director (DBA & BI Services) at Mediterranean Shipping Company with 30 years of experience working with SQL Server, Including Azure SQL.
His specialization is Administration, Performance Tuning, High Availability and Disaster Recovery.
Javier is a Microsoft MVP Data Platform since 2016 and MCT
He is a frequent speaker in technical conferences and events such as SQL Saturdays, PASS Virtual Groups, PASS Marathons, 24 Hours SQL PASS , vOpen , GroupBy , Data Saturdays and Azure Global Bootcamps.
He is one of the group leaders of SQL Argentina Community and former Microsoft Azure Data Advisory Board Member.

The meeting is over Microsoft Teams, and the joining link is https://vip.dataengineersintoronto.org/webinar

See you at the meeting!

Deep Dive into Scalable, AI-Ready Spark in Microsoft Fabric

michaelolafusi — Fri, 04 Sep 2026 05:32:52 GMT

Data Engineers in Toronto November 2026 Semimonthly Meeting

Topic: Deep Dive into Scalable, AI-Ready Spark in Microsoft Fabric

Abstract:
In this session, we’ll take a practical deep dive into Spark-based data engineering in Microsoft Fabric. We’ll walk through proven techniques and repeatable patterns for solving common and advanced data engineering challenges, covering advanced Spark SQL functions, effective approaches to semi-structured data, and strategies for building reliable, maintainable pipelines. We will be focusing on Fabric-specific aspects of Spark Notebooks, including table optimization.

AI-related capabilities in Fabric Spark will also be covered, including built-in and integrated functions that support text processing, enrichment, classification, and feature engineering directly within Spark workloads, along with guidance on where these functions fit.

Speaker: Iurii Iurchenko, Sr Data Engineer at Life Line Screening

Speaker Profile:
Iuril is an Azure advocate, Fabcon speaker, Senior Data Engineer & Tech Lead with 13 years of deep, hands-on expertise in Data Analytics, Microsoft Fabric, and the Azure data ecosystem. 2x certified in Microsoft Fabric, with 6+ successfully delivered complex projects leveraging Microsoft Fabric. Two-time SQL Saturday speaker in San Diego, Top-1% ADPList mentor, former entrepreneur, and active OHDSI CDM contributor with 7 years in consulting and 6 years in product engineering. He brings
a highly practical, frontier-level skillset at the intersection of Fabric, AI, and healthcare analytics.

He is currently innovating at Life Line Screening, bringing the best of Microsoft Fabric to its modern data analytics platform.

The meeting is over Microsoft Teams, and the joining link is https://vip.dataengineersintoronto.org/webinar

See you at the meeting!

Introduction to the data build tool (dbt)

michaelolafusi — Fri, 04 Sep 2026 05:30:23 GMT

Data Engineers in Toronto October 2026 Semimonthly Meeting

Topic: Introduction to the data build tool (dbt)

Abstract:
Many people learn from the Jaffle Shop sample dbt project. However, the complexity of the project is quite simple.

What I really wanted is a project that goes over the following dbt concepts:
• Seed raw data tables
• Transform raw data using views
• Support multiple databases with macros
• Capture slowly changing dimensions type
• Incremental loading of large data
• Use ephemeral tables as a place holder
• Use dbt utils package to make a date dimension
• Create a dimensional model for reporting
• Save pre-canned queries in analyses directory
• Test data quality using property/schema files
• Output custom data quality exceptions to a table
• Create documentation based on model design

In the end, I created my own sample dbt project using the Chinook database schema as the foundation. I must thank Luis Rocha for the OLTP database design and sample data.

This open source project was built using an Fabric SQL Database as the target. However, it can be built on any supported database using the correct dbt adapter and ODBC driver. At the end of this presentation, a data engineer will be ready to model data for any given database or warehouse.

Speaker: John Miner, Senior Data Architect at Insight Digital Innovation

Speaker Profile:
John Miner is a Senior Data Architect at Insight Digital Innovation helping corporations solve their business needs with various data platform solutions.

He has over thirty years of data processing experience, and his architecture expertise encompasses all phases of the software project life cycle, including design, development, implementation, and maintenance of systems.

His credentials include undergraduate and graduate degrees in Computer Science from the University of Rhode Island. Also, he has earned certificates from Microsoft for Database Administration (MCDBA), System Administration (MCSA), Data Management & Analytics (MCSE) and Data Science (MPP).

John has been recognized with the Microsoft MVP award ten times for his outstanding contributions to the Data Platform community.

When he is not busy talking to local user groups or writing blog entries on new technology, he spends time with his wife and daughter enjoying outdoor activities. Some of John’s hobbies include wood working projects, crafting a good beer and playing a game of chess.

The meeting is over Microsoft Teams, and the joining link is https://vip.dataengineersintoronto.org/webinar

See you at the meeting!

Beyond the Prompt: Tuning Fabric Data Agents for Flawless Results

michaelolafusi — Mon, 31 Aug 2026 22:41:14 GMT

Data Engineers in Toronto August 2026 Semimonthly Meeting

Topic: Beyond the Prompt - Tuning Fabric Data Agents for Flawless Results

Abstract:
As organizations race to deploy AI Agents, it quickly becomes clear that standard prompt engineering isn't enough. To achieve production-grade accuracy, you have to go beyond the prompt and fundamentally change how you feed your data layers to the AI.

In Microsoft Fabric, your choice of data artifact directly dictates your agent's reasoning style and native language.

In this session, we will deep-dive into how Fabric Data Agents interact with four distinct data paradigms: Semantic Models, Ontology, Data Warehouses, and Lakehouses.

Moving past high-level theory, we will practically map the three pillars of data agent optimization across the data paradigms:
Agent Instructions (System Prompts)
Data Source Descriptions (Grounding)
Example Queries (Few-Shot Prompting)

You will walk away with an actionable blueprint showing you exactly how to write context rules, embed semantic metadata, and supply technical query examples to ensure your Fabric Data Agents deliver flawless, trusted results every time.

Speaker: Thimantha Vidanagamage, Data Engineer at OMERS, Canada

Speaker Profile:
Thimantha has over 8 years of experience in Data and AI. He currently works as a Senior Data Engineer at OMERS in Canada, where he plays an active role in the organization’s adoption of Microsoft Fabric. He holds multiple Microsoft certifications, including Microsoft Certified: AI Transformation Leader and Fabric Data/Analytics Engineer Associate. Thimantha is also a co-organizer of the Toronto Data Professionals Community.

The meeting is over Microsoft Teams, and the joining link is https://vip.dataengineersintoronto.org/webinar

See you at the meeting!

Get DP-700 Certified: The Fabric Data Engineer Overview

apturlov — Fri, 14 Aug 2026 21:11:59 GMT

Abstract: This session is a fast-paced tour of everything you need to know to earn the Microsoft Certified: Fabric Data Engineer (DP-700) credential. It's built for data engineers who are already working in Microsoft Fabric — or hands-on with its core tools — but haven't yet sat the exam. If you know the platform and want a clear, structured path to certification, this is your roadmap.

What attendees will learn

The certification landscape — where DP-700 fits, what the exam expects, and the data-loading patterns, architectures, and orchestration skills a Fabric Data Engineer is measured on.

Implementing and managing analytics solutions — Fabric workspaces and items, lifecycle management, security and governance, and orchestration, all anchored on OneLake (30–35% of the exam).

Ingesting and transforming data — batch and streaming, storage choices (Lakehouse vs. Warehouse vs. SQL Database), ingestion tools (Spark, Dataflows Gen2, Pipelines, Shortcuts, Mirroring), and Real-Time Intelligence (30–35% of the exam).

Monitoring and optimizing solutions — the Fabric Monitoring Hub, capacity metrics, and admin usage reports for performance, cost control, and scalability (30–35% of the exam).

Exam prep and next steps — hands-on practice tips, the DP-700 study guide, Microsoft Learn modules, and the practice assessment.

Streaming your database - Easier said than done?

michaelolafusi — Fri, 14 Aug 2026 21:12:33 GMT

Data Engineers in Toronto September 2026 Semimonthly Meeting

Topic: Streaming your database - Easier said than done?

Abstract:

Do you need to stream your database? So did I! Here’s what to expect.

In my journey as a contractor for a major cybersecurity company, I implemented a Change Data Capture (CDC) system that aimed to seamlessly stream client databases into a centralized cloud solution for analytics and enhanced features. However, after deploying the code, I quickly realized that the reality was far more complex than anticipated.

Despite using an off-the-shelf product, I encountered unexpected challenges related to scaling, data formats, integrations, and deployment—issues that took me by surprise and required significant adjustments.

Over the course of 18 months, my team and I navigated these hurdles to create a robust system designed for scalability and resilience against failures.

In this talk, I’ll share key lessons learned and practical tips for developers and architects embarking on similar projects. You'll gain insights into preparing for potential surprises and overcoming challenges in building a reliable CDC system.

Speaker: Sigal Shaharabani, Technical Group Leader & Couchbase Ambassador

Speaker Profile:

Sigal Shaharabani is a Technical Leader and a Group Leader at Tikal and a Couchbase ambassador, with a great passion for backend and data systems. She started her technological career way back in 1996, working for the government, a variety of corporations, and small start-ups. Her favorite programming language is Kotlin, and will love any excuse to talk about it. In her spare time she enjoys swimming and Israeli folk dancing.

Building Collaborative AI Agents Using AI Foundry Agent Service

michaelolafusi — Fri, 14 Aug 2026 21:12:36 GMT

Data Engineers in Toronto September 2026 Semimonthly Meeting

Topic: Building Collaborative AI Agents Using AI Foundry Agent Service

Abstract:

As enterprise AI adoption grows, many real-world scenarios can no longer be solved by a single conversational agent. Instead, they require multiple specialized agents that collaborate, share context, and delegate responsibilities intelligently. This session focuses on how to design and implement multi-agent architectures using Azure AI Foundry’s Agent Service.

We’ll start by breaking down when and why multi-agent systems make sense, then walk through a practical scenario where agents are assigned clear roles such as data retrieval, reasoning, and action execution. Using AI Foundry Agent Service, you’ll see how these agents are created, connected, and orchestrated to work together while maintaining context and control.

Rather than focusing on theory, the session emphasizes practical design patterns, architecture decisions, and implementation considerations, including agent boundaries, communication flows, and scaling agent-based solutions safely in enterprise environments.

By the end of the session, attendees will have a solid understanding of how to move from a single-agent approach to a collaborative, multi-agent model and how to apply these patterns using AI Foundry in their own solutions.

Key Takeaways:

- Understand when multi-agent architectures are the right choice

- Learn how to design role-based agents using AI Foundry Agent Service

- Explore real-world coordination and orchestration patterns

- Gain practical guidance for scaling and governing multi-agent systems

- Leave with a reusable architecture blueprint for enterprise AI agents

Speaker: Mitul Tailor, Snr. Data & AI Engineer

Speaker Profile:

am a Senior Data & AI Engineer at Long View Systems, specializing in Azure services, data engineering, and AI-driven solutions. With expertise in Azure Data Factory, Fabric, Copilot Studio, and data migration, I have successfully delivered scalable and efficient data solutions.

I hold a Post Graduate Certificate in Generative AI, a PG Diploma in ML & AI, and a Bachelor's in Computer Science. I am DP-600 & AI-102 certified, demonstrating my expertise in Microsoft Azure Enterprise Data Analyst solutions. Additionally, I have a strong understanding of Retrieval-Augmented Generation (RAG) AI and Azure AI Foundry, exploring how these technologies can be leveraged for intelligent automation and data-driven insights.

Recognized by clients for my clear and precise development work, I am passionate about automation, AI, and robotics. Outside of work, I love to travel and have a deep interest in cars.

Running GitHub Actions Offline: Debug CI/CD workflows Locally

michaelolafusi — Fri, 14 Aug 2026 21:12:38 GMT

Data Engineers in Toronto August 2026 Semimonthly Meeting

Topic: Running GitHub Actions Offline: Debug CI/CD workflows right from your local machine

Abstract:

Ever wanted to try out your GitHub Actions without waiting for the cloud to spin up the runners? Well, you can! Testing GitHub Actions locally lets you build, test, and debug your CI/CD workflows right from your machine. Through a tool such as act, you can simulate GitHub’s environment for your workflows locally using Docker. This means no more pushes of dozens of commits just to see if your syntax in the YAML file is wrong. You can make changes, execute, and validate your automation as many times as you want, even on a plane where there's no Wi-Fi. Just install Docker, install act, load your secrets locally, and run commands like act push to test your events in the workflows. That's faster, lightweight, and the greatest productivity booster. Taking GitHub Actions offline is not about isolation; it's about freedom and control over your automation pipeline.

Speaker: Steve Yonkeu, Software Engineer & Microsoft MVP

Speaker Profile:

Steve Yonkeu is a backend software engineer dedicated to delivering excellence in architecture, performance, and modularity. For over 5 years he has been leading teams and projects and open source communities worldwide. Steve is also the founder of Django Cameroon and Python Cameroon communities today. He is also an organizer at PyCon Africa and speaker at PyConUS.

Stop Guessing: Solve SQL Performance Problems with Query Store

michaelolafusi — Fri, 14 Aug 2026 21:12:40 GMT

Data Engineers in Toronto July 2026 Semimonthly Meeting

Topic: Stop Guessing: Solve SQL Performance Problems with Query Store

Abstract:

When query performance suddenly changes, finding the root cause isn't always easy. Execution plans can change over time, and the plan cache often provides only a limited view of what happened.

Query Store helps by capturing query history, execution plans, and runtime statistics, giving you the visibility needed to identify regressions, compare plans, and maintain consistent performance.

In this session, you'll learn how to use Query Store to monitor query behavior, investigate performance issues, manage execution plans, and leverage built-in tuning capabilities in SQL Server and Azure SQL Database.

Key topics:

Query Store fundamentals

Query and plan history analysis

Performance regression detection

Query Store reports and insights

Plan forcing and plan management

Automatic tuning and best practices

Through live demos, you'll see how Query Store makes performance troubleshooting faster, simpler, and more reliable.

Speaker: Deepthi Goguri, SQL Database Administrator

Speaker Profile:

Deepthi is a SQL Server Database Administrator with several years of experience in Administering SQL Servers. She is a Microsoft Data Platform MVP, Microsoft certified trainer and Microsoft certified professional with an Associate and Expert level Certification in Data Management and Analytics. Deepthi blogs for DBANuggets.com.

Deepthi is a Co-Organizer for Microsoft Data and AI South Florida user group, Data TGIF, Cloud Data Driven User Group, Future Data Driven Summit, Databash Conference and Data Platform Diversity, Equity, and Inclusion Virtual Group. She is also DEI Steering Committee member for PASS Data Community Summit. She is a Redgate Community Ambassador. Along with this, Deepthi loves arts and crafts. You can contact her on Twitter @dbanuggets.

Introduction to PySpark in Microsoft Fabric

michaelolafusi — Fri, 14 Aug 2026 21:12:42 GMT

Data Engineers in Toronto July 2026 Semimonthly Meeting

Topic: Introduction to PySpark in Microsoft Fabric

Abstract:

With all of the engineering features in Microsoft Fabric, which medium should you use to move and transform data? Low-code data flows and pipelines? Good old relational SQL? What about this newfangled PySpark everyone is buzzing about?

If the last option piques your curiosity and you haven't tried it, this is the session for you. I'll cover basic Python principles that will make even complicated Python easy to read. Then I will explain important topics to understand when managing your Spark environment. Finally, I'll showcase Fabric features and community content that can support your next steps in learning to implement PySpark in Fabric.

Speaker: Jared Kuehn, Data Engineer

Speaker Profile:

For over a decade, Jared has been a data engineering consultant implementing Microsoft products. For more than three decades, he has been honing his skills in theater and other performing arts. As a speaker, Jared marries these two disciplines together, creating dynamic presentations that add entertainment to education. This symbiotic relationship can improve presentation engagement, support attendees in knowledge retention, and foster a culture of passion for the data industry.

As a speaker, Jared has spoken at events such as:

-Fabcon Vegas (Microsoft Fabric Community Conference)

-DataCon Seattle (Microsoft Data Conference)

-PASS Summit

-SQL Saturdays, Multiple

-Microsoft Fabric Global Online Conference

-Future Data Driven Summit

-GroupBy Conference

-and more!

He also runs the YouTube channel DataBard, focused on making Data fun and teaching techniques along the way.

Technically-speaking, Jared has multiple Microsoft certifications in both on-prem and cloud technologies. He has extensive knowledge in:

-Microsoft Fabric

-SQL Query Development and Performance tuning

-Kimball Data Modeling

-ETL design patterns

-Azure SQL DB and Azure SQL MI

In his spare time, he continues to perform in community theater productions, as well as musically for his church. Upon request, he would be happy to assist with more theatrical portions of events, such as performing musical numbers.

See you at the meeting!

Data Mesh as the Foundation for AI/ML in Financial Services

michaelolafusi — Fri, 14 Aug 2026 21:12:43 GMT

Data Engineers in Toronto June 2026 Semimonthly Meeting

Topic: Data Mesh as the Foundation for AI/ML in Financial Services

Abstract:

Financial institutions want AI/ML at scale, but brittle data pipelines, silos, and compliance demands slow progress. This talk shows how a Data Mesh—domain-oriented ownership, data-as-product, self-serve platforms, and federated governance—becomes the foundation for reliable, reusable ML features and trustworthy models. We’ll map mesh principles to FS use cases—fraud detection, risk, personalization—and show patterns for feature stores, lineage, quality, and access controls that satisfy regulators while accelerating delivery. Attendees will get a pragmatic blueprint: where to start, how to sequence capabilities, metrics that prove value, and pitfalls to avoid on the road from pilots to production.

Speaker: Santosh Durgam, Data Engineering & Analytics Leader

Speaker Profile:

Santosh Durgam is a data engineering & analytics leader with 20+ years building governed, high-scale data platforms across retirement/401(k), broader financial services, and healthcare. He leads cross-functional teams that deliver production-grade data lakes, lineage-aware pipelines, and ML-enabled analytics on cloud—translating governance into measurable business outcomes.

Recent speaking includes SQL Saturday Minnesota 2025, where he presented “From Ingestion to Insights: Building Robust Data Pipelines in AWS” to an in-person community audience. He has also contributed to international research forums and science conferences, and is invited to speak at ICDPN-2025 (International Conference on Data Processing & Networking), engaging practitioners and scholars on data engineering, governance, and analytics at scale. Santosh actively publishes/curates work via Google Scholar and shares practical playbooks for data quality, metadata/lineage, and operating models that connect data platforms to financial decisioning.

Beyond delivery, Santosh serves the community as a peer reviewer of scholarly work on data/ML methodologies and as a judge/mentor for select industry and academic competitions, reinforcing peer validation and public recognition. He champions modern data culture—mentoring engineers and product leaders, and advocating automation (incl. AI agents) to elevate reliability, speed, and auditability in regulated environments. Santosh recently completed his Executive MBA, sharpening strategy and value-creation at the intersection of data, risk, and growth

The meeting is over Microsoft Teams, and the joining link is https://teams.microsoft.com/l/meetup-join/19%3ameeting_NzZkYWIyOTAtODk1MC00MjVmLWJlNjUtNTRiODZmODA2Zjdh%40thread.v2/0?context=%7b%22Tid%22%3a%22bd9727e8-f539-4c76-983c-6c30130c0bee%22%2c%22Oid%22%3a%229e8d5a64-e773-4ca2-90f6-9a266129171e%22%7d

See you at the meeting!

Building Scalable SCD Type 2 Pipelines in MS Fabric DW Using T-SQL

michaelolafusi — Fri, 14 Aug 2026 21:14:16 GMT

Data Engineers in Toronto June 2026 Semimonthly Meeting

Topic: Building Scalable SCD Type 2 Pipelines in MS Fabric DW Using T-SQL

Abstract:

Implementing Slowly Changing Dimension (Type 2) at scale is critical for maintaining historical accuracy in analytics, but doing so efficiently across billions of rows in Microsoft Fabric Data Warehouse requires leveraging its modern ingestion and optimization capabilities.

In this session, we’ll build a Fabric-optimized SCD Type 2 pipeline using pure T-SQL patterns. We’ll start by comparing ingestion strategies like OPENROWSET for schema-on-read exploration and COPY INTO for high-throughput, parallelized loading—and explain why COPY INTO is the preferred method for large-scale ingestion in Fabric Warehouses.

Next, we’ll implement incremental load logic without MERGE (since Fabric does not currently support the MERGE statement) by using UPDATE + INSERT patterns combined with hash-based change detection and filtered indexes for current-row lookups. We’ll also cover performance accelerators like batching, minimal logging, and distribution strategies to maximize query performance.

Finally, we’ll demonstrate a full end-to-end pipeline:

- Discover external Parquet/CSV files with OPENROWSET

- Ingest into Fabric Warehouse using COPY INTO

- Apply SCD Type 2 logic using merge-like T-SQL patterns for historical tracking

You’ll leave with a production-ready template and a Fabric-specific performance playbook for handling incremental loads at scale with minimal friction.

Speaker: Jean Joseph, Principal Data & AI Engineer Tech-Insight-Group LLC

Speaker Profile:

Jean Joseph is a seasoned consultant and senior technical trainer specializing in data engineering and artificial intelligence. With a strong background in database design, administration, and cutting-edge data technologies including machine learning and generative AI.

He helps organizations build secure, scalable solutions across both legacy systems and modern cloud platforms. Formerly recognized as a Microsoft MVP and senior technical trainer at Microsoft, Jean brings deep technical insight and a passion for teaching.

He’s also a dynamic speaker, mentor, and the founder of the Cloud Data Driven User Group and the Future Data Driven Summit, where he champions innovation and promotes responsible use of emerging tech within the data community.

See you at the meeting!

Data Modernization at Traditional Banks - From Legacy Mainframe to Cloud

michaelolafusi — Fri, 14 Aug 2026 21:14:18 GMT

(Date updated to 1 June 2026) Data Engineers in Toronto May 2026 Semimonthly Meeting

Topic: Data Modernization at Traditional Banks - From Legacy Mainframe to Cloud

Abstract:

As technology evolved from the age of industrialization to the age of information, and finally to this current age of AI, the average lifecycle of technology shrank from 10 to 6, and now it is 3 years. Therefore, the definition of “legacy” has shifted. Applications developed before 2010, especially those built on mainframes or outdated technologies, are increasingly seen as candidates for modernization.

Mainframe transformation has been a top priority in financial services because of its operational inefficiencies and high maintenance costs associated with mainframes when compared to modern distributed systems. Still, the majority of the core processing engines within financial services depend on Mainframe as the mission-critical business rules are embedded

Speaker: Vishal Sharma, VP - Software Engineering at Broadridge

Speaker Profile:

As a Software Engineering leader, I bring over 20 years of expertise in enterprise architecture and digital transformation. By leading cross-functional teams, I enable the delivery of secure, scalable solutions that optimize operations and enhance client engagement. My efforts focus on integrating innovative technologies to align with strategic business goals, resulting in measurable outcomes for stakeholders.

With a strong foundation in business architecture and fintech innovation, I excel in modernizing platforms to meet evolving industry demands. Equipped with certifications such as TOGAF, I prioritize seamless system integration, cost efficiency, and regulatory alignment to drive long-term value for organizations. Passionate about solving complex challenges, I work collaboratively to empower teams and achieve sustainable success.

The meeting is over Microsoft Teams, and the joining link is https://teams.microsoft.com/l/meetup-join/19:meeting_NzZkYWIyOTAtODk1MC00MjVmLWJlNjUtNTRiODZmODA2Zjdh@thread.v2/0?context={"Tid":"bd9727e8-f539-4c76-983c-6c30130c0bee","Oid":"9e8d5a64-e773-4ca2-90f6-9a266129171e"}

See you at the meeting!

Maximize SQL Server Performance with Read Committed Snapshot Isolation

michaelolafusi — Fri, 14 Aug 2026 21:14:20 GMT

Data Engineers in Toronto May 2026 Semimonthly Meeting

Topic: Maximize SQL Server Performance with Read Committed Snapshot Isolation

Abstract:

Are your read operations frequently blocked by write operations? Do you want to retrieve data without relying on NOLOCK?

Then it's time to switch to Read Committed Snapshot Isolation (RCSI).

In this session, I’ll explain what RCSI is, how it works, and how it differs from the default Read Committed Isolation level. I’ll demonstrate how version store comes into play with this isolation level and when it can overwhelm TempDB if not managed properly. Finally, I’ll discuss how to manage the implications of enabling RCSI.

By the end of this session, you'll have a clear understanding of RCSI and when to implement it in your environment for improved concurrency and performance.

Speaker: Haripriya Naidu, Lead SQL Server Database Administrator

Speaker Profile:

Haripriya Naidu has been working as a SQL Server Database Administrator for 11 years in Boston, MA. She is passionate about databases and enjoys diving into SQL Server engine internals and optimizing performance. She is an AWS Certified Solutions Associate and won "Rookie of the Year" Award at work in 2024 for outstanding performance and "Bookworm of the Year" Award at work in 2025 for continuously learning and sharing knowledge with the team.

She recently started her speaking journey with New Stars of Data and spoke at SQL Saturdays Boston, Albany and San Diego and PASS Summit 2024.

See you at the meeting!

All About Data Retention: Tactics for optimizing computing resources

michaelolafusi — Fri, 14 Aug 2026 21:14:33 GMT

Topic: All About Data Retention: Tactics for optimizing computing resources

Abstract:

How long should data be kept for? How should it be handled as it moves from actively used hot data to seldom used cool data? What tactics can be used to optimize computing resources when handling data of different types, from different systems, and of different ages.

Retention is a seldom-discussed topic, but one that can be used to reduce cloud hosting bills, improve performance, improve predictability, and ensure that an application remains in compliance with legal and contractual obligations,

This session dives into data retention from all angles and provides a wide variety of tactics for creating retention policies and implementing technical solutions.

Speaker: Edward Pollack, Data Architect & Microsoft Data Platform MVP

Speaker Profile:

Ed Pollack is a Microsoft Data Platform MVP with a passion for learning how Data Platforms work and sharing that knowledge with the community. His experiences in data architecture, database design, performance optimization, and data security are motivation for public speaking, writing, coding, and other community activities.

Ed has spoken at SQL Saturday events, SQL Bits, PASS Summit, EightKB, and many other regional and international events. Ed is the organizer of the Capital Area SQL Server Group and SQL Saturday Albany, as well as a co-organizer of SQL Saturday New York City, and Future Data Driven. He has published a number of books, including "Dynamic SQL: Applications, Performance, and Security in Microsoft SQL Server", "Expert Performance Indexing in Azure SQL and SQL Server 2022", and "Analytics Optimization with Columnstore Indexes in Microsoft SQL Server: Optimizing OLAP Workloads". Ed is also an active contributor of content to SimpleTalk.

In his free time, Ed enjoys video games, traveling, cooking exceptionally spicy foods, and hanging out with his amazing wife and sons.

SQL Supercharged: Azure, Fabric & Copilot in Action

apturlov — Fri, 14 Aug 2026 21:14:39 GMT

This session explores the latest innovations in Azure SQL and the brand new SQL on Fabric, with new possibilities for mirroring and intelligent analytics. We’ll also unpack what’s coming in the next version of SQL Server, from performance enhancements to security features. Finally, discover how Copilot can help us to interact and manage data to enhance productivity for data professionals.

Making AI Work: Aligning Data, Teams, and Business Goals

apturlov — Fri, 14 Aug 2026 21:14:59 GMT

AI adoption often stalls not due to a lack of tools, but because of what I call “frozen yoghurt syndrome”—an overwhelming number of choices that leads to confusion, misalignment, and stalled progress. This session offers a pragmatic framework for building a scalable data strategy that cuts through the noise and focuses on delivering business value.

We’ll explore four key pillars:

- Identify the Problem and Bigger Picture – Start with the business need. AI isn’t always the right answer; sometimes automation or simpler analytics are more effective.

- Design the Strategy Around Data, AI, and the Use Case Together – Align initiatives with clear business goals and define success metrics that matter to both technical and non-technical teams.

- Build the Foundation – Encourage cross-functional collaboration, select scalable tools, and document everything from data sources to model assumptions.

- Create a Positive Feedback Loop Between Tech and Non-Tech Teams – Foster a shared product language, enable self-service access to data, and build trust through transparency and iteration.

Attendees will leave with practical frameworks to ensure AI efforts are not just technically sound — but strategically aligned, collaborative, and sustainable.