Fabric Spark connector for SQL databases (Azure SQL databases, Azure SQL Managed Instances, Fabric SQL databases and SQL Server in Azure VM) in the Fabric Spark runtime is now available. This connector enables Spark developers and data scientists to access and work with data from SQL database engines using a simplified Spark API. The connector will be included as a default library within the Fabric Runtime, eliminating the need for separate installation.
Operation Support
The Spark connector for Microsoft Fabric Spark Runtime is a high-performance library that enables you to read from and write to SQL databases. The connector offers the following capabilities:- Use Spark to perform large write and read operations on the following SQL products: Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM, Fabric SQL databases.
- The connector comes preinstalled within the Fabric runtime, which eliminates the need for separate installation.
- While you're accessing a table or a view, the connector upholds security models defined at the SQL engine level. These models include object-level security (OLS), row-level security (RLS), and column-level security (CLS).
Language Support
We are also introducing PySpark support for this connector, in addition to Scala. This means that you no longer need to use a workaround to utilize this connector in PySpark, as it is now available as a native capability.Authentication Support
This connector supports multiple authentication methods which you can choose based on your need and setup.To learn more about the Spark connector for SQL databases, please refer to the Spark connector for SQL databases documentation.