Forum Discussion

AnnOminous's avatar
AnnOminous
Microsoft Employee
1 year ago
Solved

Can't write data to Fabric DB - MashupException.Error

I'm trying to write data from a Dataflow (not Gen2) to a SQL DB in Fabric. The query is set to refresh incrementally, but when I try to publish/refresh the query, I get the following error message: ...
data engineering
  • BhaveshPatel's avatar
    BhaveshPatel
    1 year ago

    Hi AnnOminous 

     

    You need to use Notebooks only ( Advanced User ). Also You need to understand Data Lake and Delta Lake with Apache Spark. To answer the question,This is writing data back to Fabric SQL ( INSERT ). This is a script.

    # Create DataFrame to insert into the Azure/Fabric SQL  table ( Writing Data to Azure SQL / Fabric SQL Table )
     
    df1 = spark.createDataFrame([('WL2456', 'Luke1293', 'Skywalker','Null','Null',-3,'Unknown','Unknown','Null')], ['DeviceID', 'DeviceName', 'DeviceDesc','OperatorName','OperatorDesc','FacilityID','FacilityName','FacilityDesc','DeviceSerialNo'])

    # Insert the rows into the Azure SQL / Fabric SQL table
    df1.write \
        .option('user', user) \
       .option('password', pswd) \
       .jdbc('jdbc:sqlserver://' + sqlserver + ':' + port + ';database=' + database, 'dbo.Devices', mode = 'append' )
       
       
    df1.show()



    Cheers
    Bhavesh Patel



     

BhaveshPatel's avatar
BhaveshPatel
Super User
1 year ago

This is only possible in Notebooks and you have to use Python. Python is the universal language in  OLTP (Online Transaction Processing ) + OLAP ( Online Analytical Processing) these days. 

 

This is how it works: ( Linux Operating System )
Python -- > Data Lake ( Apache Spark ) --> Data Lakehouse ( Delta Lake ) 

 

Currently, This is not supported ( WRITE method ) using T-SQL ( INSERT ). 

 

You should use Incremental Refresh via Dataflow Gen 2 or alternatively you should use python notebooks. you have to understand Data Lake and why it started in about 2010 along with Matei Zaharia and why it has gone through Data Lakehouse. In a limited space, I can not write all of this.