We are exploring the possibility of transitioning from Synapse and ADF to a fully Fabric-based solution. This will be a gradual process, and one of the steps during this transition will involve transferring data from Fabric Data Lake to Synapse.
Currently, I have a Dataflow Gen 2 that successfully writes data to a Data Lake in Fabric from Microsoft D365 CRM. While this is functioning well, I've set up a pipeline to copy the data from the Data Lake to Synapse, but I keep encountering the error message below.
I've also created a separate pipeline using the same authentication to execute a stored procedure in Synapse, which works perfectly. The user I’m using for authentication has write access in Synapse for the time being.
I've attempted to use Service Principals as well, but they haven't been successful either, despite being utilized for data ingestion into Synapse through Databricks.
There’s limited documentation available, and mainly from Synapse into Fabric... so any assistance would be greatly appreciated.
ErrorCode=UserErrorSqlDWCopyCommandError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=SQL DW Copy Command operation failed with error '110802;An internal DMS error occurred that caused this operation to fail. Details: Exception: Microsoft.SqlServer.DataWarehouse.DataMovement.Common.ExternalAccess.HdfsAccessException, Message: Error occurred while accessing HDFS external file[...][0]: Java exception raised on call to HdfsBridge_CreateRecordReader_V2. Java exception message:
HdfsBridge::CreateRecordReader - Unexpected error encountered creating the record reader: HdfsBridgeAbfsRestOperationException: Operation failed: "Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.", 403, HEAD, [REDACTED URL],...
