Forum Discussion
Can't load data from SQL Server into Lakehouse. Can't add a data destination.
Super Usercristinaojer , For some customers it worked with the March 2024 gateway, but for some, it worked only with the April 2024 Gateway(Data Pipelines). But with April 2024 many of our dataflows are failing with false error.
Microsoft EmployeeHi cristinaojer
Thanks for using Microsoft Fabric Community. amitchandak thank you for sharing the details.
As we have two releases in the month of April.
April 2024 update (3000.218.8)
April 2024 update (3000.218.7)
Request you to install the latest version of on-prem gateway and try to establish the connection, this might help you.
If the issue still persists, please do let us know we will try to help.
Hope this helps.
Thank you.
- v-cboorla-msft
Hi cristinaojer
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.Thanks.
- v-cboorla-msft
Hi cristinaojer
When using Microsoft Fabric Dataflow Gen2 with an on-premises data gateway, you might encounter issues with the dataflow refresh process. The underlying problem occurs when the gateway is unable to connect to the dataflow staging Lakehouse in order to read the data before copying it to the desired data destination. This issue can occur regardless of the type of data destination being used.
The firewall rules on the gateway server and/or customer's proxy servers need to be updated to allow outbound traffic from the gateway server to the following:
- Protocol: TCP
- Endpoints: *.datawarehouse.pbidedicated.windows.net, *.datawarehouse.fabric.microsoft.com, *.dfs.fabric.microsoft.com
- Port: 1433
Please refer to the documentation for more details : Set new firewall rules on server running the gateway
For similar issue please refer : Error when trying to create a dataflow gen 2. - Microsoft Fabric Community
I hope this information helps.
Thank you.
- v-cboorla-msft
Hi cristinaojer
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.Thanks.
Experiencing the sames issues. Dataflow Gen2 successfully loads onprem SQL tables to Fabric lakehouse tables, but a copy activity in a Fabric pipeline dosn't work. I also setup an ADF pipeline to copy to the lakehouse and get the same results.
The copy activity is creating the table in the lakehouse as expected. When looking at the underlying files of the table, a folder called '_delta_log' is there with files small json files within it. Looks like each of these files is being created after a timeout occurs? Here's the contents of the json:
{"protocol":{"minReaderVersion":1,"minWriterVersion":2,"readerFeatures":null,"writerFeatures":null}}{"metaData":{"id":"8db4335b-4ae9-4b5e-8262-3e5096d44d39","name":"delta","description":"","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\": <columns names removed>]}","partitionColumns":[],"createdTime":1714817850126,"configuration":{}}}{"commitInfo":{"operation":"Write","engineInfo":"Microsoft.DI.Delta","additionalCommitInfo":null}}No records are being inserted into the lakehouse. Data preview works in the pipeline.I also tried landing the files as lakehouse files instead of tables, but still did not work. Seems like the issue is with writing data to the lakehouse from a pipeline copy activity (either Fabfic or ADF).
