Dataflows using variable library and SQL end point as source type failing when deployed.

[Nikhilesh_NG]

Hi,

Dataflows using variable library and lakehouse SQL end point as source type is not working during deployment from one workspace to another workspace use case.

I am facing an issue while using the Variable Library integration for Dataflow Gen2, as described in the Microsoft documentation:

https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-variable-library-integration

Our scenario involves deploying a Dataflow from one Fabric workspace to another. The connection is dynamically updated using the variable library (workspace ID and lakehouse name). This works correctly  

but when the source is the SQL Endpoint of a Lakehouse ( to pass SQL query instead of lakehouse table directly ) configured as Database , the deployment fails with the following behavior: 

• The dataflow throws an error stating “Data source credentials missing” or sometimes a module error.
• Even though the variable library is correctly configured and other sources are resolving properly, the SQL Endpoint connection does not update or authenticate during deployment.

Could anyone please help me understand whether:

1. The SQL Endpoint of a Lakehouse is supported for variable-based connection switching during deployment.
2. If so, please let me know if I might have missed any configuration point before deployment.

Thanks in Advance.

[v-echaithra]

Hi @Nikhilesh_NG ,

Thank you @nielsvdc  for your inputs.
I hope the above details help you fix the issue. If you still have any questions or need more help, feel free to reach out. We are always here to support you.

Regards,
Community Support Team.

[Nikhilesh_NG]

Hi v-echaithra,

At present, I am able to manage the issue with a temporary workaround. I trust that future updates from Fabric will address this limitation, enabling users to utilize the variable library in Dataflow Gen2 when connecting to the Lakehouse through the SQL endpoint.

[v-echaithra]

Hi @Nikhilesh_NG ,

Thank you for the update and for sharing the workaround that has been effective in your case.

If you need any further assistance, please don't hesitate to reach out. We're always here to help!

Best regards,
Chaithra E.

[nielsvdc]

Hi @Nikhilesh_NG

No, you did not miss any configuration. In the documentation under Considerations and limitations and you will the point “Variables can't alter connection information. Connections remain fixed to the authored resource path configurations.”. So deploying to another workspace will currently not automatically re-bind a generic SQL/Database connection to that workspace’s lakehouse SQL endpoint, and the “data source credentials missing” is expected.

Instead of connecting to the SQL Endpoint of the Lakehouse, you can try to connect to the Lakehouse itself and parameterize is using the WorkspaceId and LakehouseId. Check it this article https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-variable-references.

Hope this helps. If so, please give kudos 👍 and mark as Accepted Solution ✔️ to help others.

[Nikhilesh_NG]

Hi nielsvdc,

Thank you so much for your inputs. I have looked for a workaround option by creating views through which I can connect to lakehouse directly using one lake instead of using sql end point.