Created this message here after having a call with microsoft support.
The behaviour inside a fabric pipeline has recently updated when defining a lakehouse connection.
This new approach does not seem to be feature complete, since it provides less functionality then the old approach.
Below my remarks for a dynamic connection and a regular connection.
regular connection
Now it creates an actual connection inside the "Manage Connections and Gateways" overview.
The connection which lives here runs under personal oauth credentials.
In the old situation it just had a Lakehouse connection, without a designated connection in the mentioned overview.
Having personal oauth credentials in a connection is undesriable in a production setup.
Besides that we now have a new connection to manage, and it complexifies (or even bypasses) read/write permissions set in the lakehouse.
dynamic content
Prior behaviour was that you can enter the lakehouse-id, then you get input boxes Connection Type; and Workspace ID.
Now you also get an additional field Lakehouse ID, which is strange, singe you allready added that inside the connection.
When running this, it now fails that it cannot find the connection, I guess becuase we entered a lakehouse-id inside the connection-id.
So I guess we now have to define a connection, and use that ID, but then we run into the issues mentioned under regular connection.
documentation refference
https://learn.microsoft.com/en-us/fabric/data-factory/connector-lakehouse-copy-activity
Connection: Select a Lakehouse connection from the connection list. If no connection exists, then create a new Lakehouse connection. If you apply Use dynamic content to specify your Lakehouse, add a parameter and specify the Lakehouse object ID as the parameter value. To get your Lakehouse object ID, open your Lakehouse in your workspace, and the ID is after /lakehouses/in your URL.
4 Comments
- I believe you may be confusing the "Lakehouse Connection ID" with the "Lakehouse ID". The former is Lakehouse agnostic, in that it does not point to any one Lakehouse. You can find its ID under Manage Connections and Gateways. There you set the Authentication Type (OAuth) and the credentials. The Lakehouse ID, on the other hand, points to a specific Lakehouse inside a Workspace. You can get it by clicking on the Lakehouse itself, then look for the appropriate GUID in the browser address. Hope this helps.
- Thank you for your reponse. The prior behaviour was that I could use the "Lakehouse ID" inside the Connection field via dynamic content. Now I have to use the "Lakehouse Connection ID", but that works via Oauth / my personal credentials. How does the relation ship work, that this connection is Oauth, and other permissions set inside the lakehouse / workspace?
- To be honest, I never really understood why you need BOTH a generic 'Warehouse' (or 'lakehouse') connection under Connections and Gateways that is workspace independent, AND then specify a specific warehouse (or lakehouse) in the pipeline. But that's what Microsoft has given us. Worse, these connections under Connections and Gateways ONLY support OAuth, NOT Service Principals :(, meaning that the account listed under Authentication needs to log in every 30 days to refresh the Access Token.
I stumbled into my own solution recently. It is also possible to selection connection = none. Which then seems to default back into the old behavior where you do setup a connection using oauth credentials.
