Dataflow Gen2 unexpected error creating data destination

[_seanrich_]

I have a dataflow gen2 with two queries that have data destinations set to a single lakehouse in a separate workspace. This functions just fine in my DEV workspaces. However, in my QA workspaces, I'm getting "Unexpected error occurred while creating your data destination. Please try again later." on just one of the queries. For testing, this query that is failing only has 39 records, all of which are test or whole number data types, with no null values.

• The data source for both queries is Azure Databricks SQL query; connections are successful.
• There are no transformations being done via the queries, as this dfg2 is simply meant for staging data, with notebooks doing transformations downstream.
• Permissions across workspaces are aligned. My account has owner permissions and was the original creator of workspaces and artifacts.

Resolution attempts:

• I ended up recreating this QA environment manually instead of using Git integration to no avail.
• Set Default data destination in the dataflow for the queries to no avail.
• Renamed the destination lakehouse thinking maybe there was a name conflict to no avail.

WorkspaceId: f7ea13ab-80e1-4b3a-bb51-c5ee7a2a9d8f

DataflowId: 2fa57c0d-941e-49a9-bf44-43ec5f6e67f7

SessionId: c8855bf8-ad11-4239-8854-8757329c1300

[v-pbandela-msft]

Hi @_seanrich_,

I wanted to follow up since we haven't heard back from you regarding our last response. We hope your issue has been resolved.
If the community member's answer your query, please mark it as "Accept as Solution" and select "Yes" if it was helpful.
If you need any further assistance, feel free to reach out.

Thank you,
Pavan.

[v-pbandela-msft]

Hi @_seanrich_,

I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, kindly "Accept  as  Solution" and give it a 'Kudos' so others can find it easily.

Thank you,
Pavan.

[v-pbandela-msft]

Hi @_seanrich_,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please "Accept  as  Solution" and give a 'Kudos' so other members can easily find it.

Thank you,
Pavan.

[v-pbandela-msft · Accepted solution]

Hi @_seanrich_,

Thank you for sharing the details of your issue. Based on the error you're encountering, here are a few troubleshooting steps to help resolve the issue in your QA environment:

1.Double-check that the permissions on the lakehouse in the QA workspace are correctly configured and aligned with DEV. Ensure that the service principal or user has the necessary permissions to write data.

2.You can investigate the session logs using the provided SessionId (c8855bf8-ad11-4239-8854-8757329c1300) to look for additional error details that may not be immediately visible in the UI error message.

3.Try deleting and recreating the destination lakehouse in the QA workspace. Sometimes stale configurations or hidden issues can cause errors when connecting to the destination.

4.Compare all settings, configurations, and the destination lakehouse between your DEV and QA environments. Make sure that everything matches and there are no resource limitations in the QA workspace.

5.Review whether there are any throttling or resource limits in the QA environment that might be affecting the dataflow’s ability to create a destination. You can check the Azure resource limits for the workspace and ensure there are no quotas being exceeded.

Please continue using Microsoft community forum.

If you found this post helpful, please consider marking it as "Accept as Solution" and give it a 'Kudos'. if it was helpful. help other members find it more easily.

Regards,
Pavan.