Assistance with Migrating Dataflows from Gen1 to Gen2 in Fabric Lakehouse

[SRUK]

Hi Team,

I am new to the Fabric/Power BI admin side of things. Any suggestions or pointers will help me with the following questions:

1. At present, we have created many dataflows using personal gateways to access source data from cloud-hosted SQL databases. We created them in the semantic model and dataflows.

2. We are moving to Fabric Lakehouse and have obtained Fabric licenses. As a next step, we are creating a Data Lakehouse. My question is: can I migrate existing dataflows from Gen1 to Gen2? The challenges I faced when creating in Gen2 include on-premises gateway settings and login details not being saved, refresh failures, and sudden switching between Fabric licenses and Power BI Pro. I have gone through Microsoft Learn, but it looks straightforward and easy as they only provide simple examples. Any suggestions on where I can find good information on migrating Gen1 dataflows to Gen2 dataflows in Power BI into Fabric?

3. Finally, our data refreshes fail whenever the Power BI report owner changes their password. Is it good to have a service account login for database access and refresh scheduled by the report owner so that the refresh doesn't fail when the report owner's Windows credentials change?

Thanks in advance , and sorry for the many questions.

[v-kpoloju-msft]

Hi @SRUK,

Thank you for reaching out to the Microsoft Fabric Community Forum.

We really apologies for the inconvenience, after reviewing the issue of migrating dataflows from Gen1 to Gen2 in fabric Lakehouse, here are few steps to may resolve the issue.

Please kindly go through the below following steps:

• Please ensure you understand the differences in capabilities between Gen1 and Gen2 dataflows. The migration process requires recreating the dataflows in the new environment rather than performing a direct one click migration.
• Ensure that you’re on-premises data gateway is correctly configured to connect to your cloud hosted SQL databases. Verify that the gateway can connect to the data sources without any interruptions. This can be done in the Power BI Service under the gateway settings.
• When configuring data sources in Gen2, ensure that you accurately input and save the credentials in the dataflow settings. If you encounter any issues, please re enter the connection details, including the authentication account.
• Use a service account with the required permissions for database access. Configure it in your data connection settings to avoid refresh failures when the report owner's password changes.
• Utilize the Power BI Service to check failure messages for refresh jobs. This can identify issues with credentials or connectivity. Enable email notifications for quick responses to failures.
• Ensure all users have the correct licenses to avoid confusion between Fabric and Power BI Pro. Verify that features exclusive to Fabric are properly licensed for all relevant users.

After recreating dataflows in Gen2, test thoroughly to ensure data ingestion and refresh processes work correctly. Compare outputs with old dataflows for consistency.

Also, please go through the below mentioned document links for better understanding:
Migrate from Dataflow Gen1 to Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

Migrate from Dataflow Gen1 to Dataflow Gen2: Migration scenarios - Microsoft Fabric | Microsoft Lear...
Move queries from Dataflow Gen1 to Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.

Thank you.

[v-kpoloju-msft]

Hi @SRUK,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[v-kpoloju-msft]

Hi @SRUK,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thankyou.

[v-kpoloju-msft]

Hi @SRUK,

I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.

Thank you.

[nilendraFabric]

@SRUK if this resolved the question please accept the solution, it will help the other users to find the solution quickly 

[SRUK]

Thanks Nilendra,  you help is much apperciated.   I moved into some other urgent task. Will try next week and update you.  Thanks so much. Sree

[nilendraFabric · Accepted solution]

Hi @SRUK 

Yes, it is possible to migrate existing dataflows from Gen1 to Gen2. However, there isn't a direct migration tool available. Instead, you'll need to follow a manual process:

• 
  Export your Gen1 dataflow as a Power Query Template (.pqt file)
• Create a new Gen2 dataflow in your Fabric workspace.
• Import the .pqt file into the new Gen2 dataflow
• Reconfigure data source connections and credentials
• Set up data destinations (e.g., Lakehouse, Warehouse) for your Gen2 dataflow

This process allows you to reuse your existing dataflow logic while taking advantage of Gen2 features.

Resources for Gen1 to Gen2 Migration

https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-migrate-from-dataflow-gen1-scena...

https://www.hubsite365.com/en-ww/crm-pages/migrate-from-power-bi-dataflow-gen1-to-fabric-dataflow-ge...

https://youtu.be/FmZo2eDTPYY

Using Service Accounts for Data Refresh

Your suggestion to use a service account for database access and scheduled refreshes is a good practice

.This approach offers several benefits:

1. Prevents refresh failures due to individual password changes.
2. Improves security by not relying on personal credentials
3. Simplifies management of data source connections across multiple reports.

To implement this:

1. Create a dedicated service account with the necessary permissions.
2. Update your dataflow and dataset connections to use this service account.
3. Configure the on-premises data gateway to use the service account credentials

The issues you're experiencing with on-premises gateway settings, login details, and refresh failures are common when transitioning to Gen2 dataflows. Here are some suggestions:

1. Ensure your on-premises data gateway is up-to-date and properly configured for Fabric
2. Double-check that you have the necessary permissions in Fabric to manage dataflows and gateways
3. For login and credential issues, consider using a service account instead of personal credentials
4. If you're experiencing sudden switching between Fabric licenses and Power BI Pro, ensure that your Fabric capacity is properly configured and that your workspace is assigned to it

If this is helpful , please accept the solution and give kudos 🙂

Thanks