Dataflow Gen2 Unable to Write

Question

Hello All,&nbsp;I've been unable to get dataflows to work in two different workspaces. I am trying to load data from Salesforce objects to Fabric. Here are the errors I am recieving. It seems like my instance of Fabric is just broken.. I get errors just trying to open some of the warehouses.&nbsp;Also, the submit button on the support form doesn't seem to work. I am on the Free trial, is it related to that?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Dataflow nameDataflow refresh statusTable nameStart timeEnd timeDurationEngineRefresh statusStatus detailsSF_Account_DataflowFailedNA2023-08-31T20:17:32.13+00:002023-09-01T04:23:15.97+00:008:05:43NAFailedThere was a problem refreshing the dataflow. Please try again later. (Request ID: 49104a42-3a60-40eb-a2d1-4ce3ea19d784).SF_Account_DataflowFailedAccount2023-08-31T20:17:41.647+00:002023-08-31T20:23:05.013+00:000:05:23NASucceededNA&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Dataflow nameDataflow refresh statusActionStart timeEnd timeDurationEngineRefresh statusStatus detailsSF_Account_DataflowFailedWriteToDatabaseTableFrom_TransformForOutputToDatabaseTableFrom_Account2023-08-31T20:23:05.0295499+00:002023-09-01T04:23:06.4325146+00:008:00:01NAFailedTimeout Entity refresh timed out. Please try again or consider reducing the size or complexity of the entity and try again.

himanshus-msft · Answer

Hello Anonymous&nbsp; Thanks for using the Fabric community.
Looking at the error "Timeout Entity refresh timed out. Please try again or consider reducing the size or complexity of the entity and try again." , sounds like either its s momentary issue or the model is really complex . Since you are using a trial version which has a 64 CU alloted . I suggest you to try with something more simple and see if that works .&nbsp;
Let me know how it goes .
ThanksHImanshu

anonymous · Answer

Gen2 Dataflows are still not usable in a production environment as they still have a lot of issues on the load side of their operation.

We are actively engaged with Microsoft Engineers playing whack-a mole and in fairness a lot of progress has been made.

In the mean time we have adopted the follwing solution
1: Load to staging datamart
2: Create a data pipeline that first executes a delete stored procedure on target table in warehouse, then loads data from datamart to datawarehouse using a copy data operation

it is a bit of a faff to setup but once built works well with no intermittent failures that can be exeperienced with Gen2 Dataflows

Forum Discussion

Dataflow Gen2 Unable to Write

2 Replies

Related content

Writing to a specific warehouse schema using a dataflow gen2

Dataflow Gen2 write no value in Lakehouse

Gen 2 Dataflow unable to use other dataflow source

Dataflow Gen2 writing to Warehouse suddenly stopped working - DmsPbiServiceUserException

dataflow gen1

Dataflow name	Dataflow refresh status	Table name	Start time	End time	Duration	Engine	Refresh status	Status details
SF_Account_Dataflow	Failed	NA	2023-08-31T20:17:32.13+00:00	2023-09-01T04:23:15.97+00:00	8:05:43	NA	Failed	There was a problem refreshing the dataflow. Please try again later. (Request ID: 49104a42-3a60-40eb-a2d1-4ce3ea19d784).
SF_Account_Dataflow	Failed	Account	2023-08-31T20:17:41.647+00:00	2023-08-31T20:23:05.013+00:00	0:05:23	NA	Succeeded	NA




Dataflow name	Dataflow refresh status	Action	Start time	End time	Duration	Engine	Refresh status	Status details
SF_Account_Dataflow	Failed	WriteToDatabaseTableFrom_TransformForOutputToDatabaseTableFrom_Account	2023-08-31T20:23:05.0295499+00:00	2023-09-01T04:23:06.4325146+00:00	8:00:01	NA	Failed	Timeout Entity refresh timed out. Please try again or consider reducing the size or complexity of the entity and try again.