dataflow
6 TopicsFilter CSV dataset based on a column
Hi, I am a newbie to ADF. i have CSV dataset referring from SFTP holding 14 million records. Currently, it has copyactivity which copies to CSV dataset(unzipped) from source zipped format dataset! Then dataflow , which refers this CSV and loads to AzureSQL (upsert operation) . The pipeline fails at times. Its an ask to filter read data only from 8 days backdated based on a column "DateColumn" present in the CSV file. I added this condition(filter) in dataflow (Source ->filter ->sink) where Filter takes expression as DateColumn>= toString(addDays(currentDate(), -8)). The pipeline doesnt haveconsistent success! any other method would help since the data would be growing for years .Solved801Views0likes4Commentsconnect button is disable while trying to setup a new data source connection against Azure Cosmos DB
I was trying to create a new Data pipeline within Fabric. I have my Azure Cosmos DB for MongoDB resource setup in my subscribtion. I added copy Data activity and was tryingto create a new source connection with MongoDB. Got the new data source connection popup and searched for Azure Cosmos DB for MongoDB. I filled in all the fields but the Connect button below appears disabled. Any resolution for this issue?Solved993Views0likes4CommentsGet Metadata activity doesnt terminate sometimes
In our datafactory solution we have a pipeline which uncludes a few "Get Metadata"-activities. The pipeline runs every 30minutes and sometimes it fails (around 1/70 of the runs) because one of the "Get Metadata"-activities doesnt terminate. Subsequently, the outer-activity and/or the pipeline fails. The activity which is not terminating as not always the same andtherefore doesnt access the same folders in a ADLS Gen 2 always, but they all access the same DataLake. I checked the logs and it seems there are also logs regarding the termination of the activity( no "Error"- and no "Succeeded" logs). When i monitor the pipeline runs it says the activities are still running since days even thouigh the pipeline terminated already. The reasons why the pipeline failed are either: "Operation on target TestDimTableLoop failed: Activity failed because an inner activity failed" if the "Get Metadata"-activity is within a loop and "InternalServerError executing request" if not, but I dont see anything more and dont know how to look any deeper into that. Does anyone might have an idea what the problem might be?Solved1.6KViews0likes5CommentsDataflow Id as variable is not working in a pipeline
I created a pipeline that sets a variable with a value as an existing dataflow Gen2's name. In the conditional, I check for the variable to match that name. Execute the dataflow with the dataflowId as the variable's value. (See image) When I run the pipeline, I get the error "Microsoft.DataTransfer.Execution.Core.ExecutionException: Refresh Dataflow failed with status: NotFound message: Not Found". I'd expect the dataflow to run as it exist already. Can you tell what the issue could be?Solved1.5KViews0likes3CommentsGet Metadata is failing to extract metadata like structure, exists, file type for large data excel
Hi Team, Get Metadata activity is failing to extract metadata like structure, exists, file type for large data excel file. Also unable to copy data from it using Copy Activity. Range option is not working for excel dataset. Could you please help us with these issues as soon as possible?Solved1.9KViews0likes2CommentsData Pipeline - DataFlow Gen 2, Variables assignment
1. I have a dataflow in Data Factory to which I applied some transformations such as 'Unpivot Columns' and 'Group By'. I want to create a pipeline which ensures that new data gets added in my dataflow automatically and any transformation such as 'unpiviot' or 'group by' that I applied to my dataflow earlier gets applied to the new data that is added. 2. I have a question regarding the 'Variable' feature in Data Pipeline. I created a pipeline wherein I have dataflow gen 2 as first activity and on success of that activity, I want to assign my dataflow a variable. The pipeline runs successfully but I do not see anything in the output.'Solved4.9KViews0likes9Comments