general comment
102 TopicsWhat are the best practices for building ETL pipelines in Microsoft Fabric Data Factory?
Hi everyone, I'm currently learning Microsoft Fabric and exploring Data Factory for ETL and data integration. I'd like to understand how experienced users design efficient and maintainable pipelines. I have a few questions: How do you organize pipelines in a real-world Fabric project? When do you prefer Data Factory over Spark notebooks for data transformation? What are some best practices for handling pipeline failures and monitoring? How do you manage incremental data loads instead of full refreshes? Are there any common mistakes that beginners should avoid? I'd appreciate any advice, examples, or resources that have helped you. Thank you!Solved224Views0likes3CommentsInvoke Pipeline Returns "Could not found the requested item"
I'm having issues with the Invoke Pipeline(Preview) activity where I am getting the error {"requestId":"e9e54d07-8d30-4de4-b3f4-3be3ad15a476","errorCode":"ItemNotFound","message":"Could not found the requested item"} To test this issue, I set up a: A pipeline with the only activity in it being Invoke Pipeline(Preview) No parameters - The original pipeline I set up had parameters and I was getting the same error code so I tried to remove it to see if it was a parameter issue A child pipeline with the only activity in it to Get Metadata for a table in a lakehouse No parameters This pipeline will successfully run on its own When I run #1 I get the error message {"requestId":"e9e54d07-8d30-4de4-b3f4-3be3ad15a476","errorCode":"ItemNotFound","message":"Could not found the requested item"} however I'm not sure why the Item would not be found since I was able to specify it. I am attaching my configuration settings as screenshots to this post. Did I miss something in the Invoke Pipeline settings? Parent Pipeline Child PipelineSolved4.5KViews0likes6CommentsHow to automatically switch database connections when deploying Data Pipeline from Dev to QA
Body: Hello Community, I am working with Microsoft Fabric workspaces and facing an issue during deployment. Scenario: I deployed a workspace from Dev to QA. The workspace includes a Data Pipeline. After deployment, the pipeline’s Lookup activity and the pipeline database connection are still pointing to the Dev database connection. Problem: Currently, I have to manually update the database connection in the Lookup activity and Data Pipeline after every deployment. Expectation: I would like to configure Deployment Rules (or any other supported mechanism) so that during deployment, the pipeline and lookup automatically point to the correct QA database connection instead of Dev. Question: Is there a supported way to parameterize database connections in Data Pipelines during workspace deployment? Or is this a current limitation of Fabric Deployment Rules? Any guidance or workaround will be very helpful. Labels: Data Factory, Administration, General DiscussionSolved4.7KViews0likes8CommentsEvolving the ADF to Fabric migration experience
Hi team, When working with Azure Data Factory and moving into Microsoft Fabric, the new “migration” experience is a really interesting step forward. It helps bring both worlds closer and makes the transition smoother. After trying it out, I’ve seen that it opens up a lot of possibilities, and it’s great to see that we can also propose ideas to continue improving it. In this context, I’ve shared an idea that I believe could bring even more value. Right now, the experience allows you to connect and integrate existing ADF assets into Fabric, which is very helpful to start working within the ecosystem. As a next step, it could be interesting to move towards a deeper integration where: Assets are fully integrated into Fabric They are executed from Fabric And their consumption can be associated with Fabric capacity Additionally, it could be valuable to give customers the flexibility to choose where the cost is allocated (Fabric or Azure), enabling a more gradual adoption. This is a small evolution on top of what already exists, but it could make a big difference in how the experience aligns with different adoption scenarios. If this idea resonates with you, you can vote for it here: https://lnkd.in/ejT9QSRj And if you find it useful, I would really appreciate your support with a like on the comment and voting for the idea. This helps increase visibility and gives it a better chance to evolve.499Views2likes1CommentMicrosoft Fabric - Edit the Connection Properties
Hi Team, I am using the Microsoft Fabric and created the on Premise Gateway and created the ODBC connections which are used in the pipelines. Now my On Premise Gateway and ODBC DSN has changed. So I want to Edit my existing connection to repoint my ODBC DSN & On Premise Gateway. But in connection level there is no option to edit these properties. Its allowing only to edit the Username & Password. Can you please confirm, any other way to change the existing connection or only option is create a new connection and all the pipeline level need to point my new connection. Regards, Sathish KumarSolved3.1KViews0likes8CommentsDataflow Gen2: Variable Library references fail in Connection Settings
Hi everyone, I am working on variabilizing my Dataflow Gen2 for deployment pipelines, following the documentation on Variable references. I’ve noticed a discrepancy in how variables are handled, and I suspect it might be a bug or a UI limitation. The Issue: When I define parameters using a Blank Query and try to use them to replace values in the Source Connection settings (via the user interface), the connection fails or isn't recognized. And replace in the source: let Source = Lakehouse.Contents([]), #"Navigation 1" = Source{[workspaceId = workspace_id]}[Data], #"Navigation 2" = #"Navigation 1"{[lakehouseId = lakehouse_id]}[Data], #"Navigation 3" = #"Navigation 2"{[Id = "Table1", ItemKind = "Table"]}[Data] in #"Navigation 3" That's not working. The Workaround (Manual M Code): However, if I bypass the UI and manually insert the Variable.ValueOrDefault syntax directly into the Source Step using the Advanced Editor, it works perfectly. let Source = Lakehouse.Contents([]), // Retrieving variables manually works fine workspace_id = Variable.ValueOrDefault("$(/**/My Library/Workspace ID)", "default_value"), lakehouse_id = Variable.ValueOrDefault("$(/**/My Library/Lakehouse ID)", "default_value"), #"Navigation 1" = Source{[workspaceId = workspace_id]}[Data], #"Navigation 2" = #"Navigation 1"{[lakehouseId = lakehouse_id]}[Data], #"Navigation 3" = #"Navigation 2"{[Id = "Table1", ItemKind = "Table"]}[Data] in #"Navigation 3" My Question: Has anyone else encountered this issue where the UI connection dialog doesn't seem to parse the variable reference, forcing us to hardcode M logic manually? Is there a specific syntax for the UI that differs from the Advanced Editor? Thanks for your help! References: - Dataflow Gen2 Variable Library IntegrationSolved2.6KViews1like3CommentsIncremental File Ingestion from NFS to LakeHouse using Microsoft Fabric Data Factory
I have an NFS drive containing multiple levels of nested folders. I intend to identify the most recently modified files across all directories recursively and copy only last modifed files into a LakeHouse. I am seeking guidance on the recommended approach to implement this file copy operation using Microsoft Fabric Data Factory. An example of a source file path is: 1. \\XXX.XX.XXX.XXX\PROTOCOLS\ACTVAL\1643366695194009_SGM-3\221499200020\__NOPROGRAM___10004457\20240202.HTM 2. \\XXX.XX.XXX.XXX\PROTOCOLS\ACTVAL\1643366695194009_SGM-3\221499810020\__NOPROGRAM___10003395\20240202.HTM 3. \\XXX.XX.XXX.XXX\PROTOCOLS\ACTVAL\1760427099988857_P902\__NOORDER__\__NOPROGRAM_____NOMOLD__\20251014.HTMSolved1.5KViews0likes4CommentsUserErrorSourceNotSeekable
Hi , im getting error while runnin the pipeline , where i selected the http as an option which i pasted the zip link in the source Failure typeUser configuration issue Details ErrorCode=UserErrorSourceNotSeekable,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Your HttpServer source doesn't support random read which is required by current copy activity setting. Please create two copy activities to work around it: the first copy activity will binary copy your HttpServer source to a staging file store (like Azure Blob, Azure Data Lake, File, etc.), the second copy activity copies from this staged file store to your destination with current settings.,Source=Microsoft.DataTransfer.ClientLibrary,' Thank you Naveen YedlaSolved2KViews0likes5CommentsDataFlow(Gen2) managed by multiple users
When DataFlow(Gen2) is used for work, it is necessary for multiple people to manage it together. However, at present, only one owner is able to check the contents, let alone touch them. Does this not meet the business needs at all? Or is there a proper way to do this that I just don't know about? If anyone knows, I would be very grateful if you could enlighten me.Solved5.5KViews0likes5CommentsA Typo in the new parameter window of Dataflow Gen2 (Korean)
Hi Team, When I tried to create a parameter in the Dataflow Gen2, I noticed a typo in the new parameter window: 어느하나라도 The labels were different from Power BI Desktop. In Dataflow Gen2, that should be modified as follows. 어느하나라도(Any) -> 임의 Korean English Power Query editor (Power BI Desktop 2.146.705.0 64-bit (August 2025)) Korean English Thanks, HongSolved1KViews0likes1Comment