Forum Discussion
Parameterizing copy job that uses a rest connection
Hi,
I'm trying to use the copy job feature specifying a rest connection, after selected it from OneLake catalog.
For a particular Base URL I'd like to specify a dynamic content for the relative URL to pass the table name as a parameter.
I want to use the copy job to get data from a Dynamics 365 Business Central solution by the Rest API connector.
As the Base URL I can indicate the OData link of the BC API service, but for the Relative URL I cannot specify a table name parameter as a dynamic content and therefore I need to create one copy job for each BC table to read and not only one copy job.
Now, any suggests to me, please? Many thanks
6 Replies
- v-dineshya
Community Support
Hi pmscorca ,
Thank you for reaching out to the Microsoft Community Forum.
The Copy job UI for REST connectors in Fabric currently does not support parameterization of the Relative URL directly. It expects a static endpoint for each job. This is the reason that you cannot inject dynamic content like @{tableName} or similar expressions.
Please try below alternative workarounds.1. Instead of creating multiple copy jobs manually, Create a Data Pipeline in Fabric. Define a pipeline parameter like tableName. In the Copy Activity, use the REST connector and reference the parameter in the Relative URL like tables/@{pipeline().parameters.tableName}.Then run the pipeline in a loop for all table names using ForEach activity.
Note: By using above approach you only maintain one pipeline and dynamically pass table names.
2. If you are using Dataflow Gen2, you can create a query that uses Power Query M functions to construct the URL dynamically.
Please refer below sample M code.
let
BaseUrl = "https://api.businesscentral.dynamics.com/v2.0/tenantId/ODataV4/",TableName = "customers",
FullUrl = BaseUrl & TableName,
Source = OData.Feed(FullUrl)
inNote: You can then expose TableName as a parameter in the dataflow.
3. If you have more tables, Please consdier a Fabric Notebook or Azure Data Factory pipeline that Iterates over a list of table names. Calls the REST API dynamically. Writes data to OneLake.Please refer below links.
Configure REST in a copy activity - Microsoft Fabric | Microsoft Learn
REST API Capabilities for Copy job in Fabric Data Factory - Microsoft Fabric | Microsoft Learn
Items - Create Copy Job - REST API (CopyJob) | Microsoft Learn
Copy job definition - Microsoft Fabric REST APIs | Microsoft Learn
Dynamic relative URL for dataset in ADF - Microsoft Q&A
How can I parameterize Rest API DataSet or Linked Service while using Data Flow - Microsoft Q&A
I hope this information helps. Please do let us know if you have any further queries.
Regards,
Dinesh
- v-dineshya
Community Support
Hi pmscorca ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. And, if you have any further query do let us know.
Regards,
Dinesh
- pmscorca
Post Prodigy
Hi v-dineshya , thanks for your reply.
Does it make sense to provide an advanced feature like copy job without being able to parameterize it?
The actual copy job feature cannot be used for operations to iterate, but only for a single operation. It's a bad limitation.
Instead, it would be very useful to have a fully parameterizable copy job.
The suggested alternative workarounds have been already tested.Thanks
- Ugk161610
Super User
Hi pmscorca ,
I faced the same problem while trying to use a REST connection in a Copy Job. I wanted to make the Relative URL dynamic (for example, pass the table name from Business Central as a parameter), but it looks like the Copy Job UI doesn’t support that right now.
The only way I got this working was by using a Fabric Data Pipeline instead of the Copy Job wizard.
Here’s what I did:Created a REST connection with the main Base URL.
Then built a dataset with a parameter for the relative path (for example, tableName).
In the pipeline, I used a ForEach activity to loop through the table names and passed them one by one into that parameter.
Finally, ran the Copy activity inside the loop — this way it picked up each table dynamically.
So basically, the Copy Job wizard is fixed to one endpoint, but the pipeline lets you make it dynamic.
Hopefully, Microsoft adds this feature in the future because it would save a lot of setup time.Hope that helps!
Gopi Krishna