topic Dynamic datasources are not supported. in Dataflow

Dynamic datasources are not supported.

M0hamedFazil — Wed, 07 Aug 2024 11:53:01 GMT

Hey,

Apologies if my methods doesn't make any sense as I am new to fabrics

Now my work is to get a JSON response from API and transform and load it to a destination. Since there are many tranformations need to be done I used Data flow that calls the API and do some transformations to get the final output. I want to include this dataflow to my datapipeline where some access tokens and some other things required to call the API is done.

I planned on passing the access token as a parameter to the dataflow where the API is comprised of the access token. When I publish the dataflow I get the error like "Dynamic datasources are not supported".

Thanks in Advance

Regards
Fazil M

Re: Dynamic datasources are not supported.

NandanHegde — Wed, 07 Aug 2024 13:38:47 GMT

Based on my understanding,As of today dataflow Gen 2 sources cannot be parameterized.

Meaning the source has to be a hardcoded version and hence you might be getting the error

Re: Dynamic datasources are not supported.

M0hamedFazil — Thu, 08 Aug 2024 04:19:54 GMT

Hi NandanHegde

Thanks for the reply

I wonder is there any other way I pass my data from an activity to dataflows. I have gone through somw blogs and videos and they suggest to stage the data and make the dataflows read from the stage.

Is there any other ways to do that?

Regards,
Fazil M

Re: Dynamic datasources are not supported.

NandanHegde — Fri, 09 Aug 2024 03:46:12 GMT

Unfortunately as of today, it is not possible to have dynamic source /changing data within dataflow Gen2 at run time.

The source within the Dataflow Gen 2 has to be static so you can try directly connecting the source within dataflow Gen2 else as stated by othershave a staging layer like blob or any other source

Re: Dynamic datasources are not supported.

M0hamedFazil — Fri, 09 Aug 2024 04:21:16 GMT

Thank you for the reply NandanHegde

I'll apt to dump my data to a stage and let the dataflow read the data from stage

Regards,

Fazil M