How to Handle Case Sensitivity in Dynamic JSON Input Keys for Derived Columns in ADF Dataflow?

[hrithik-swym]

I am working on a dataflow pipeline in ADF with an input dataset schema that includes a column 'A'. The dataflow processes various JSON files with dynamic schemas, where some JSON files use 'A' and others use 'a' as keys. Since the schema is case insensitive, I cannot include both 'A' and 'a' in the input dataset schema. Currently, I have a derived column dependent on the value of 'A'/'a' (it combines <A> + '/' + <date> and the sink stores the folders based on the derived column path data), and it functions correctly for JSON files containing 'A'. How can I modify the dataflow to handle JSON files with 'a' as the key? Is there any other workaround to achieve this in ADF?

Here are examples of two JSON files with dynamic schemas that I'm working with:
Example 1 with "A" as input key   Example 2 with "a" as input key   

[v-nuoc-msft · Accepted solution]

Hi @hrithik-swym 

Thank you very much for your question! This community is dedicated to discussing issues related to Microsoft Fabric.

To provide you with more professional help, we recommend using a dedicated Microsoft Q&A forum to handle Azure Data Factory (ADF) related queries.

They are always ready to help you better. Here's the link to your question in Microsoft's Questions library:

Ask a question - Microsoft Q&A

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.