I have several dataflows that use library variables so they can be deployed through the dev-test-prod deployment pipeline and refer to different workspaces & lakehouses within those different stages. These variables are accessed through the Variable.ValueOrDefault method, with the default value referring to the development environment. After the source data is accessed, I wrote a custom line to test if the table has 0 rows of data and it throws an error if true (to prevent the dataflow from overwriting the destination with 0 rows).
I ran the dataflow in the test environment at a time when the development environment had 0 rows of data in the source table, but the test environment should have been pulling from the test environment lakehouse so I didn't expect an error. The validation step of the dataflow failed, and only succeeded when I changed the Default values to the test environment IDs or after I loaded new data into the development environment lakehouse table.
It seems to me that the validation step is not fully running the Variable.ValueOrDefault method to get the current environment and is just taking the default values, so the validation is going to fail any time there is an error based on the development environment. It makes more sense to me for the validation step to use the library variables in the current environment so this doesn't cause issues with pipelines in the test & production environments when the development environment is going through changes.
3 Comments
- miguel
Community Admin
Status changed:NewtoAbandonedFor this particular case, we recommend using the Variable.Value approach which is now supported. This will make sure that your validations and every single evaluation has to be evaluated against the actual variable library. You can also wrap this around a "try ... otherwise ...." approach (as described here Error handling - Power Query | Microsoft Learn) to make it a bit more robust and provide some other mechanisms to fail in the event that the variable couldn't be loaded.
The Variable.ValueOrDefault has an internal mechanism by design that, on any issues retrieving the variable, it'll use the default value so it wouldn't truly fail. There are other ways in which you could tackle this by also leveraging a conditional mechanism for the Default value and validating other components, but they are not as straightforward as just using the Variable.Value approach.
Hope this helps!
- Thanks miguel, that clears it up! Perhaps this nuance could be described better in the learn documentation here, as I was led to believe that ValueOrDefault was the preferred method: https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-variable-library-integration#:~:text=Using%20a%20default%20value%20through%20Variable.ValueOrDefault%20helps%20ensure%20that%20your%20formula%20resolves%20even%20when%20you%20copy%20or%20move%20your%20solution%20to%20another%20environment%20that%20doesn%27t%20have%20the%20referenced%20variable%20library.
- miguel
surpriseturtle would you mind creating a new thread in the community forum with more details around your scenario? would love to learn more about how you have it setup and where things might have not been yielding the result that you were expecting with Variable.ValueOrDefault
the ValueOrDefault is definitely the recommendation for most scenarios given that you can copy / paste this M code into other products and it'll continue to work without you having to modify your script. The Variable.Value only functions with the right identifiers in the Dataflow Gen2 CICD context, so it always forces things to correctly evaluate or it'll throw an error, which would be the easiest approach for you in really nuance cases where a Default might not be the right approach
Recent ideas
executeDaxQueries-like API call with effectiveUserName on SSO-enabled model with Service Principal
Please provide a way for a service principal to execute DAX - with an effective user name - against an SSO-enabled directlake semantic model and get per-user results that respect lakehouse OneLake Se...
