Forum Discussion
Query references other queries or steps, so it may not directly access a data source
Hi,
Newbie here. I'm trying to calculate a delta between two dates in my query (RequestedCompletionDate and ActualCompletionDate). I found ImkeF's Networkdays function and was able to use it initally, but now it gives me an error:
"Formula.Firewall: Query 'ProjectTiming' (step 'Added Custom1') references other queries or steps, so it may not directly access a data source. Please rebuild this data combination."
I've read a couple of other posts on this topic, and a common proposed solution is to integrate the custom function within your query. I can't seem to get the syntax correct. Here's my query (note: GetDataProjectTiming is a custom function that contains the particulars of the API call to our work system--my query paginates the data from this function; this works fine--it's integrating the Networkdays function that's throwing the error):
let
Source = List.Generate(
()=> [Result = try GetDataProjectTiming(0) otherwise null, Offset = 0],
each List.IsEmpty([Result][data]) <> true,
each [Result = try GetDataProjectTiming([Offset]+2000) otherwise null, Offset = [Offset]+2000],
each [Result]),
#"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"data"}, {"data"}),
#"Expanded data" = Table.ExpandListColumn(#"Expanded Column1", "data"),
#"Expanded data1" = Table.ExpandRecordColumn(#"Expanded data", "data", {"ID", "name", "owner", "referenceNumber", "actualCompletionDate", "DE:Requested completion date", "DE:Project Timing", "DE:Proposition this request falls under."}, {"ID", "name", "owner", "referenceNumber", "actualCompletionDate", "DE:Requested completion date", "DE:Project Timing", "DE:Proposition this request falls under."}),
#"Expanded owner" = Table.ExpandRecordColumn(#"Expanded data1", "owner", {"name"}, {"name.1"}),
#"Renamed Columns" = Table.RenameColumns(#"Expanded owner",{{"ID", "ProjectID"}, {"name", "ProjectName"}, {"name.1", "OwnerName"}, {"referenceNumber", "ReferenceNumber"}, {"actualCompletionDate", "actualCompletionDateOLD"}, {"DE:Requested completion date", "RequestedCompletionDate"}, {"DE:Project Timing", "ProjectTiming"}, {"DE:Proposition this request falls under.", "Proposition"}}),
#"Added Custom" = Table.AddColumn(#"Renamed Columns", "ActualCompletionDate", each Text.RemoveRange([actualCompletionDateOLD], 19, 4)),
#"Changed Type" = Table.TransformColumnTypes(#"Added Custom",{{"ActualCompletionDate", type datetimezone}}),
#"Changed Type1" = Table.TransformColumnTypes(#"Changed Type",{{"ActualCompletionDate", type date}, {"ProjectID", type text}, {"ProjectName", type text}, {"OwnerName", type text}, {"ReferenceNumber", Int64.Type}, {"RequestedCompletionDate", type date}, {"ProjectTiming", type text}, {"Proposition", type text}}),
#"Removed Columns" = Table.RemoveColumns(#"Changed Type1",{"actualCompletionDateOLD"}),
#"Added Custom1" = Table.AddColumn(#"Removed Columns", "Delta", each NetworkDays([ActualCompletionDate], [RequestedCompletionDate], Holidays[HolidayDate]))
in
#"Added Custom1"Any help would be greatly appreciated. Thanks!
I tried removing the holiday component as a parameter in the function and hard-coding a reference to the list object:
I tested it, and the function itself works--I was able to get the correct values by inputting test start and end dates. However, when I use the function in my main query (again, passing only the start and end dates), I still get the "references other queries" error.
I may have to just stick with the combo method (calling the SharePoint list of holidays directly in my main query) or try calculating the delta in a DAX measure.
13 Replies
- rodrigosanResponsive Resident
Hi, boolittlek
The Formula.Firewall error typically occurs when you mix a streaming data source (like your List.Generate API pagination) with another external data source (your Holidays table) without buffering the static data first. The engine gets confused about data privacy partition.
I have refactored your code below to do two things:
List.Buffer: I loaded the Holidays list into memory before the API call loop starts. This treats the holidays as a constant list, usually resolving the Firewall partition issue.
Clean Up: I removed spaces and special characters from the step names (e.g., changing #"Added Custom" to AddedCustom). This is a best practice in M to make the code more stable and readable.
Here is the refactored code:
let // 1. Buffer the Holidays list into memory FIRST to avoid Firewall errors // Assuming 'Holidays' is your table and 'HolidayDate' is the column name containing dates HolidaysList = List.Buffer(Holidays[HolidayDate]), // 2. Your API pagination Source = List.Generate( () => [Result = try GetDataProjectTiming(0) otherwise null, Offset = 0], each List.IsEmpty([Result][data]) <> true, each [ Result = try GetDataProjectTiming([Offset] + 2000) otherwise null, Offset = [Offset] + 2000 ], each [Result] ), // 3. Transformation Steps (Renamed to remove spaces/special chars) ConvertedToTable = Table.FromList( Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error ), ExpandedColumn1 = Table.ExpandRecordColumn(ConvertedToTable, "Column1", {"data"}, {"data"}), ExpandedData = Table.ExpandListColumn(ExpandedColumn1, "data"), ExpandedDataRecords = Table.ExpandRecordColumn( ExpandedData, "data", { "ID", "name", "owner", "referenceNumber", "actualCompletionDate", "DE:Requested completion date", "DE:Project Timing", "DE:Proposition this request falls under." }, { "ID", "name", "owner", "referenceNumber", "actualCompletionDate", "DE:Requested completion date", "DE:Project Timing", "DE:Proposition this request falls under." } ), ExpandedOwner = Table.ExpandRecordColumn(ExpandedDataRecords, "owner", {"name"}, {"name.1"}), RenamedColumns = Table.RenameColumns( ExpandedOwner, { {"ID", "ProjectID"}, {"name", "ProjectName"}, {"name.1", "OwnerName"}, {"referenceNumber", "ReferenceNumber"}, {"actualCompletionDate", "actualCompletionDateOLD"}, {"DE:Requested completion date", "RequestedCompletionDate"}, {"DE:Project Timing", "ProjectTiming"}, {"DE:Proposition this request falls under.", "Proposition"} } ), // Clean text range AddedCustom = Table.AddColumn( RenamedColumns, "ActualCompletionDateClean", each Text.RemoveRange([actualCompletionDateOLD], 19, 4) ), ChangedType = Table.TransformColumnTypes( AddedCustom, {{"ActualCompletionDateClean", type datetimezone}} ), ChangedTypeFinal = Table.TransformColumnTypes( ChangedType, { {"ActualCompletionDateClean", type date}, {"ProjectID", type text}, {"ProjectName", type text}, {"OwnerName", type text}, {"ReferenceNumber", Int64.Type}, {"RequestedCompletionDate", type date}, {"ProjectTiming", type text}, {"Proposition", type text} } ), RemovedColumns = Table.RemoveColumns(ChangedTypeFinal, {"actualCompletionDateOLD"}), // Calculate Delta using the Buffered Holidays List AddedDelta = Table.AddColumn( RemovedColumns, "Delta", each NetworkDays([ActualCompletionDateClean], [RequestedCompletionDate], HolidaysList) ) in AddedDeltaNote on Privacy Settings: You might find that going to File > Options and settings > Options > Privacy and selecting "Ignore the Privacy Levels" fixes the error immediately.
However, be careful with that approach: it disables important security checks and might cause refresh failures once published to the Power BI Service (Gateway) if the server settings don't match. The code fix above (List.Buffer) is the robust way to solve it ensuring your refresh works everywhere.
Let me know if this helps!
- boolittlekFrequent Visitor
It's a good thought, and a very helpful explanation of what's causing the issue, but I'm getting the same Formula.Firewall error message.
I started playing around a bit, and I think I figured out how to insert the coding I use to create the Holidays table within the pagination query. I still need to do some additional testing to make sure it doesn't throw errors in PBI Desktop or PBI Online Service.
Thanks for the clean-up tips, too!
- Shubham_rai955Super User
This error is from the privacy/firewall rules: in one query you both call an API (GetDataProjectTiming) and reference another query/table (Holidays), which is not allowed in that partition.
Option 1 – Turn holidays into a list parameter
In a separate query Holidays, keep only the HolidayDate column and turn it into a list: HolidaysList = List.Buffer(Holidays[HolidayDate]).
Create a new function fnNetworkDays that takes StartDate, EndDate and HolidaysList as parameters (copy Imke’s function, but use a parameter instead of referencing the Holidays query).
In ProjectTiming use:
#"Added Custom1" = Table.AddColumn( #"Removed Columns", "Delta", each fnNetworkDays([ActualCompletionDate], [RequestedCompletionDate], HolidaysList) )Now the query only calls a function and does not directly reference the Holidays query, so the firewall is satisfied.
Option 2 – Put everything in one query (quick and dirty)
If this is for personal use only, you can instead set Current File ➜ Privacy to “Ignore the privacy levels and potentially improve performance” and refresh again.
This disables the firewall check, but is not recommended for shared or sensitive data models.- boolittlekFrequent Visitor
I'm able to create the list (Option 1, Step 1), but I'm unclear what needs to change in the new fnNetworkDays function (Option 1, Step 2).
I was able to put to put my code for the Holidays table (which I extract from SharePoint) within my pagination query (that pulls in data from the GetDataProjectTiming function). I set both data sources to the Organizational privacy setting, and this appears to be working in both Desktop and the Online Service (I did not have to use the 'Ignore privacy levels' setting).
- V-yubandi-msftCommunity Support
Thank you for the update.
For Option 1, the key point is that the custom fnNetworkDays function just needs an additional parameter for your Holidays list. Once that's included, the function should use this list internally, rather than referencing the Holidays query directly. This helps avoid the privacy or firewall issue.
Your current method bringing the Holidays data into the same query as your API pagination and setting both to the Organizational privacy level also works well. Since it's functioning in both Desktop and the Service, you're in a good position. Option 1 is just a cleaner and more reusable approach if you want a solution that works independently across queries.
I hope this helps clarify things. If you have any further questions, feel free to ask.
- V-yubandi-msftCommunity Support
Hi boolittlek ,
Just checking in are you still experiencing any issues? If you need more information or clarification, please let us know.
Thank You.
- V-yubandi-msftCommunity Support
Hi boolittlek ,
May I know if your issue is resolved? If you need any additional help or further details, please let us know.
Thank you.- boolittlekFrequent Visitor
Hi--I will proceed with combining everything into a single query. Thanks--I appreciate everyone's advice and suggestions.
- V-yubandi-msftCommunity Support
Thank you for sharing your approach.