Forum Discussion
Dataflow timeout with Sharepoint.Files connector
I'm getting a dataflow timeout with some files I'm trying to pull from a Sharepoint folder. To my knowledge, there is no way to specify a timeout duration for the Sharepoint.Files connector - is there some way I can pull the data without timing out? I am unable to proceed with the query transformation due to this. Files being pulled are approximately 35mb each (total 200+mb).
Code below:
let
Source = SharePoint.Files("https://xxxxx.sharepoint.com/sites/xxxxx/xxxxx/", [ApiVersion = 15]),
#"Filter Path" = Table.SelectRows(Source, each Text.StartsWith([Folder Path], "https://xxxxx.sharepoint.com/sites/xxxxx/xxxxx/Data Repository/VData/")),
#"Get Year" = Table.AddColumn(#"Filter Path", "Year", each Number.FromText(Text.BetweenDelimiters([Name], "(", ")")), type number),
#"Filter Files" = Table.SelectRows(#"Get Year", each [Year] >= 2014 and [Year] <= 2021 and not Text.Contains([Name], "_q")),
#"Get Data" = Table.AddColumn(#"Filter Files", "Data", each Excel.Workbook([Content])),
#"Select Data Column" = Table.SelectColumns(#"Get Data", "Data")
in
#"Select Data Column"
2 Replies
We utilize a rudimentary retry approach, like so:
... #"Added custom" = Table.AddColumn(#"Filtered rows", "Data", each Table.SelectColumns(Table.PromoteHeaders( try Csv.Document([Content], [Delimiter = ",", Encoding = 1252, QuoteStyle = QuoteStyle.Csv]) otherwise Csv.Document(Binary.Buffer([Content]), [Delimiter = ",", Encoding = 1252, QuoteStyle = QuoteStyle.Csv]) , [PromoteAllScalars = true]), Columns)), ...That allows us to ingest 160 CSV files (up to 300 MB each) from a sharepoint folder.
Thanks, I think this has something to do with our dataflow code overall since when I get this error, the dialog box shows the all of the queries that are inside the dataflow which gives me the impression that we will have to silo some of our data on a different dataflow/query altogether.
