Forum Discussion
Get all columns from a csv file
- 2 years ago
You're welcome 🙂
Yes, that query was just an example of the process to determine the number of columns.
It can equally be applied to a file from SharePoint or elsewhere with some adjustment.
I would actually recommend creating a function to make your life easier, so that it can be applied to SharePoint files within other queries.
Here is one I created and tested myself just now.
Paste this code into a blank query and call it CsvDocumentVariableColumns (or another name of your choosing).
let func = ( #"File Contents" as binary, optional #"Sample Rows" as number, optional Delimiter as text, optional Encoding as number, optional #"Quote Style" as number ) as table => let // 1. SET PARAMETERS TO DEFAULT VALUES IF NOT SPECIFIED // Set Delimiter to tab if not specified DelimiterFinal = Delimiter ?? "#(tab)", // Set Encoding to 1254 if not specified EncodingFinal = Encoding ?? 1254, // Set QuoteStyle to QuoteStyle.None if not specified QuoteStyleFinal = #"Quote Style" ?? QuoteStyle.None, // 2. PROCESS TEXT FILE // Return a list of lines of text Lines = Lines.FromBinary(#"File Contents", null, null, EncodingFinal), // Take a sample of the lines LinesSample = List.FirstN(Lines, #"Sample Rows" ?? (each true)), // Return the maximum number of columns across all sampled lines MaxColumns = List.Max( List.Transform( LinesSample, each List.Count(Splitter.SplitTextByDelimiter(DelimiterFinal)(_)) ) ), // Read the file using Csv.Document Source = Csv.Document( #"File Contents", [ Delimiter = DelimiterFinal, Columns = MaxColumns, Encoding = EncodingFinal, QuoteStyle = QuoteStyleFinal ] ) in Source, documentation = [ Documentation.Name = "CSVDocumentVariableColumns", Documentation.LongDescription = "Wrapper for Csv.Document that determines the required number of columns based on all rows or a sample of rows" ] in Value.ReplaceType(func, Value.ReplaceMetadata(Value.Type(func), documentation))Then in your code, replace Csv.Document calls like this:
= Csv.Document(Parameter1,[Delimiter=" ", Encoding=1254, QuoteStyle=QuoteStyle.None])with this:
= CsvDocumentVariableColumns(Parameter1)Parameter1 in your original code was (I assume) the result of a function returning binary file contents (may have been auto-generated).
The CsvDocumentVariableColumns function takes at minimum one parameter File Contents (type binary). You can optionally specify Sample Rows, Delimiter, Encoding and Quote Style, but these currently default to your values from above.
Regarding connecting to SharePoint folders, yes you can definitely do that 🙂
There are various articles/guides out there.
- Microsoft documentation:
https://learn.microsoft.com/en-us/power-query/connectors/sharepoint-folder - Guide on YouTube:
https://www.youtube.com/watch?v=mYkNDdjbFvo
Regards
- Microsoft documentation:
Hi jamuka
This is an interesting one 🙂
When the Columns field is omitted, the number of columns is detected from the first row.
Your particular file appears to have only one column populated in the first row, so omitting Columns doesn't work as intended in this case.
You could add steps to detect the maximum number of columns across all rows (or a certain number of rows), then pass the result to the Csv.Document step. My only possible concern would be performance with a large text file.
Here is some sample code which you could adapt:
let
FilePath = "C:\temp\test.txt",
// Tab character
Delimiter = "#(tab)",
// If null then all rows are sampled
NumRows = null,
// Return a list of lines of text
Lines = Lines.FromBinary(File.Contents(FilePath), null, null, 1254),
// Take a sample of the lines
LinesSample = if NumRows = null then Lines else List.FirstN(Lines, NumRows),
// Return the maximum number of columns across all sampled lines
MaxColumns = List.Max(
List.Transform(LinesSample, each List.Count(Splitter.SplitTextByDelimiter(Delimiter)(_)))
),
// Read the file using Csv.Document
Source = Csv.Document(
File.Contents(FilePath),
[Delimiter = Delimiter, Columns = MaxColumns, Encoding = 1254, QuoteStyle = QuoteStyle.None]
)
in
Source
Does this help?
Regards
- jamuka2 years agoHelper IV
Dear OwenAuger,
thank you for your help, it worked when I used your code for a file under C:
But I can't make it work for a file under sharepoint, also instead of connecting to a file, I'd like to connect to a folder because my files name will be change too.
I never used sharepoint as a source, I tried to use several connection types yet none of them work. Also I'm not sure whether it's possible or not to to connect a sharepoint folder, I couldn't find anything about it.
regards