Forum Discussion

somears's avatar
somears
Frequent Visitor
7 years ago

DataFlow Azure Blob Storage Combine Binaries Multiple Csv Files

Is their a way to consume multiple csv files from a Azure Blob storage container using DataFlows? All the files in the blob are of the same format. I’ve successfully created a DataFlow and it is conn...
Dataflow
Need Help
somears's avatar
somears
Frequent Visitor
7 years ago

Thank you for your response.

 

Here is some test data that I'm working with. Each file is pipe delimeted with 10 columns. The blob storage will have hundreds of these files.

 

TestFile1.csv
AMAZ05|20190113|20190119||16.07|C6PATCH1WH|B00016W704||QA|17
AMAZ05|20190113|20190119||16.07|C6PATCH1WH|B00016W704||QP|0
AMAZ05|20190113|20190119||16.07|C6PATCH1WH|B00016W704||QS|1

TestFile2.cvv
AMAZ05|20190113|20190119||16.07|C6PATCH1WH|B00016W704||QU|0
AMAZ05|20190113|20190119||16.07|C6PATCH1WH|B00016W704||QE|17
 

Proof of concept

Create a dataflow in the PowerBI service that will combine all the files in the Azure blob storage container. No additional transformations need to happen. Just expose the below files as 10 columns in the dataflow. Where I'm stuggling is that I don't see a "Combine File" option in the Content column like I see when I'm in the desktop version (left arrow). I do see a Combine tables dropdown option in the tool bar (right arrow). I'm not sure how to use that to combine the tables.

 

Thank you for your help

 

See attached picture below from the power query window in Power BI service Dataflow design.

 

 
 
 
 
 
DataMountain's avatar
DataMountain
Advocate I
7 years ago

This feature isn't available yet and I will be posting an idea.

 

If you have a finite number of files, you can utilize the "Duplicate Query" option to obtain all of the files, then use the "Append Queries as New" option in your picture.

  • Anonymous's avatar
    Anonymous
    Not applicable
    6 years ago

    Hi everyone,

     

    Any idea regarding the above Problem as I'm stuck with the same issue.