Forum Discussion
Append to an existing CSV file
This may not be the correct forum for this question, but I'm hoping my developer friends may have some suggestions.
I have multiple CSV files.
MainFile.csv (15,000 KB in size)
NewFile.csv (12 KB in size)
I want to append the rows from NewFile.csv to the MainFile.csv
Both files contain the same headers, but I only want the non-header rows from NewFile.csv to append to MainFile.csv (MainFile.csv will maintain its existing headers and rows).
Basically NewFile.csv is a set of new orders that I want automatically pulled into MainFile.csv
I've tried using a batch file script to do this, and it works, but it's slow. It combines the two files into a brand new combined.csv file, so it takes a good 20 minutes to build it. I'm thinking there's a better way to just append the new records to the existing file without having to rebuild it each time. Below is the batch file script I've been using. Any thoughts on better code or a better method than batch file to accomplish this?
echo off
ECHO Set working directory
pushd %~dp0
ECHO Deleting existing combined file
del combined.csv
setlocal ENABLEDELAYEDEXPANSION
set total=0
set count=0
REM Set total
for %%i in (*.csv) DO set /a total+=1
for %%i in (*.csv) DO (
cls
echo:Combining CSV files [!count!/%total%]
if !count!==0 (
for /f "delims=" %%j in ('type "%%i"') do echo %%j >> combined.csv
) else if %%i NEQ combined.csv (
for /f "skip=1 delims=" %%j in ('type "%%i"') do echo %%j >> combined.csv
)
set /a count+=1
)
move C:\Source\MainFile.csv C:\Source\Archived
copy C:\Source\combined.csv C:\Source\MainFile.csv
del C:\Source\NewFile.csv
pause
2 Replies
- JirkaZ
Solution Specialist
This being a Power BI forum I'll reply with what's closest to where we are - use SSIS package.
- JustSayJoe
Advocate IV
I don't have enough data to justify using a SQL Server at this point. Appreciate the response and welcome any other suggestions.