Remove header row from CSV's - excel

Question

I have a directory with circa 3k CSV files containing various data, I need to collate these into a single file at some point, but first I need to remove all of the header rows from each file.
Usually for this I would collate the files, and then simply open in Excel, and filter to the header rows before deleting them all. Unfortunately, these sum to something around 9M rows, and Excel doesn't like that...
Can anybody think of a way around this? Preferably some sort of batch script that will run through all files in a directory.
Thanks in advance,
A.

Answer 1

The following assumes the first line of each file is the header line to be eliminated.
It will only work properly if none of the files contain the <TAB> character, and none of the files is too large. I can't remember the specifics, but at some point, MORE with redirected output will hang waiting for a keypress if the input file gets too large.
(for %F in (*.csv) do #more +1 "%F") >concat_csv.txt
I made sure to give the output file a different extension so that the command does not try to process the output! An alternative is to redirect the output to a CSV file but in a different folder.
If you want to use this in a batch file, then double up the percents (%F becomes %%F)

Answer 2

I am not sure this is what you are looking for... Here is one way to get rid of the duplicate headers in C#. The main purpose of the code is to store one header is string header and to read the files by skipping the first row (while (rdr.Peek() != -1)).
I have also used a dictionary to store the rows of each csv file. This will prevent duplicate rows in different csv files to be included (I am not sure if this function will be helpful in your case).
Imagine fname is a string array that contains the files you wish to merge.
Dictionary<string, string> dict = new Dictionary<string, string>();
string destinationFile = <write path of your destination file>;
string dir = <write path of your original directory>
string header = "";
if (dir.Length != 0)
{
foreach (string f in fnames)
{
using (StreamReader rdr = new StreamReader(dir + "\\" + f))
{
header = rdr.ReadLine();
while (rdr.Peek() != -1)
{
string ln = rdr.ReadLine();
string[] split_ln = ln.Split(',');
string value = (split_ln.Length != 2) ? string.Join(",", split_ln.Skip(1)) : split_ln[1];
dict.Add(split_ln[0], value);
}
}
}
using (StreamWriter wr = new StreamWriter(destinationFile))
{
wr.WriteLine(header);
foreach (var pair in dict)
{
wr.WriteLine("{0},{1}", Convert.ToString(pair.Key), pair.Value);
}
}
}