Good way to tokenize line from file without using external libraries? - string

Question

I am trying to tokenize a database dump separated by commas. I only need to read the first word, which will tell me if this is the line I need and then tokenize the line and save each separated string in a vector.
I have had trouble keeping all of the datatypes in order. I use a method of getline:
string line;
vector<string> tokens;
// Iterate through each line of the file
while( getline( file, line ) )
{
// Here is where i want to tokenize. strtok however uses a character array and not a string.
}
The thing is, I only want to continue reading and tokenize a line if the first word is what I am after. Here is a sample of a line from the file:
example,1,200,200,220,10,550,550,550,0,100,0,-84,255
So, if I am after the string example, it goes ahead and tokenizes the rest of the line for my use and then stops reading from the file.
Should I be using strtok, stringstream or something else?
Thank you!

Answer 1

#include <iostream>
#include <fstream>
#include <vector>
using namespace std;
void do(ifstream& file) {
string line;
string prefix = "example,";
// Get all lines from the file
while (getline(file,line).good()) {
// Compare the beginning for your prefix
if (line.compare(0, prefix.size(), prefix) == 0) {
// Homemade tokenization
vector<string> tokens;
int oldpos = 0;
int pos;
while ((pos = line.find(',', oldpos)) != string::npos) {
tokens.push_back(line.substr(oldpos, pos-oldpos));
oldpos = pos + 1;
}
tokens.push_back(line.substr(oldpos)); // don't forget the last bit
// And here you are!
}
}
}

Answer 2

How do I tokenize a string in C++?
http://www.daniweb.com/software-development/cpp/threads/27905
Hope this helps, though I am not proficient C/C++ programmer. For the record it would be nice if you could specify in the tags or in post language you are using.

Answer 3

Tokenizer.h
#ifndef TOKENIZER_H
#define TOKENIZER_H
#include <string>
#include <vector>
#include <sstream>
class Tokenizer
{
public:
Tokenizer();
~Tokenizer();
void Tokenize(std::string& str, std::vector<std::string>& tokens);
};
#endif /* TOKENIZER_H */
Tokenizer.cpp
#include "Tokenizer.h"
using namespace std;
string seps(string& s) {
if (!s.size()) return "";
stringstream ss;
ss << s[0];
for (int i = 1; i < s.size(); i++)
ss << '|' << s[i];
return ss.str();
}
void tok(string& str, vector<string>& tokens, const string& delimiters = ",")
{
seps(str);
string::size_type lastPos = str.find_first_not_of(delimiters, 0);
string::size_type pos = str.find_first_of(delimiters, lastPos);
while (string::npos != pos || string::npos != lastPos)
{
tokens.push_back(str.substr(lastPos, pos - lastPos));
lastPos = str.find_first_not_of(delimiters, pos);
pos = str.find_first_of(delimiters, lastPos);
}
}
Tokenizer::Tokenizer()
{
}
void Tokenizer::Tokenize(string& str, vector<string>& tokens)
{
tok(seps(str), tokens);
}
Tokenizer::~Tokenizer()
{
}
To tokenize a string
#include "Tokenizer.h"
#include <string>
#include <vector>
#include <iostream>
#include <cstdlib>
using namespace std;
int main()
{
// Required variables for later below
vector<string> t;
string s = "This is one string,This is another,And this is another one aswell.";
// What you need to include:
Tokenizer tokenizer;
tokenizer.Tokenize(s, t); // s = a string to tokenize, t = vector to store tokens
// Below is just to show the tokens in the vector<string> (c++11+)
for (auto c : t)
cout << c << endl;
system("pause");
return 0;
}