Question
A simple tokenizer Introduction You have joined a new team as a C++ developer. Your first task is to make a simple but efficient string
A simple tokenizer
Introduction
You have joined a new team as a C++ developer. Your first task is to make a simple but efficient string tokenization engine. There is no place for dynamic memory allocations or heavy object copies here.
Problem Statement
The Input format
1.The input for a tokenizer is either empty or contains a sequence of tokens.
2.Each token consists of akey=valuepair, wherekeyis always an integral value, whereas=is a literal, andvalueis any string.
3.Tokens in the input text are separated by a single-character separator (i.e.,;).
4.The last token can be also followed by the same separator.
5.There are no empty tokens, which means that1=value1;;3=value3is an illegal input.
6.Input sequence will never contain a whitespace next to the=literal or a separator.
The interface
In order to make the tokenizer stateless and not to force it to make a container of tokens, which would demand dynamic memory allocation, the interface requires providing anOutputFunc.
template<char Separator, typename OutputFunc>
void tokenize(std::string_view txt, OutputFunc out);
OutputFuncis a functor providing the following interface:
void(int key, std::string_view value)
This functor has to be called for every token parsed from an input stream.
Use case example
The above interface allows the following use case:
tokenize<';'>("1=value1;2=value2", [](int key, std::string_view value) {
std::cout << "[key = " << key << ", value = " << value << "]";
});
Hints
1.Implement your tokenizer intokenizer.handtokenizer.cppfiles.
2.It might be a good idea to add your own custom unit tests to theunit_tests.cppfile.
Step by Step Solution
There are 3 Steps involved in it
Step: 1
Get Instant Access to Expert-Tailored Solutions
See step-by-step solutions with expert insights and AI powered tools for academic success
Step: 2
Step: 3
Ace Your Homework with AI
Get the answers you need in no time with our AI-driven, step-by-step assistance
Get Started