Compiler Tokens

When a C++ compiler analyses a program, it breaks the program into tokens. A token is a distinct unit that is recognizable by the compiler. There are five categories of tokens: 

• keywords
• literals 
• identifiers
• operators 
• white space includes: blanks, horizontal and vertical tabs, new line and form feed characters and comments 

Keywords & Literals  

There are in total 48 keywords defined in C++.Keywords should be considered as reserved. Keywords are not allowed to be used as variable names. Literal · may be, character constant, integer constant, floating point _ constant, string constant.  

Identifiers 

Rules for valid identifiers: 

1. Sequence of letters, digits, or underscores (_).
2. No limit to length, though some compilers consider only the first 32 characters.
3. No spaces or special characters; only letters, digits, and underscores allowed.
4. Must begin with a letter or underscore, not a digit.
5. Case-sensitive; all characters are significant.
6. Keywords cannot be used as identifiers.