Question

Discuss in detail Tokens in Python.

Answer 1

Python breaks each logical line into a sequence of elementary lexical components known as Token.
The normal token types are

1. Identifiers
2. Keywords
3. Operators
4. Delimiters and
5. Literals.

Identifiers:

• An Identifier is a name used to identify a variable, function, class, module, or object.
• An identifier must start with an alphabet
  (A..Z or a..z) or underscore (_). Identifiers may contain digits (0 .. 9). „
• Python identifiers are case sensitive i.e. uppercase and lowercase letters are distinct. Identifiers must not be a python keyword.
• Python does not allow punctuation characters such as %,$, @, etc., within identifiers.

Keywords:

Keywords are special words used by Python interpreters to recognize the structure of the program. As these words have specific meanings for interpreters, they cannot be used for any other purpose.

Operators:

• In computer programming languages operators are special symbols that represent computations, conditional matching, etc.
• The value of an operator used is called operands.
• Operators are categorized as Arithmetic, Relational, Logical, Assignment, etc. Value and variables when used with the operator are known as operands

Delimiters:

Python uses the symbols and symbol combinations as delimiters in expressions, lists, dictionaries, and strings.
Following are the delimiters knew as operands.

(	)	]	{	}
,	:	.	'	;

Literals:

Literal is raw data given in a variable or constant. In Python, there are various types of literals.

• Numeric
• String
• Boolean