Learn to use Lex, in particular ocaml's version of lex (ocamllex).
Generate an scanner (tokenizer) using ocaml's lex tool so that you recognize the following tokens:
1. Integer constants, which do not have any leading redundant zeros, and may have a leading minus sign.
2. Real Number constants: Begin with an optional sign, have no redundant leading in the mantissa or ordinate; nor trailing zeros (in the decimal part).
3. Identifiers, which are alphanumeric (a-zA-Z0-9), with possible underscore (_) and apostrophe ('), beginning with a capital letter in the case of variables.
4. Arithmetic operators: +, -, *, div, mod, /
5. Boolean operators: /\, \/, ~
6. Comparison operators: >, <, >=, <=, =, /=
7. Control operators: semicolon (;), if, then , else,
8: Parentheses (,) and brackets [, ]
9. Whitespace: blank, tab, newline
10. Separators: comma (,)
11. Field qualifiers: dot (.), #
12. List operators: @
13. Case separators: |
You need to define an appropriate ocaml data type called "token" which has the appropriate representations for each of the above tokens