This illustrates why lexer tokens should preferably not be defined as exactly what the language allows. Instead the internal lexer definition should include invalid tokens (like “0xfor”) that are only sorted out in a later step (in this case, when actually converting it to an integer value).
back
1 comments
Yes, that is basically how C works, though the C standard describes it in a weird way. There is the notion of a “preprocessor number” which starts with a digit and allows any characters that can appear in a number or identifier. At a later step in parsing it is converted to a strict number, which would fail in a case like 0xfor.