Re: [PATCH olang v2 1/3] lexer: add tokenize support to binary op tokens

["Carlos Maniero" <carlos@maniero.me>]

> +            case '=': {
> +                size_t start_offset = lexer->offset;
> +
> +                if (lexer_peek_next_char(lexer) == '=') {
> +                    lexer_skip_char(lexer);
> +                    lexer_skip_char(lexer);
> +                    lexer_init_str_value_token(lexer, token, TOKEN_CMP_EQ, start_offset);
> +                    return;
> +                }
> +
> +                lexer_init_char_value_token(lexer, token, TOKEN_EQ);
> +                lexer_skip_char(lexer);
> +                return;
> +            }

We could apply the *maximal munch* [1] here avoiding looking to the
future by skipping the char right after founding it:

+            case '=': {
+                size_t start_offset = lexer->offset;
+                lexer_skip_char(lexer);
+
+                if (lexer_current_char(lexer) == '=') {
+                    lexer_skip_char(lexer);
+                    lexer_init_str_value_token(lexer, token, TOKEN_CMP_EQ, start_offset);
+                    return;
+                }
+
+                lexer_init_char_value_token(lexer, token, TOKEN_EQ);
+                lexer_skip_char(lexer);
+                return;
+            }

I know it is a silly change, but this help us to keep the parsing
process as deterministic as possible.

[1]: https://en.wikipedia.org/wiki/Maximal_munch#:~:text=In%20computer%20programming%20and%20computer,as%20possible%20should%20be%20consumed.