Re: [PATCH olang v3 2/2] lexer: create --dump-tokens cli command

["Carlos Maniero" <carlos@maniero.me>]

Nice work man! I just have a few comments:

> +    while (token.kind != TOKEN_EOF) {
> +        printf("%s:%lu:%lu: <%s>\n",
> +               opts.file_path,
> +               token.location.row + 1,
> +               (token.location.offset - token.location.bol) + 1,
> +               token_kind_to_cstr(token.kind));
> +        lexer_next_token(&lexer, &token);
> +    }
IMO, EOF token should be printed to, as it is a token returned by the
lexer.

> +    if (lexer_is_eof(lexer)) {
> +        *token = (token_t){ .kind = TOKEN_EOF };
> +        return;
> +    }

Missing token location. I know it seems silly to have the EOF position.
But it is useful for parser error messages such as "expected } found
EOF". Remember that this code appears twice, before and after the while.


> +lexer_next_char(lexer_t *lexer)

s/lexer_next_char/lexer_current_char
the current name of the function give me the impression that it changes
the offset.

> +    if (lexer->source.chars[lexer->offset] == '\n') {
call lexer_next_char/lexer_current_char instead.

> +static bool
> +_isspace(char c)
> +{
> +    return c == ' ' || c == '\f' || c == '\r' || c == '\t' || c == '\v';
> +}

What do you think about just add the *\n* guard before calling the
*isspace* that way it is clean for someone reading the code why you have
to reimplement the function.

  return c != '\n' && isspace(c);

> +static void
> +lexer_init_char_token(lexer_t *lexer, token_t *token, token_kind_t kind);
> +
> +static void
> +lexer_init_str_token(lexer_t *lexer, token_t *token, token_kind_t kind, size_t start_offset);
> +
> +static token_kind_t
> +lexer_str_to_token_kind(string_view_t text);

I don't have a suggestion to it, but IMO *lexer_init_char_token* and
*lexer_init_str_token* makes me feel we are initializing a "string" and
a 'char" token. But I haven't a better name, I thought calling it
*lexer_init_single_char_token* and *lexer_init_multi_char_token* but
IDK if it is really better.