From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mail-a.sr.ht; dkim=pass header.d=johnnyrichard.com header.i=@johnnyrichard.com Received: from out-189.mta1.migadu.com (out-189.mta1.migadu.com [IPv6:2001:41d0:203:375::bd]) by mail-a.sr.ht (Postfix) with ESMTPS id 3AA3C2032D for <~johnnyrichard/olang-devel@lists.sr.ht>; Wed, 28 Feb 2024 18:10:21 +0000 (UTC) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=johnnyrichard.com; s=key1; t=1709143821; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XTcPqY5FGaGjn4rufnKcT1hPz5VadV7p7sFPTs+L1Ok=; b=YwSaD5BxFuWs3oGARvagIOZabpYvJkBbm4Ucwo7Fgl7W/hubxoz+ZBsVK7P4J0CfjbZBNd 3DQrbppg0GRCkejVO41xUdfhXBdAa0K4Kai65Ihp7VqJetXrWTjFQt8hRc3mhja8LBVn6j hJwI2IGXlIe4rzEhKGlFNemgq6s/fvBlgFcLX2hJ+VqDdReNnE29Cwj545RMMwWdbnwsFh uIwkK8DHB3qi5wAnP7r3KgEfG7cRsk4nZLpruM2zAXExJJrqGtsKcA0u7HpEKQyy8p8Hp+ PmW4PaV06mStZqujP9L//d5HJi7W+n1k3PAKbo+y2Eyk4IqJ3UJFSL1ppUNiZQ== From: Johnny Richard To: ~johnnyrichard/olang-devel@lists.sr.ht Cc: Johnny Richard Subject: [PATCH olang v1 3/4] lexer: add token lookahead capability Date: Wed, 28 Feb 2024 20:04:20 +0100 Message-ID: <20240228190956.78191-4-johnny@johnnyrichard.com> In-Reply-To: <20240228190956.78191-1-johnny@johnnyrichard.com> References: <20240228190956.78191-1-johnny@johnnyrichard.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-TUID: M/qzYhSkqzLb In order to skip line breaks (LF) we have to be able to spy the next token without consume it. This patch also adds a function to **lexer_peek_next** token, which is equivalent to **lexer_look_ahead(n = 1)**. Signed-off-by: Johnny Richard --- src/lexer.c | 22 ++++++++++++++++++++++ src/lexer.h | 6 ++++++ 2 files changed, 28 insertions(+) diff --git a/src/lexer.c b/src/lexer.c index b107762..c7756a6 100644 --- a/src/lexer.c +++ b/src/lexer.c @@ -233,3 +233,25 @@ lexer_str_to_token_kind(string_view_t text) return TOKEN_IDENTIFIER; } + +void +lexer_peek_next(lexer_t *lexer, token_t *token) +{ + lexer_lookahead(lexer, token, 1); +} + +void +lexer_lookahead(lexer_t *lexer, token_t *token, size_t n) +{ + size_t previous_offset = lexer->offset; + size_t previous_row = lexer->row; + size_t previous_bol = lexer->bol; + + for (size_t i = 0; i < n; ++i) { + lexer_next_token(lexer, token); + } + + lexer->offset = previous_offset; + lexer->row = previous_row; + lexer->bol = previous_bol; +} diff --git a/src/lexer.h b/src/lexer.h index 8c09e02..729c957 100644 --- a/src/lexer.h +++ b/src/lexer.h @@ -68,6 +68,12 @@ lexer_init(lexer_t *lexer, string_view_t source); void lexer_next_token(lexer_t *lexer, token_t *token); +void +lexer_peek_next(lexer_t *lexer, token_t *token); + +void +lexer_lookahead(lexer_t *lexer, token_t *token, size_t n); + char * token_kind_to_cstr(token_kind_t kind); -- 2.43.2