From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:1008:1e59::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id UPX7Ni5J4mWWQAEAbAwnHQ (envelope-from ) for ; Fri, 01 Mar 2024 22:31:26 +0100 Received: from aspmx1.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id uKeoMy5J4mW7PQEA62LTzQ (envelope-from ) for ; Fri, 01 Mar 2024 22:31:26 +0100 X-Envelope-To: patches@johnnyrichard.com Authentication-Results: aspmx1.migadu.com; none Received: from mail-a.sr.ht (mail-a.sr.ht [46.23.81.152]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C761324DD9 for ; Fri, 1 Mar 2024 22:31:26 +0100 (CET) DKIM-Signature: a=rsa-sha256; bh=z7M5osXuOpNauGNInids3xzvvw1Wfq/NGFWKlL+eN34=; c=simple/simple; d=lists.sr.ht; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-Unsubscribe:List-Subscribe:List-Archive:List-Post:List-ID; q=dns/txt; s=20240113; t=1709328686; v=1; b=D13BFG/i0E7yOvXRczoNwZ73DnLlTqjy6q0jv8n7ntQ0ZTuEGyep3qvSHon5glGNBMNpLKYz 1sxZTG7iDikkaMNFv9aY6KVRf6EwiiDFoz9eu/9/ZOlmhO1GPu8rV8DGDIPL+YC8bPYybR/1a5m BNkdKvZN7uHNXkVaM1B5bYa+YOYc6oulQ+bR0v2rhp7NOGycY9u72Bdb7iCcB7XqftzA3fOv0jX 9COYFcA3nlJipKH9WgBLhzdY+h1EZ2heqSFlOXiLmPCh4NzJ3oRW96ujUiwA6VzsBygSG2VY9IX pDEC/4m358Lw6bQ/ecZTB6psaBZlaYjFcpM38oZxZV6UQ== Received: from lists.sr.ht (unknown [46.23.81.154]) by mail-a.sr.ht (Postfix) with ESMTPSA id 2F1AE203F4 for ; Fri, 1 Mar 2024 21:31:26 +0000 (UTC) Received: from out-175.mta1.migadu.com (out-175.mta1.migadu.com [95.215.58.175]) by mail-a.sr.ht (Postfix) with ESMTPS id 936FA2038E for <~johnnyrichard/olang-devel@lists.sr.ht>; Fri, 1 Mar 2024 21:31:25 +0000 (UTC) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=johnnyrichard.com; s=key1; t=1709328685; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XTcPqY5FGaGjn4rufnKcT1hPz5VadV7p7sFPTs+L1Ok=; b=2AsEjVljeGSHhicZQMe2cPb22iVFIMkn8N4SMwZ2rYpGVP6N0geBlyyNv35OQCSGO1sIBi WnExcAojPzT0wfrzki4XvGB5cHXNSaBtXrxpg+vUxFE1waWF17lHGiF1Wx9I9fCumNcCBT 1Pg0EAcpoHEFbBnAzBC2OeeW/ZuE1cHkl58pvdi3Dko4BKmbGnSd09RGkSXP7XnVODyP5r /hrosRFR5Y93+yrtRkHKPVueYLOquzymlFVrz5bEpzPZh2fmPk29CiFtZtS9tVyoya8Dkm zn9Drd08LZANqNHgItKEIGqX64n2EacKGAyuCHQiI6Zk6ilwUnlyr9K7CYOIbw== From: Johnny Richard To: ~johnnyrichard/olang-devel@lists.sr.ht Cc: Johnny Richard Subject: [PATCH olang v2 3/4] lexer: add token lookahead capability Date: Fri, 1 Mar 2024 23:24:08 +0100 Message-ID: <20240301223127.167341-4-johnny@johnnyrichard.com> In-Reply-To: <20240301223127.167341-1-johnny@johnnyrichard.com> References: <20240301223127.167341-1-johnny@johnnyrichard.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Sourcehut-Patchset-Status: UNKNOWN List-Unsubscribe: List-Subscribe: List-Archive: Archived-At: List-Post: List-ID: ~johnnyrichard/olang-devel <~johnnyrichard/olang-devel.lists.sr.ht> Sender: ~johnnyrichard/olang-devel <~johnnyrichard/olang-devel@lists.sr.ht> X-Migadu-Country: NL X-Migadu-Flow: FLOW_IN X-Migadu-Scanner: mx12.migadu.com X-Migadu-Spam-Score: -4.00 X-Spam-Score: -4.00 X-Migadu-Queue-Id: C761324DD9 X-TUID: 1bt0ZQG5qE/O In order to skip line breaks (LF) we have to be able to spy the next token without consume it. This patch also adds a function to **lexer_peek_next** token, which is equivalent to **lexer_look_ahead(n = 1)**. Signed-off-by: Johnny Richard --- src/lexer.c | 22 ++++++++++++++++++++++ src/lexer.h | 6 ++++++ 2 files changed, 28 insertions(+) diff --git a/src/lexer.c b/src/lexer.c index b107762..c7756a6 100644 --- a/src/lexer.c +++ b/src/lexer.c @@ -233,3 +233,25 @@ lexer_str_to_token_kind(string_view_t text) return TOKEN_IDENTIFIER; } + +void +lexer_peek_next(lexer_t *lexer, token_t *token) +{ + lexer_lookahead(lexer, token, 1); +} + +void +lexer_lookahead(lexer_t *lexer, token_t *token, size_t n) +{ + size_t previous_offset = lexer->offset; + size_t previous_row = lexer->row; + size_t previous_bol = lexer->bol; + + for (size_t i = 0; i < n; ++i) { + lexer_next_token(lexer, token); + } + + lexer->offset = previous_offset; + lexer->row = previous_row; + lexer->bol = previous_bol; +} diff --git a/src/lexer.h b/src/lexer.h index 8c09e02..729c957 100644 --- a/src/lexer.h +++ b/src/lexer.h @@ -68,6 +68,12 @@ lexer_init(lexer_t *lexer, string_view_t source); void lexer_next_token(lexer_t *lexer, token_t *token); +void +lexer_peek_next(lexer_t *lexer, token_t *token); + +void +lexer_lookahead(lexer_t *lexer, token_t *token, size_t n); + char * token_kind_to_cstr(token_kind_t kind); -- 2.43.2