From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id 6KuhHzWL92XlBAAAqHPOHw:P1 (envelope-from ) for ; Mon, 18 Mar 2024 01:30:45 +0100 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0.migadu.com with LMTPS id 6KuhHzWL92XlBAAAqHPOHw (envelope-from ) for ; Mon, 18 Mar 2024 01:30:45 +0100 X-Envelope-To: patches@johnnyrichard.com Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=lists.sr.ht header.s=20240113 header.b=iBEu4f1P; dkim=none ("invalid DKIM record") header.d=maniero.me header.s=hostingermail1 header.b=Atg9r4Mn; spf=pass (aspmx1.migadu.com: domain of lists@sr.ht designates 46.23.81.152 as permitted sender) smtp.mailfrom=lists@sr.ht; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=maniero.me (policy=none); arc=pass ("mailchannels.net:s=arc-2022:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=johnnyrichard.com; s=key1; t=1710721845; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=byyPjoIUKMjcnoDTt6f0s4tgwJACGfTEc92jC0iqqks=; b=5u5p1abD35tj5kRKwMNSwfHa90/KAn1HxDssMHOgXBOnBI3vs4oKdQQKC83ZIEWSAMKbgn qg7SNpykubThoY8y5fpThtHbuV7EtMnaM4YZZzo8TFN67S6pDrsXTC2ikdohAUhN3GKsjf LrqH9CRoJi7+Sy7sNwFy0oEmxkQMS1F/e3smZS2mGNUdb81lMmDOemLZS64dh5sySjOwd9 6UKIak0lZL+sl8WzdFUr8OTOiI43IPlXYPLJIQz/mmMjDKmFawdlgEzZajzi0QrzZHSXG2 ILbZPATXIpFzKBsuqSn72+pZSmge9pszExvUvRR5uVB5xLoD34wxCgFXzZMytA== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=pass header.d=lists.sr.ht header.s=20240113 header.b=iBEu4f1P; dkim=none ("invalid DKIM record") header.d=maniero.me header.s=hostingermail1 header.b=Atg9r4Mn; spf=pass (aspmx1.migadu.com: domain of lists@sr.ht designates 46.23.81.152 as permitted sender) smtp.mailfrom=lists@sr.ht; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=maniero.me (policy=none); arc=pass ("mailchannels.net:s=arc-2022:i=1") ARC-Seal: i=2; s=key1; d=johnnyrichard.com; t=1710721845; a=rsa-sha256; cv=pass; b=Eb3vCW7doB06qP5Q5laQcJgRZwj60OgHf9wdOSSn+Dc1g8Yn4CIIlSiIDyhatVub/z4Wg6 SWdJTIrQ0OzK1SZl8HHKzlNXywrb5RiQVTeg0+vk56d2e6hLVX6yMxEgQZ5GsVPfmLg+Wo HH4gtZ/u4Y1mjcCl7XVc0g++EngQkKfy+pWNFlkWNlorOLqE5x6rfLDE8P5p3HMdrMFC7A bgA3X767l7GWyuULHMI74NLJ2FMRzgej2ZrmEPys2qv/rrkZPR8o87NRmAs1Uj/0gvcCOG DEh9pmXdzFVqvlI8y8zQcRJICklyb7cIblr/F/KCz2TMENhUFr69dy2GzIN41w== Received: from mail-a.sr.ht (mail-a.sr.ht [46.23.81.152]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 27AA063EB7 for ; Mon, 18 Mar 2024 01:30:44 +0100 (CET) DKIM-Signature: a=rsa-sha256; bh=dAGv9w3/cDvc8Hjdksb9JEiFD2M29VJAdTUfeX+6s8I=; c=simple/simple; d=lists.sr.ht; h=Date:Subject:From:To:References:In-Reply-To:List-Unsubscribe:List-Subscribe:List-Archive:List-Post:List-ID; q=dns/txt; s=20240113; t=1710721844; v=1; b=iBEu4f1PHhJH993puXWNkxF6PaxXBere+SuptAJHmnayzRQCTv8fBMdbhrnWXDGBqkCor3Vb 6K8Gka/Lz5gJUxbeChLeHuLW9sHwS8LZNgKDSNZOkdHIiRmaqHxExQIYG4RIoiNQM7QqUZPFV9a kjCq2jFgpHoWqssHkz6lvm7YMA6vspELadyYqZKlWG74f91N4dOeyzffZrK0lS27SkUG3H0VZPs 2F1aBHM7Mrbj6Z/ukMg8nA6tsYXtpAZZjw/m1/3tPdfxMp4HPF7Qg48QQn3HCN8EtYPDyLotwcZ TCC+8nHjPODAdITNFhrAYy9pi1ksrMS4pL0IPaRl9+Wmw== Received: from lists.sr.ht (unknown [46.23.81.154]) by mail-a.sr.ht (Postfix) with ESMTPSA id 9D96120107 for ; Mon, 18 Mar 2024 00:30:44 +0000 (UTC) Received: from cross.elm.relay.mailchannels.net (cross.elm.relay.mailchannels.net [23.83.212.46]) by mail-a.sr.ht (Postfix) with ESMTPS id 7B9752009F for <~johnnyrichard/olang-devel@lists.sr.ht>; Mon, 18 Mar 2024 00:30:43 +0000 (UTC) X-Sender-Id: hostingeremail|x-authuser|carlos@maniero.me Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id 821D9101705 for <~johnnyrichard/olang-devel@lists.sr.ht>; Mon, 18 Mar 2024 00:30:41 +0000 (UTC) Received: from nl-srv-smtpout4.hostinger.io (unknown [127.0.0.6]) (Authenticated sender: hostingeremail) by relay.mailchannels.net (Postfix) with ESMTPA id D286E101601 for <~johnnyrichard/olang-devel@lists.sr.ht>; Mon, 18 Mar 2024 00:30:40 +0000 (UTC) ARC-Seal: i=1; s=arc-2022; d=mailchannels.net; t=1710721841; a=rsa-sha256; cv=none; b=cfQFy8nXWeAKP1udeyka/EH94oyjEaGXogYPL9V6Cnd0uCqFtgWJZMBFY9mHiLKHwhbTEu Ni9DYNn7+59BCetbMm7oRE2iaWItyr1ywIKcEqCPm8TAqyPxySzV3M0aIei9lScglRwcZj Ye8H0HYs5Fmig3V4ZrhjcrEYPnGm7n2TWxRLOeH6rhG9FggKvJDHNBCBmkbbUjJd/91FwZ cISCOZ07DVwHqgcGODMs2CVT+9Vl1d0Jr3E0VvsFthvJ7y12Vl1IzG5vA410A49Ha1n8g0 Tu9Aj8eEs3QmODeeayRE0057nWlpIV+m0jrxUYlH3hKq9mV+2iaYUPyl/dQwSA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=mailchannels.net; s=arc-2022; t=1710721841; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=byyPjoIUKMjcnoDTt6f0s4tgwJACGfTEc92jC0iqqks=; b=EmvlQVkDX1h6cM4nAPZYmDul+bdj1Ulf8LDL7c3JczUKat1AslJPJE5xIoZW9PIzVw43QG sQmYs4QCIQFp0kraPsyaXXWqOEMe0GbC3/Rx3Q3cZr34Vs7HeA0FfaKQvb4PgsG/xqRicR ISLmeUB2fH7tMM5zeH9p6OzgppKEiQFQ8Lckuok2PItdSI1lpSWN1trd9C9y8vEbEqw3DR gIuIebJGsGGLrqcJmaBvska8sn4/Tjae8HUEua2rtym2i5+Nh9Be43z/CqDUk4eOiz8HIM jeNg6RbYgHKT2zvSOs6LKJfP9VPJqjyTBwPugDjOG25UXUd60ew5I4ayhfZ/Cg== ARC-Authentication-Results: i=1; rspamd-76c7995f89-p48d7; auth=pass smtp.auth=hostingeremail smtp.mailfrom=carlos@maniero.me X-Sender-Id: hostingeremail|x-authuser|carlos@maniero.me X-MC-Relay: Neutral X-MailChannels-SenderId: hostingeremail|x-authuser|carlos@maniero.me X-MailChannels-Auth-Id: hostingeremail X-Eyes-Wide-Eyed: 6c3af1ff10e1de76_1710721841339_2290680272 X-MC-Loop-Signature: 1710721841338:2913180428 X-MC-Ingress-Time: 1710721841338 Received: from nl-srv-smtpout4.hostinger.io (nl-srv-smtpout4.hostinger.io [145.14.159.45]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384) by 100.119.164.55 (trex/6.9.2); Mon, 18 Mar 2024 00:30:41 +0000 Mime-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=maniero.me; s=hostingermail1; t=1710721839; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=byyPjoIUKMjcnoDTt6f0s4tgwJACGfTEc92jC0iqqks=; b=Atg9r4Mn9hVbo8e0DybFmUpGzxYy7fNcApA2eUiSt5SSLwIzZ7DSH36FTVFpm2jicyauXJ c9r0RgyC2GIbSNr1tzETFDGAXgDZitWVOZURcdgaX75V7OmxMisRIy8vaGJyjn0C5kfjzw LypcTt6MzxgaV4cVF8k8a2njL+0/yqVNsP2isQKeMiNPI/Ytp2wlFyXrsc4GUg9pXNIHWd Rpwk+iU2PiS8ThDTFFZ0k9LFjB9Msrj1tvUakgocdkDex15McqU/18Y+tCY/WR0VMkOVcE a5SCCLIg9iYJ9dNRj62GyeE5kRaTKSFxqYG6CEYEH2pzBorj2BChRsaDCUdDlw== Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Sun, 17 Mar 2024 21:30:34 -0300 Message-Id: Subject: Re: [PATCH olang v2 1/3] lexer: add tokenize support to binary op tokens From: "Carlos Maniero" To: "Johnny Richard" , <~johnnyrichard/olang-devel@lists.sr.ht> X-Mailer: aerc 0.15.2-211-g37d5fc691aff References: <20240317213638.131057-1-johnny@johnnyrichard.com> <20240317213638.131057-2-johnny@johnnyrichard.com> In-Reply-To: <20240317213638.131057-2-johnny@johnnyrichard.com> X-CM-Envelope: MS4xfCqj0fYXheVaSwIJhtdHnJSCZpLbk/GAuS8Z6vj9WXpWXqXC3EwOUHqMyzLLg0eDgiZeo7ObXoNNo1IQTtbVk86TDzsKvXTgT7ykFUr0YnWZrMLZuyVc 0BJj/CWI1Vcxo7B0Yz8W2IfvXTLjg8tZG72dp8H6f/oRfCgWddqEqkT9luFMfE6qG5jv9OrJ4sItASYZ2w/6k97jd5NHA0KFpf6H6Sl4N3MoXtjQsPZkvmdN CIXm+ZF+elGQSOx7Ov8RVA== X-CM-Analysis: v=2.4 cv=YaC75RRf c=1 sm=1 tr=0 ts=65f78b2f a=5+VMC1FZ3J4mVPAKpPmAqg==:117 a=5+VMC1FZ3J4mVPAKpPmAqg==:17 a=IkcTkHD0fZMA:10 a=MKtGQD3n3ToA:10 a=1oJP67jkp3AA:10 a=8pif782wAAAA:8 a=n2FU1lq4D2n3wfte0iMA:9 a=QEXdDO2ut3YA:10 a=BXDaF_L80NY05PYiAFlV:22 X-AuthUser: carlos@maniero.me List-Unsubscribe: List-Subscribe: List-Archive: Archived-At: List-Post: List-ID: ~johnnyrichard/olang-devel <~johnnyrichard/olang-devel.lists.sr.ht> Sender: ~johnnyrichard/olang-devel <~johnnyrichard/olang-devel@lists.sr.ht> X-Migadu-Flow: FLOW_IN X-Migadu-Country: NL X-Migadu-Spam-Score: -7.97 X-Spam-Score: -7.97 X-Migadu-Queue-Id: 27AA063EB7 X-Migadu-Scanner: mx13.migadu.com X-TUID: WrFoQ4URxR0J > + case '=3D': { > + size_t start_offset =3D lexer->offset; > + > + if (lexer_peek_next_char(lexer) =3D=3D '=3D') { > + lexer_skip_char(lexer); > + lexer_skip_char(lexer); > + lexer_init_str_value_token(lexer, token, TOKEN_CMP_E= Q, start_offset); > + return; > + } > + > + lexer_init_char_value_token(lexer, token, TOKEN_EQ); > + lexer_skip_char(lexer); > + return; > + } We could apply the *maximal munch* [1] here avoiding looking to the future by skipping the char right after founding it: + case '=3D': { + size_t start_offset =3D lexer->offset; + lexer_skip_char(lexer); + + if (lexer_current_char(lexer) =3D=3D '=3D') { + lexer_skip_char(lexer); + lexer_init_str_value_token(lexer, token, TOKEN_CMP_EQ,= start_offset); + return; + } + + lexer_init_char_value_token(lexer, token, TOKEN_EQ); + lexer_skip_char(lexer); + return; + } I know it is a silly change, but this help us to keep the parsing process as deterministic as possible. [1]: https://en.wikipedia.org/wiki/Maximal_munch#:~:text=3DIn%20computer%20= programming%20and%20computer,as%20possible%20should%20be%20consumed.