From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id MMkgIv3x92Ur3wAA62LTzQ:P1 (envelope-from ) for ; Mon, 18 Mar 2024 08:49:17 +0100 Received: from aspmx1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id MMkgIv3x92Ur3wAA62LTzQ (envelope-from ) for ; Mon, 18 Mar 2024 08:49:17 +0100 X-Envelope-To: patches@johnnyrichard.com Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=lists.sr.ht header.s=20240113 header.b=YXCsRVUw; dkim=pass header.d=johnnyrichard.com header.s=key1 header.b=FVuiK7S+; spf=pass (aspmx1.migadu.com: domain of lists@sr.ht designates 46.23.81.152 as permitted sender) smtp.mailfrom=lists@sr.ht; dmarc=pass (policy=quarantine) header.from=johnnyrichard.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=johnnyrichard.com; s=key1; t=1710748157; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-unsubscribe:list-subscribe: list-post:dkim-signature; bh=qwlDiXGVJsh2cTy+AnyE5zm1QbU6ZEkvKqA8du3WDP4=; b=4WI0YTDSa1WV4PVl51EjQS6Mi373I3idVo62DSaRUSi0dUGLmMJgya23gTLf5WlgZ7nOJ7 x1Z3nifKhI5hC/bacXPWx5IFWXXThNRDAZ735oZMlwpf9qzvk88RuV5iMmc31tvTGDIABK oMCF9OrMDjYBZ+X2LG74OZKvxRa2ZTjNGSYXk+TQld/VnFwhTNfTpaNHw5jo1LtYsYDmbB NEvETZ9riBBEtJKePSOVzJUcYbasFOpRWX2gEGUWJ0vj4BgxfYFHDHXsVQdEoBA54aWNpP ooOYqP1NqJoRkqITAuOkZ/dFCr+a25vIXIyCvp694OAv9lMgVGy6ESfMnC1q/A== ARC-Seal: i=1; s=key1; d=johnnyrichard.com; t=1710748157; a=rsa-sha256; cv=none; b=xzlaYQEZvy0MXpcuyLCbhpRyogEE3Lbg2hDEXUootj8D8LYmTwkYrqctZMQ5cMYrglTxN7 2afUNWOiUi8V3YvjuFjyN3RVk9HbNRdmJFjWXQyBCAu6v8J229VYcGP8KrQkmoaEZZlqP+ bVlBYLXw6PzcLCktXdFq+rfZtQs25njjouGF/GMvi1+AZjNqcmSqaUcJR0I3b5EF7daTvO 5eE4c8oI0pgs/zXSXWWDD/jc831znc0isJO3lArNaN/1tQD1ZK88ICX5REhTxe10xjMQ/L JYyWQyqBm0j/RpM83WjRHjwjcrC5c3ckUKFoV1M6MpPyM0e9xV+hwJIIsivBfw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=lists.sr.ht header.s=20240113 header.b=YXCsRVUw; dkim=pass header.d=johnnyrichard.com header.s=key1 header.b=FVuiK7S+; spf=pass (aspmx1.migadu.com: domain of lists@sr.ht designates 46.23.81.152 as permitted sender) smtp.mailfrom=lists@sr.ht; dmarc=pass (policy=quarantine) header.from=johnnyrichard.com Received: from mail-a.sr.ht (mail-a.sr.ht [46.23.81.152]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 79AAA40137 for ; Mon, 18 Mar 2024 08:49:17 +0100 (CET) DKIM-Signature: a=rsa-sha256; bh=Qn8jfhsVrJl++A/bi+1Zvu5jIjsuSXxZf/hVMCFNdNg=; c=simple/simple; d=lists.sr.ht; h=Date:From:To:Cc:Subject:References:In-Reply-To:List-Unsubscribe:List-Subscribe:List-Archive:List-Post:List-ID; q=dns/txt; s=20240113; t=1710748156; v=1; b=YXCsRVUwNzjn5oFbl/jHFCPNXosqISWO06E9psHzE1ybR66Dn8Vz7a/Buo0EUBueABxoXV4c a0L13h/RKOo60rUHSPmoRkWRNF08pGYER8IVaD319TPsnlObsG22bAzUSjj94actAuSqcc+P7pD V6BzWAHM/znt8AJ8BD0XZuCqTxDjFZO0RjvXNiel8hhqjRXrgRRh4lixttHYzIxEhL0fiwaTiix P0QOcLDo0tvJ/dEkehujJHL4PTaEEAGIzIksZaZ90HFabIp22mT7ma6ETw39QsxrOLIEiP9zzDA 8Hlu8JSvVxvizlqBOe1zriAlPv54up2sXlWx4LH1mD0xw== Received: from lists.sr.ht (unknown [46.23.81.154]) by mail-a.sr.ht (Postfix) with ESMTPSA id 5CB6A20100 for ; Mon, 18 Mar 2024 07:49:16 +0000 (UTC) Received: from out-187.mta0.migadu.com (out-187.mta0.migadu.com [91.218.175.187]) by mail-a.sr.ht (Postfix) with ESMTPS id B0443200F1 for <~johnnyrichard/olang-devel@lists.sr.ht>; Mon, 18 Mar 2024 07:49:15 +0000 (UTC) Date: Mon, 18 Mar 2024 09:49:07 +0100 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=johnnyrichard.com; s=key1; t=1710748155; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=qwlDiXGVJsh2cTy+AnyE5zm1QbU6ZEkvKqA8du3WDP4=; b=FVuiK7S+LwUSZziY3cAOjK14wlzfVcduLJsGSeeAUR9/0eucRPe+DJuz/Od3HPQKNbuXX/ CAP80z79oE2RY0GU87AaQsYXT5NZo24610kFnHrQE+mMbaoUBiOXxWHK2Cbt1LLZk29cwh nA7FdUojREM/fKOUsOBl/n2Rkf9UqstFtCg5nQNBL3te37j9f1dtvQjoN7VrUZ3Ha2jyCq po4p6Tv34lUV2R3NLIhhw6BvT9ZlBgKp9YUPVtPyhjTfp9VUpBSy7FPQPbZJVMG4zEFIDE mH6vBkJq63l5M6MbOtMWdyZBhlTTrYz0O4dEgLIN1B6Mbig3w37N/dADIdYzvw== X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Johnny Richard To: Carlos Maniero Cc: ~johnnyrichard/olang-devel@lists.sr.ht Subject: Re: [PATCH olang v2 1/3] lexer: add tokenize support to binary op tokens Message-ID: <7cluu7ds2dietj2akdocfng6xd67wbc3eb2w2m3n46lu3ukcge@ophga22ly6wa> X-Sourcehut-Patchset-Update: SUPERSEDED References: <20240317213638.131057-1-johnny@johnnyrichard.com> <20240317213638.131057-2-johnny@johnnyrichard.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: List-Unsubscribe: List-Subscribe: List-Archive: Archived-At: List-Post: List-ID: ~johnnyrichard/olang-devel <~johnnyrichard/olang-devel.lists.sr.ht> Sender: ~johnnyrichard/olang-devel <~johnnyrichard/olang-devel@lists.sr.ht> X-Migadu-Flow: FLOW_IN X-Migadu-Country: NL X-Migadu-Spam-Score: -9.27 X-Spam-Score: -9.27 X-Migadu-Queue-Id: 79AAA40137 X-Migadu-Scanner: mx11.migadu.com X-TUID: V0fgodh3Lx+y On Sun, Mar 17, 2024 at 09:30:34PM -0300, Carlos Maniero wrote: > > + case '=': { > > + size_t start_offset = lexer->offset; > > + > > + if (lexer_peek_next_char(lexer) == '=') { > > + lexer_skip_char(lexer); > > + lexer_skip_char(lexer); > > + lexer_init_str_value_token(lexer, token, TOKEN_CMP_EQ, start_offset); > > + return; > > + } > > + > > + lexer_init_char_value_token(lexer, token, TOKEN_EQ); > > + lexer_skip_char(lexer); > > + return; > > + } > > We could apply the *maximal munch* [1] here avoiding looking to the > future by skipping the char right after founding it: Thanks, I kind agree on removing the necessity of lookahead here. > > + case '=': { > + size_t start_offset = lexer->offset; > + lexer_skip_char(lexer); > + > + if (lexer_current_char(lexer) == '=') { > + lexer_skip_char(lexer); > + lexer_init_str_value_token(lexer, token, TOKEN_CMP_EQ, start_offset); > + return; > + } > + > + lexer_init_char_value_token(lexer, token, TOKEN_EQ); This function expect the offset being point to the char in question. In your solution we already moved the offset. I solution for this problem would replace the /lexer_init_char_value_token/ with /lexer_init_str_value_token/. I have sent a new revision for this patch with the correction. > + lexer_skip_char(lexer); This solution is moving twice to consume a single char. > + return; > + } > > I know it is a silly change, but this help us to keep the parsing > process as deterministic as possible. I don't think it's a silly change, that's a nice improvement, thanks for suggesting this change.