Re: [RFC PATCH olang v1] docs: create zero programming language specification

Re: [RFC PATCH olang v1] docs: create zero programming language specification

[Ricardo Kagawa]

> You've replied to the CI build reply.  Next time try to reply to the
> right thread.

I just opened the "Reply to thread" link from sourcehut's web interface.
It automatically filled the TO, CC, Subject and some thread ID header,
which values I just trusted. It seems the UI is not to be trusted, but
then, which values should I use? The address in TO seems to have bounced
my reply, so I thought it didn't even make it to sourcehut.

> Your message has few weird line breaks.

I am not myself sure, but I suspect it is an issue with the file format
generated by `vim`. My default `email` file type seems to be forcing the
`dos` format (CRLF line breaks), which might be being interpreted as two
separate line breaks somewhere between Thunderird, Gmail and sourcehut.
I'll see if I can force it to use the `unix` format (LF only) and if
that fixes things at all.

> I'm not sure how you want to version lock this variant.  Should I add
> a specific github/git tag version to the document?

Yes, sort of. The web tool itself cannot be version locked, since it
simply does not have that option, but it does link to its GitHub
project. The project does not itself contain the description of its
EBNF syntax, but it does have a link to what it uses to implement its
EBNF parser, which in turn describes its syntax. You could include a
version locked link to [that][1].

[1]: https://github.com/Engelberg/instaparse/tree/v1.4.12

> > Is the language going to support Unicode?
>
> I would say to keep it simple as much as we can on this earlier stage
> (ASCII only) unless you have a big concern.

I guess that would be OK. I don't think it would be too difficult to
migrate later. Maybe tricky, but not difficult, since Unicode is a
superset of ASCII. Just need to be careful not to depend too much on
the fact that ASCII characters are stored in 8-bit variables, as
Unicode uses variable-length characters (variable within a string, but
characters are multiples of 8 bits).

> If we don't add a token in here like **=** it will be very weird.

Actually, I mentioned Kotlin also to imply that there would be an
equals sign before the expression.

> > - I have not checked if this syntax would avoid that edge case with
> >   JavaScript I mentioned in the beginning. I might check that next
> >   time (I'm still not sure of how).
>
> Maybe we are going to discovery it on the implementation process.

I _suspect_ it would be enough to give precedence to interpreting line
breaks as end-of-statement, and if so, there might be a way to represent
that precedence in the EBNF grammar (by convention). I would still need
to mull over it for a while to be sure.

Another revision:

- Function body now accepts a single expression.
     - This introduced the `<end-of-file>` token, which is not an actual
       sequence of characters. It allows the function body expression at
       the end of the program without a following line break or
       semicolon. Earlier declarations must include a line break or
       semicolon.
- `\v` (vertical tab) and `\f` (form feed) included as line breaks for
   completeness over ASCII (based on `\s` regex class, which agrees with
   Unicode properties over the ASCII range).
- Integer literals can now include underlines as separators.
     - The literal is allowed to terminate with an arbitrarily long
       sequence of separators, though.
     - It would be possible to restrict the last character to be a digit,
       but maybe it is not worth the trouble?
- Introducing hexadecimal integer literals.
     - Hexadecimal literals are allowed to have an arbitrarily long
       sequence of zeroes to the left, after the `0x` prefix. This is
       intentional, and the parser should ignore excess zeroes.
     - The alphabetic characters in the literal should be parsed without
       case-sensitivity. There is no difference between the `0x` and `0X`
       prefixes in literals. If mixed case is not desirable, let the
       linter or formattter restrict that.

```
(* Entry Point *)
<program>             ::= <ows> <function-definition> <ows> <end-of-file>

(* Functions *)
<function-definition> ::= 'fn' <ws> <function-name> <ows> 
<function-parameters> <ows> ':' <ows> <return-type> <ows> <function-body>
<function-name>       ::= <identifier>
<function-parameters> ::= '(' <ows> ')'
<return-type>         ::= <type>
<function-body>       ::= <block> | '=' <ows> <expression> <ows> 
(<end-of-statement> | <end-of-file>)

(* Statements *)
<block>               ::= '{' <ows> <statement> <ows> 
(<end-of-statement> <ows> <statement> <ows>)* <end-of-statement>? <ows> '}'
<end-of-statement>    ::= ';' | <line-break>
<statement>           ::= <return-statement>
<return-statement>    ::= 'return' <ws> <expression>

(* Expressions *)
<expression>          ::= <integer>

(* Identifiers *)
<type>                ::= 'u32'
<identifier>          ::= (<alpha> | '_') (<alpha> | <digit> | '_')*

(* Literals *)
<integer>             ::= <integer-base10> | <integer-base16>
<integer-base10>      ::= #'[1-9]' (<digit> | '_')* | '0'
<integer-base16>      ::= #'0[Xx]' <hex-digit> (<hex-digit> | '_')*

(* Utilities *)
<ws>                  ::= <white-space>+
<ows>                 ::= <white-space>*
<white-space>         ::= <linear-space> | <line-break>
<line-break>          ::= #'[\n\v\f\r]' | '\r\n'
<linear-space>        ::= #'[ \t]'
<alpha>               ::= #'[a-zA-Z]'
<digit>               ::= #'[0-9]'
<hex-digit>           ::= <digit> | #'[a-fA-F]'
<end-of-file>         ::= #'$'
```

Re: [RFC PATCH olang v1] docs: create zero programming language specification

[Carlos Maniero]

I'd like to begin by echoing Johnny's words of thanks. It's truly
fantastic to see more people getting involved in making olang a
remarkable language.

It's also quite refreshing to have someone on board who is well-versed
in the theory behind creating a programming language. I'm certain that
I, along with others, will learn a great deal from you.

I'll start with a question from your first reply:

> My limited understanding is that the semicolon would indeed be more
> convenient, as it would be a definitive end-of-statement symbol,
> requiring no lookahead to resolve as such. The LF token could be
> ambiguous on its own (between end-of-statement and white space), so
> some lookahead would be required to resolve it.

I had a hard time trying to understand why the "LF token could be
ambiguous on its own", but now I got it. Before, I was just thinking
about the function body where, to me, a blank line in the function's
body is just an empty statement, same as a sequence of semicolons in C.
But I wasn't considering that there are a lot of other places where the
programmer could add a blank space, like in function declarations:

  fn
  main()
  :
  u32
  {
    return 0
  }

Even though the code above is ugly AF, it's still syntactically correct.
And that made me realize why it's more convenient to have an
end-of-statement token. I'm just putting this out here in case someone
else has the same doubt. Although, it seems like we are  all on the same
page about wanting to make the language more user-friendly, even if it
means giving the parser a bit of a hard time.

About the "some lookahead would be required to resolve it" we definitely
added some of these in the parser, but given a better look, we could
easily get rid of then by replacing the function:

  static void
  skip_line_feeds(lexer_t *lexer)
  {
      token_t token;
      lexer_peek_next(lexer, &token);

      while (token.kind == TOKEN_LF) {
          lexer_next_token(lexer, &token);
          lexer_peek_next(lexer, &token);
      }
  }

With:

  static void
  next_non_lf_token(lexer_t *lexer, token_t *token)
  {
      do {
          lexer_next_token(lexer, token);
      } while (token->kind == TOKEN_LF);
  }

I'm sure it may be some corner cases where it cannot be applied, but I
think that it reduces backtrack for most of the cases.


> - Function body now accepts a single expression.
> ...
> - `\v` (vertical tab) and `\f` (form feed) included as line breaks for
>    completeness over ASCII (based on `\s` regex class, which agrees with
>    Unicode properties over the ASCII range).
> - Integer literals can now include underlines as separators.
> ...
> - Introducing hexadecimal integer literals.
> ...

>From your last email, I'm totally up for discussing these topics. But I
reckon we might want to split them into new threads to avoid trying to
hash out the entire language in one go.

What do you all think about us trying to:

1. Nail down the current state of the language, leaving new features for
   later.
2. Figure out how we're gonna document new features.
3. Kick off a new thread for each feature we're thinking of adding.

Sound good?

Re: [RFC PATCH olang v1] docs: create zero programming language specification

[Johnny Richard]

Amazing job. Wonderful changes and thank you so much for you
contribution.

On Fri, Mar 15, 2024 at 05:54:10PM -0300, Ricardo Kagawa wrote:
> > You've replied to the CI build reply.  Next time try to reply to the
> > right thread.
> 
> I just opened the "Reply to thread" link from sourcehut's web interface.
> It automatically filled the TO, CC, Subject and some thread ID header,
> which values I just trusted. It seems the UI is not to be trusted, but
> then, which values should I use? The address in TO seems to have bounced
> my reply, so I thought it didn't even make it to sourcehut.

Sad to hear the Sourcehut reply is miss behaving.  Could you please
report the problem to them?

The good practice is always **reply all** (in case someone who is not
subscribed to the mailing list can follow up the thread discussion).
Since I wrote the email you replied, you should reply it to me and
**CC** the mailing list and carry the **CC** from previous Email (reply
already behaves like so).  You might want to have a copy (optional), in
that case you can **BCC** yourself.

In order to have set up correctly the thread you want to reply to, you
have to set the **In-Reply-To** header with the **Message-ID** of the
message you want to reply to.

It seems to be very hard to active but your email client should do the
work.  You can download and import the email thread from the archive
right into your email client and **reply all**. 

There is another mailing list archive[1] which has instruction on how to
reply to emails using **git-send-email** that maybe you might be interested.

[1]: http://lists.johnnyrichard.com/olang/

For example, you could follow the steps bellow to reply to the right
thread using **git-send-email**:

    Reply instructions:

    You may reply publicly to this message[2] via plain-text email
    using any one of the following methods:

    * Save the following mbox file, import it into your mail client,
      and reply-to-all from there: mbox[3]

      Avoid top-posting and favor interleaved quoting:
      https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

    * Reply using the --to, --cc, and --in-reply-to
      switches of git-send-email(1):

      git send-email \
        --in-reply-to=y2brhif5vjjbjmd26wwxykhlt2qnnm2k7n2lpiz7dr7pccmoc7@qqx67mvyzl2c \
        --to=johnny@johnnyrichard.com \
        --cc=ricardo.kagawa@gmail.com \
        --cc=~johnnyrichard/olang-devel@lists.sr.ht \
        /path/to/YOUR_REPLY

      https://kernel.org/pub/software/scm/git/docs/git-send-email.html

    * If your mail client supports setting the In-Reply-To header
      via mailto: links, try the mailto: link[4]

    Be sure your reply has a Subject: header at the top and a blank line before the message body. 

[2]: http://lists.johnnyrichard.com/olang/y2brhif5vjjbjmd26wwxykhlt2qnnm2k7n2lpiz7dr7pccmoc7@qqx67mvyzl2c/#t
[3]: http://lists.johnnyrichard.com/olang/y2brhif5vjjbjmd26wwxykhlt2qnnm2k7n2lpiz7dr7pccmoc7@qqx67mvyzl2c/raw
[4]: mailto:johnny@johnnyrichard.com?In-Reply-To=%3Cy2brhif5vjjbjmd26wwxykhlt2qnnm2k7n2lpiz7dr7pccmoc7@qqx67mvyzl2c%3E&Cc=ricardo.kagawa%40gmail.com%2C~johnnyrichard%2Folang-devel%40lists.sr.ht&Subject=Re%3A%20%5Bolang%2Fpatches%2F.build.yml%5D%20build%20success

> > Your message has few weird line breaks.
> 
> I am not myself sure, but I suspect it is an issue with the file format
> generated by `vim`. My default `email` file type seems to be forcing the
> `dos` format (CRLF line breaks), which might be being interpreted as two
> separate line breaks somewhere between Thunderird, Gmail and sourcehut.
> I'll see if I can force it to use the `unix` format (LF only) and if
> that fixes things at all.

I think you made it, this email looks good.  Perhaps you can set the vim
as your default editor on Thunderird (I know you can do it but not sure
how). Gmail should not mess up with you message.

If you like to live on terminal I suggest **mutt** or **aerc** clients
which works out of the box with plain text emails and they are very flexible.

> > I'm not sure how you want to version lock this variant.  Should I add
> > a specific github/git tag version to the document?
> 
> Yes, sort of. The web tool itself cannot be version locked, since it
> simply does not have that option, but it does link to its GitHub
> project. The project does not itself contain the description of its
> EBNF syntax, but it does have a link to what it uses to implement its
> EBNF parser, which in turn describes its syntax. You could include a
> version locked link to [that][1].
> 
> [1]: https://github.com/Engelberg/instaparse/tree/v1.4.12

Sounds good.

> > > Is the language going to support Unicode?
> > 
> > I would say to keep it simple as much as we can on this earlier stage
> > (ASCII only) unless you have a big concern.
> 
> I guess that would be OK. I don't think it would be too difficult to
> migrate later. Maybe tricky, but not difficult, since Unicode is a
> superset of ASCII. Just need to be careful not to depend too much on
> the fact that ASCII characters are stored in 8-bit variables, as
> Unicode uses variable-length characters (variable within a string, but
> characters are multiples of 8 bits).

Nice, let us know whenever we make such ASCII dependency. 

> > If we don't add a token in here like **=** it will be very weird.
> 
> Actually, I mentioned Kotlin also to imply that there would be an
> equals sign before the expression.
> 
> > > - I have not checked if this syntax would avoid that edge case with
> > >   JavaScript I mentioned in the beginning. I might check that next
> > >   time (I'm still not sure of how).
> > 
> > Maybe we are going to discovery it on the implementation process.
> 
> I _suspect_ it would be enough to give precedence to interpreting line
> breaks as end-of-statement, and if so, there might be a way to represent
> that precedence in the EBNF grammar (by convention). I would still need
> to mull over it for a while to be sure.

I can test the precedence solution later as well.

> Another revision:
> 
> - Function body now accepts a single expression.

Hmm... I think we still haven't agreed on having it.  We are trying to
make a very simple language and maybe avoid having multiple ways of
achieving the stuff will make it simpler.

> - Introducing hexadecimal integer literals.
>     - Hexadecimal literals are allowed to have an arbitrarily long
>       sequence of zeroes to the left, after the `0x` prefix. This is
>       intentional, and the parser should ignore excess zeroes.
>     - The alphabetic characters in the literal should be parsed without
>       case-sensitivity. There is no difference between the `0x` and `0X`
>       prefixes in literals. If mixed case is not desirable, let the
>       linter or formattter restrict that.

Nice you added support to hex. We will need octal in the future as well
(starting with 0).  But let's craft more features in new patches, this
thread is become out of control.  Perhaps we can merge an initial
version and propose small changes to it (making new discussion threads).

I can set up a PATCH with most of your changes (except the function
definition with single line, we can discuss it on a new PATCH thread). 

Re: [RFC PATCH olang v1] docs: create zero programming language specification

[Carlos Maniero]

Re: [RFC PATCH olang v1] docs: create zero programming language specification

[Johnny Richard]

A patch v2 has been sent, this patch is SUPERSEDED.

Re: [RFC PATCH olang v1] docs: create zero programming language specification

[Carlos Maniero]

> This grammar adds the token SEMICOLON (';') for every statement.  I know we
> agreed make it optional, but the SEMICOLON makes the parser much more
> convenient to implement.
> 
> And this is the first topic I would like to discuss. Let me know if you
> agree otherwise I can adapt the grammar to make SEMICOLON optional.

I understand that using line feed (LF) as an end of statement (EOS)
marker can simplify the parsing process. However, at the top of my mind
I see two main downsides in using LF as EOS:

1. It could increase the number of tokens that the lexer produces,
   requiring the parser to ignore some of them.
2. It requires an strategy to skip LF for enable multi-line statements.

While the second point does add some complexity, I believe the first one
only slightly alters the implementation without significantly increasing
its complexity. Therefore, I'm curious about your statement that using a
semicolon makes the parser much more convenient to implement. Could you
elaborate on this? Have you encountered any new considerations that might
complicate the implementation?

> The grammar was made by using a EBNF evaluator tool[1].
> 
> [1]: https://mdkrajnak.github.io/ebnftest/

I would add this link at the markdown, so then people can play with it.

Furthermore, I was playing with the grammar you send and I did a few
adjusts:

- I introduced the <end-of-statement> to make SEMICOLON optional;
- It seams it has a pattern in where you used DASH or UNDERLINE but IMO
  it reads better when standardized.
- I made a simplification on <identifier>.
- I reordered the non-terminal keeping the single chars and strings at
  the bottom.

  <program>               ::= <function-definition>
  <function-definition>   ::= <fn-keyword> <space>+ <identifier>+ <space>* <fn-args> <space>* <colon> <space>+ <type> <space>* <block>
  <identifier>            ::= <alpha>+ <number>* <identifier>*
  <fn-args>               ::= '(' <space>* ')'
  <block>                 ::= <ocurly> <space>* <statement> <space>* <ccurly>
  <statement>             ::= <return-statement> <end-of-statement>+
  <return-statement>      ::= <return-keyword> <space>* <number>+ <space>*
  <end-of-statement>      ::= <semicolon>* <space>* '\n'
  <fn-keyword>            ::= 'fn'
  <return-keyword>        ::= 'return'
  <type>                  ::= 'u32'
  <alpha>                 ::= #'[a-zA-Z_]'
  <number>                ::= #'[0-9]'
  <space>                 ::= #'[ \t\r\n]'
  <semicolon>             ::= ';'
  <ocurly>                ::= '{'
  <ccurly>                ::= '}'
  <colon>                 ::= ':'

Other than that I would like to say thank you to always been looking
forward and helping us to improve our process.

Re: [RFC PATCH olang v1] docs: create zero programming language specification

[Johnny Richard]

I messed with the grammar, here I have few adjusts.

On Sat, Mar 09, 2024 at 01:05:39AM +0100, Johnny Richard wrote:
> diff --git a/docs/pages/language_specification.md b/docs/pages/language_specification.md
> new file mode 100644
> index 0000000..9d27eda
> --- /dev/null
> +++ b/docs/pages/language_specification.md
> @@ -0,0 +1,41 @@
> +zero programming language specification
> +=======================================
> +
> +ABSTRACT
> +--------
> +
> +This document specifies the semantics and behavior of the Zero Programming
> +Language for compiler programmers be informed how the language is designed.

s/be informed how/be informed on how/

> +This specification is on DRAFT and will evolve through discussions on olang-dev

s/is on DRAFT/is a DRAFT/

> +<identifier>            ::= <alpha>+
> +                          | <alpha>+ <number>*

I missed the recusion here, it should be:

    <identifier> ::= <alpha>+
                   | <alpha>+ <number>* <identifier>
                   ;

> +<type>                  ::= 'u32 '

s/u32 /u32/

[RFC PATCH olang v1] docs: create zero programming language specification

[Johnny Richard]

This document specifies the semantics and behavior of the Zero Programming
Language for compiler programmers be informed how the language is designed.

This document will help newcomers to understand how the language looks
like and as a DRAFT guide on the language design discussions.

The grammar was made by using a EBNF evaluator tool[1].

[1]: https://mdkrajnak.github.io/ebnftest/

Signed-off-by: Johnny Richard <johnny@johnnyrichard.com>
---
This grammar adds the token SEMICOLON (';') for every statement.  I know we
agreed make it optional, but the SEMICOLON makes the parser much more
convenient to implement.

And this is the first topic I would like to discuss. Let me know if you
agree otherwise I can adapt the grammar to make SEMICOLON optional.

 docs/pages/language_specification.md | 41 ++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)
 create mode 100644 docs/pages/language_specification.md

diff --git a/docs/pages/language_specification.md b/docs/pages/language_specification.md
new file mode 100644
index 0000000..9d27eda
--- /dev/null
+++ b/docs/pages/language_specification.md
@@ -0,0 +1,41 @@
+zero programming language specification
+=======================================
+
+ABSTRACT
+--------
+
+This document specifies the semantics and behavior of the Zero Programming
+Language for compiler programmers be informed how the language is designed.
+
+This specification is on DRAFT and will evolve through discussions on olang-dev
+mailing list.
+
+Language Syntax
+---------------
+
+This is the Zero Programming Language EBNF grammar specification 
+
+NOTE: This grammar spec is a DRAFT and it covers only a small portion of the
+language.
+
+```
+<program>               ::= <function-definition>
+<function-definition>   ::= <fn_keyword> <space>+ <identifier> <space>* <f-args> <space>* <colon> <space>* <type> <space>* <block>
+<identifier>            ::= <alpha>+
+                          | <alpha>+ <number>*
+                          ;
+<f-args>                ::= '(' <space>* ')'
+<block>                 ::= <ocurly> <space>* <statement>* <space>* <ccurly>
+<statement>             ::= <return-statement>
+<return-statement>      ::= <return_keyword> <space>* <number>* <space>* <semicolon>
+<semicolon>             ::= ';'
+<ocurly>                ::= '{'
+<ccurly>                ::= '}'
+<type>                  ::= 'u32 '
+<colon>                 ::= ':'
+<alpha>                 ::= #'[a-zA-Z_]'
+<number>                ::= #'[0-9]'
+<fn_keyword>            ::= 'fn'
+<return_keyword>        ::= 'return'
+<space>                 ::= #'[ \t\r\n]'
+```
-- 
2.44.0