Re: [olang/patches/.build.yml] build success

[Ricardo Kagawa <ricardo.kagawa@gmail.com>]

 >> This grammar adds the token SEMICOLON (';') for every statement.  I 
know we

 >> agreed make it optional, but the SEMICOLON makes the parser much more

 >> convenient to implement.

 >>

 >> And this is the first topic I would like to discuss. Let me know if you

 >> agree otherwise I can adapt the grammar to make SEMICOLON optional.

 >

 > (...) Therefore, I'm curious about your statement that using a

 > semicolon makes the parser much more convenient to implement. Could you

 > elaborate on this? Have you encountered any new considerations that might

 > complicate the implementation?



My limited understanding is that the semicolon would indeed be more

convenient, as it would be a definitive end-of-statement symbol,

requiring no lookahead to resolve as such. The LF token could be

ambiguous on its own (between end-of-statement and white space), so

some lookahead would be required to resolve it.



But it should be alright, as long as the language remains context-free.

Even if it becomes ambiguous, non-deterministic, or requires a long

lookahead. Ideally it should be determinitstic for linear time

performance, but it seems there are parsers that can run close to it in

the average case, as long as the language remains close to

deterministic.



And I don't have a strong opinion on the semicolon issue, except that

it must be an option. But whatever we do, we must avoid the following

pitfall from JavaScript:



```javascript

example

;(x)

```



The semicolon is mandatory here, because otherwise `(x)` is handled as

an argument list, and `example` would be called as a function. That is,

it would be a multi-line statement, instead of two separate statements.



And why anyone would do this?



```javascript

const x = y.example

;(() => {

   console.log(x)

})()

```



Immediately invoked function expressions are a thing in JavaScript, and

it would not be uncommon to have some expression ending with an

identifier right before them.



 >> The grammar was made by using a EBNF evaluator tool[1].

 >>

 >> [1]: https://mdkrajnak.github.io/ebnftest/

 >

 > I would add this link at the markdown, so then people can play with it.



I would make an even stronger argument for including the link in the

docs. A good language specification also specifies which language

specification grammar is used for the specification itself. And the

EBNF in particular is not properly standardized, so you really need to

specify which EBNF variant you are using.



The link should thus be good enough to refer to the EBNF implementation

used in this specification, although a permanent (version locked) link

would be better.



----

As for my revision of the grammar:



- Separated rules into sections.

- Added optional white space around the program.

- You don't actually need non-terminal symbols for keywords. Especially

   if you are including the keyword in the symbol name.

- You don't need non-terminal symbols for symbols either, unless you

   have a more "semantic" name for it. There should not be another

   "semicolon" besides `;`, for example.

- In Johnny's version the function name is a single identifier. I don't

   know why Carlos's version made it multiple. I have made it single

   again.

- In Johnny's version the space before the return type is optional. I

   don't know why Carlos's version made it mandatory. I have made it

   optional again.

- Replaced `<identifier>` in `<function-definition>` with

   `<function-name>` to express that this identifier is the name of the

   declared function. Then, `<function-name>` is just `<identifier>`.

- Renamed `<fn-args>` to `<function-parameters>`, since parameters are

   the variables in a function declaration, while arguments are the

   values bound to those variables during function calls.

- Replaced `<type>` for `<return-type>` in `<function-declaration>` to

   express that this type identifier is the return type of the function.

   Then, `<return-type>` is just `<type>`.

- Replaced `<block>` in `<function-definition>` for `<function-body>` to

   express that this block is the body of the declared function.

- Reworked `<block>`, `<statement>` and `<end-of-statement>` to allow

   for:

     - Single statement followd by optional end-of-statement;

     - Statement list with mandatory end-of-statement between statements;

     - But the statements could be made optional, yet I did not in this

       version, as there is no `void` return type, currently.

- Replaced `<number>` in `<return-statement>` with `<expression>` to

   prepare for them in the future. The only allowed expression is still

   an integer literal, though.

- Renamed `<number>` to `<integer>`, and reworked it to actually

   represent decimal integer literals. Sequences of zero digits are now

   forbidden at the left side, but a lone zero digit is still allowed.

- Reworked `<identifier>` to better express that it starts with

   `<alpha>` or underline, followed by zero or more `<alpha>`, `<digit>`

   or underline.

- Removed `_` from `<alpha>` to better reflect the name (as underline is

   not an alphabetic character).

- Renamed `<space>` for `<ws>` to avoid ambiguity with the character

   U+0020 Space, and made it a one-or-more list. Also introduced `<ows>`

   for "optional white space". Shorter names were preferred here due to

   these symbols in particular being used very frequently.

- Also introduced `<line-break>` as either LF, CR or CRLF. Otherwise the

   CRLF sequence would be parsed as two separate line breaks. Not that it

   would matter that much, except maybe for mapping line numbers.



```

(* Entry Point *)

<program>             ::= <ows> <function-definition> <ows>



(* Functions *)

<function-definition> ::= 'fn' <ws> <function-name> <ows> 
<function-parameters> <ows> ':' <ows> <return-type> <ows> <function-body>

<function-name>       ::= <identifier>

<function-parameters> ::= '(' <ows> ')'

<return-type>         ::= <type>

<function-body>       ::= <block>



(* Statements *)

<block>               ::= '{' <ows> <statement> <ows> 
(<end-of-statement> <ows> <statement> <ows>)* <end-of-statement>? <ows> '}'

<end-of-statement>    ::= ';' | <line-break>

<statement>           ::= <return-statement>

<return-statement>    ::= 'return' <ws> <expression>



(* Expressions *)

<expression>          ::= <integer>



(* Identifiers *)

<type>                ::= 'u32'

<identifier>          ::= (<alpha> | '_') (<alpha> | <digit> | '_')*



(* Literals *)

<integer>             ::= <integer-base10>

<integer-base10>      ::= #'[1-9]' <digit>* | '0'



(* Utilities *)

<ws>                  ::= <white-space>+

<ows>                 ::= <white-space>*

<white-space>         ::= <linear-space> | <line-break>

<line-break>          ::= '\n' | '\r' | '\r\n'

<linear-space>        ::= #'[ \t]'

<alpha>               ::= #'[a-zA-Z]'

<digit>               ::= #'[0-9]'

```



Further discussion:



- Is the language going to support Unicode? If so, `<alpha>` could use

   the _L:Letter_ Unicode category instead of being limited to

   `[a-zA-Z]`. But the EBNF tool does not support Unicode categories in

   its regular expressions (it does not support flags). Also don't

   forget to rename it to `<letter>` in that case.

     - It would help developers in non-English speaking countries, but it

       could be difficult to work with multi-byte characters and Unicode

       normalization.

- There are more linear space and line break characters than the ones

   included here, even within ASCII, although they are not all that

   important. Even more in Unicode (some under _Cc:Other/control_,

   others under _Z:Separator_). Should we support them?

- The function definition could accept a single expression as an

   alternative to its `<block>`, similar to Kotlin.

- The integer literal could include optional underline separators for

   readability. Just need to be careful not to start with underline, to

   avoid ambiguity with identifiers.

- I guess we don't have to support the full set of Unicode digits, since

   we don't know if these digits would even be decimal in the first

   place. The numbering system could be very different from our own, so

   it is likely not feasible to support them.

- I have not checked if this syntax would avoid that edge case with

   JavaScript I mentioned in the beginning. I might check that next

   time (I'm still not sure of how).

- It might seem strange that I included semantic non-terminals here,

   despite having removed non-terminals for symbols and keywords. I can't

   say for sure, since this is my first time trying this style, but I

   suspect that besides making the language specification easier to

   understand, the important bits to hook into in the parser will be

   around these symbols. That is, it could simplify some work on the

   parser.