Re: [RFC] Namespaces in OLANG

["Carlos Maniero" <carlos@maniero.me>]

I confess that this is my last attempt to explain what I'm
suggesting. I believe neither you and me have energy to continue
discussing this subject. So if you don't get it I believe we could just
drop it.

> > > > - It simplifies the resolution of imports.
> > >
> > > I would suggest to not go much further with import resolution (unless
> > > you already want to define modules).  Perhaps we could have namespace
> > > doing nothing else than namespacing...
> > 
> > If by "modules" you are referring to the file level, and not to
> > something like packages or libraries, then that is exactly what I want
> > to define! Influenced by Clojure, I recommended calling it a
> > "namespace". However, I believe that naming it ‘mod' or ‘module' is more
> > suitable for its purpose.
> > 
> >   mod olang.core.math
> > 
> >   fn add(a: u32, b: u32) {
> >     return a + b
> >   }
>
> Not sure if I understand what you meant here. Maybe would be better
> clarify what you understands as module.  This concept is still blurry to me.

module = namespaced files.

All symbols inside a file will share the same namespace implying the
symbols of the object file will start with the name you gave to the
module. I gave some examples bellow.

> > > > These advantages come with the minor stipulation of initiating all files with a
> > > > namespace statement, which, in my view, is a small price to pay for the
> > > > benefits gained.
> > >
> > > I'm not keen on the idea of enforcing strict adherence to the folder
> > > structure.
> > >
> > > How about we introduce a namespace block instead? Within this
> > > block, everything would automatically have the namespace added as a
> > > prefix. This could offer more flexibility while still maintaining
> > > organization.
> > 
> > Don't you think that in practice almost every single file will
> > namespace? C++ follows this pattern, and look at this Qt mirror [1], 6k
> > files, all namespaced, they even created a macro to facilitate the
> > work.
> > 
> > [1] https://github.com/search?q=repo%3Aradekp%2Fqt+%2FQT_BEGIN_NAMESPACE%5Cn%2F&type=code&p=1
>
> I prefer being explicit in this case (even if we need to write it for
> every file).  If you land on a random file, you can say for what the
> binary the code will translate to without checking the folder structure.
> (Dir tree organization is completely up to the developers). 

The approach I suggested allows you to explicitly declare the module in
each file. This way, anyone who opens a file can immediately understand
what the code will translate to, without having to check the folder
structure.

  1. mod olang.core.math # <--- THIS LINE IS PART OF THE FILE
  2. 
  3. fn add(a: u32, b: u32) {
  4.   return a + b
  5. }

I never suggested we should automatically create namespaces based on the
file structure. Actually I described this approach in the Alternative
section "1. Automatically create namespaces based on the filename" which
I may not make it clear enough, but I dislike a lot.

> If you land on a random file, you can say for what the
> binary the code will translate to without checking the folder structure

In the code organization example I provided, the first line of each file
explicitly declares the module. Just check the first line and done! Just
like java/golang package.

> (Dir tree organization is completely up to the developers). 

And they are!

Let's say you have this code organization:

  - shared/
    - arena.ol {mod arena}
    - list.ol {mod list}
    - hashmap.ol {mod hashmap}
  - fe/
    - ast {mod fe.ast}
    - parser {mod fe.parser}
  - main.ol {mod main}

Note that everything inside {} is the first line of the file. To compile
this program we could use the follow command line:

  olang -I shared/ -o main main.ol

Considering the *main.ol* start with following lines:

  mod main

  import arena
  import list
  import hashmap
  import fe.ast
  import fe.parser

The olang compiler can automatically find the files inside the *fe*
directory, because they match the file structure. But *arena*, *list*,
and *hashmap* do not follow the file structure, so you should specify
the directory where you can find them.

The only convention is that dots in the module name mean slashes in the
path location. This is required to avoid the ambiguity of having slashes
meaning both division and module names.

Although, In my opinion, if a developer creates a file named
*veggies/zucchini.ol* and attempts to declare the module as *mod
veg.zuc*, it would lead to confusion and errors. This is because the
module name does not align with the file structure.

A practical solution in this scenario would be to rename the file to
*veg_zuc.ol* and declare the module as *mod veg_zuc*. This way, the
module name matches the filename, making it easier to understand and
manage.

When compiling, you would specify the directory of *veg_zuc.ol* in the
olang command. The generated symbols would remain largely the same (once
*veg.zuc* translates to *veg_zuc*), maintaining the integrity of the
structure.

In practice is up to you how you use modules. You can create modules
matching your entire project structure and the compiler will make the
work without any extra argument, or you can create any structure you
want and declare the include dir.

> > > I think module has a different meaning.  If you want to have modules,
> > > for sure we have to discuss import resolution.  IMHO namespace shouldn't
> > > do anything else than namespacing.
> > 
> > I believe you're right. It's almost impossible to discuss modules
> > without bringing up imports. To me, the way C handles this is one of the
> > most painful things in my life (hehe).
> > 
> > The main issue is that I never know where something is coming from,
> > which is especially painful when I'm trying to replicate something I've
> > already done. This also, often leads to unused includes over time
> > because it's hard to determine if an include is actually being used.
>
> Could you please enlighten me the unused definitions problem?  Does it
> have any implications with performance?
>
> If this is so bad, why we cannot make the compiler smart enough to
> detect them?

I don't know what is the implication in terms of performance. I guess
the object file may be larger if the header file has more than
prototypes. But if you find unused includes no big deal in code
maintenance, I have nothing else to say.

> > If we abandon modules and just go with C++-like namespaces, I believe we
> > we may want to endure C's painful include system. This is because the
> > language won't have control over function names. The way the include
> > system is designed sends a message to developers that including a file
> > is akin to concatenating all the definitions into a single file. But
> > yet I think it would be ok to have names imports even if we don't
> > control the language names but it would be just a semantic tool.
> > 
> > Named Imports
> > -------------
> > 
> >   mod myprog
> > 
> >   import olang.core.math
> > 
> >   fn main(): u32 {
> >     return olang.core.math::sum(1, 2)
> >   }
> > 
> > And even associate identifiers to it.
> > 
> >   mod myprog
> > 
> >   import olang.core.math as ocm
> > 
> >   fn main(): u32 {
> >     return ocm::sum(1, 2)
> >   }
> > 
> > Note that there is no actually difference in between mangling and my
> > module purpose, except the fact modules generates deterministic and
> > friendly names that can be easily used in C and also easy to gen code,
> > once to generate the assembly symbol of *olang.core.math::sum* we can
> > just replace dots by underscores and double column to double
> > underscores.
> > 
> > External Linkage
> > ----------------
> > 
> > We probably don't wanna to make all function global for external
> > linkage.
>
> Could you please clarify what you mean with external linkage?

  When you write an implementation file (.cpp, .cxx, etc) your compiler
  generates a translation unit. This is the source file from your
  implementation plus all the headers you #included in it.

  Internal linkage refers to everything only in scope of a translation
  unit.

  External linkage refers to things that exist beyond a particular
  translation unit. In other words, accessible through the whole program,
  which is the combination of all translation units (or object files).

  https://stackoverflow.com/questions/1358400/what-is-external-linkage-and-internal-linkage

But looking at you answer above, it seems you understood the concept.

> > So we may need a visibility keyword.
>
> The C already has this functionality, the *static* keyword...
> Everything else is "public" by default.  Do you see a problem on
> following the same pattern?

I prefer everything to be private by default to prevent leaking of
implementation details which leads in less symbols in the object file.

> > And to me, everything that can be imported should also be available
> > for external linkage, even if we decide to do not generate an object
> > file per module. I would recommend the usage of *export* or *pub*. I
> > like *export* better.
>
> Why this has to be so complex?  Seems like we would have to implement a
> very complex system for compile a program with multiple files.  I would
> love to have something very simple where you compile down to a `.o` file
> and the linking is done simple as GAS or GCC does.

I said and I quote: 

> > ... even if we decide to do not generate an object
> > file per module

"EVEN IF". I'm not suggesting us to do anything other than create a *.o*
file. I'm not suggesting anything complex I don't know what makes you
understand that.

Basically what I said is: Let's do not make anything public by default.

I just mentioned mentioned that we may don't wanna to generate an object
file per module because olang has no header file. Meaning that we we
always need to examine all files of the project anyway. But it just
created more noise at the discussion, so you can ignore it.

>
> > Note that this is required no meter how we decide to handle imports.
> > 
> > No mangling
> > -----------
> > 
> > But what if I really need something to have the exact name? Lets say you
> > wanna integrate with a bootloader that is integrated on the link process and
> > expects a symbol called *kmain* to jump into?
> > 
> > Ok, I admit, in that case namespaces it is gonna be a pain in the ass. But
> > the good news is that these are exceptional, you don't need this for
> > your entire application but only for a few functions.
> > 
> > I was wondering that we could have a *global* keyword where everything
> > that is global assumes its own name and is always have public visibility.
>
> You are assuming that all files has a mandatory namespace.  I still
> prefer the namespace being optional.  The default behavior would be
> every symbol wont have prefix. (1 to 1) with ELF binary symbols.
>
> IMHO this is what makes the language simple as C.  Adding extra
> functionality to protect the programming doing stuff and enabling
> functionality with a lot of gymnastics feels like an unpleasant
> experience.

You gonna need to make a lot of gymnastic. Actually we do a lot of
gymnastic every single time we prefix a C function to avoid conflicts.
And many projects create macros to do the same job.

I'm purposing that the language should provide the appropriated tooling
for it. But I agree that we could make modules optional and you could
provide the include directory in this case.

> > Summarizing
> > -----------
> > 
> > We have a few options in the table.
> > 
> > 1. Use the names the way they are. (C approach)
> > 
> > Pros:
> > - Simple, no magic, it is what it is.
> > - Easy to produce debug info since the assembly symbol will be the
> >   function name.
>
> - No build gymnastics as well. The *make* program would solve our
>   problems for free (Even with partial compilation).

No gymnastics in the modules suggestion. You can use make. You can make
partial compilation. Never purpose the oppose.

> > 
> > Cons:
> > - More challenge to keep the code out of name conflicts in large
> >   codebases.
> > - Requires developers to manually namespace functions.
> > 
> > 2. Use mangled names (C++, Rust approach)
>
> The mangled names are basically to enable function overload.  I think we
> don't want this functionality on the language.  I think we can ignore
> this one.

It is used to "enable function overload" but it isn't the only thing it
solves. By mangled names I mean any strategy that generates names to
avoid conflicts. It could even be a random string.

  Name mangling

  The names used in object file symbol tables and in linking are often not
  the same names used in the source programs from which the object files
  were compiled. There are three reasons for this: avoiding name
  collisions, name overloading, and type checking. The process of turning
  the source program names into the object file names is called name
  mangling. This section discusses mangling typically done to names in C,
  Fortran, and C++ programs.

  https://archive.ph/20130126113557/http://www.iecc.com/linker/linker05.html#selection-481.0-487.90

>
> > 3. Use modules (Zig approach (I think))
> > 
> > Pros:
> > - Keep the code out of naming conflicts
> > - Deterministic assembly symbols permit integrate with C without any
> >   magic, you just need to follow the convention ns__fn.
> > 
> > Cons:
> > - If you really need to have a specific name for your function you gonna
> >   need a no-mangle approach.
>
> Why? I mean you can still have the namespace and the freedom to use
> namespace where you want.  Are you assuming the namespace is mandatory
> here?

As I answered early in this thread a module is a namespaced file. If you
need a function called *zucchini* and this function lies inside a file
*veggies*/mod *veggies*, the *zucchini* file inside the object file
will be called *veggies__zucchini*.