[PATCH olang 0/4] comment based integration tests

[PATCH olang 0/4] comment based integration tests

[Carlos Maniero]

The old munit based integration tests was replaced by a new mechanism
that makes testing more tasteful and easy to perform.

To create an integration tests you just need to create a olang file at
*tests/integration/tests* directory. The assertions are performed
thought a couple of comments.

# spec: some text to describe what are you testing
# compiler_exit: the compilers exit code
# compiler_output: the compilers output, %empty% for empty output
# program_exit: the program exit code
# ast: the ast output
# tokens: the tokens

This patchset also includes parser tests errors and creates a TODO
mechanism that can be used to track what need to be implemented.

Carlos Maniero (4):
  tests: add comment based integration tests mechanism
  tests: remove previous integration tests structure
  tests: include integration tests for function parser errors
  tests: print integration tests TODOs

 Makefile                                      |   1 -
 tests/integration/Makefile                    |  33 +--
 tests/integration/cli_runner.c                | 100 ---------
 tests/integration/cli_runner.h                |  32 ---
 tests/integration/cli_test.c                  | 132 ------------
 tests/integration/proc_exec.c                 |  63 ------
 tests/integration/proc_exec.h                 |  37 ----
 tests/integration/test.sh                     | 195 ++++++++++++++++++
 tests/integration/tests/0001_main_exit.ol     |  27 +++
 .../tests/0002_fn_without_fn_keyword.ol       |   8 +
 .../integration/tests/0003_fn_without_type.ol |   8 +
 .../tests/0004_fn_without_type_colon.ol       |   8 +
 .../0005_fn_without_open_curly_brackets.ol    |   8 +
 .../0006_fn_without_close_curly_brackets.ol   |   7 +
 .../tests/0007_fn_without_return.ol           |   8 +
 15 files changed, 277 insertions(+), 390 deletions(-)
 delete mode 100644 tests/integration/cli_runner.c
 delete mode 100644 tests/integration/cli_runner.h
 delete mode 100644 tests/integration/cli_test.c
 delete mode 100644 tests/integration/proc_exec.c
 delete mode 100644 tests/integration/proc_exec.h
 create mode 100755 tests/integration/test.sh
 create mode 100644 tests/integration/tests/0001_main_exit.ol
 create mode 100644 tests/integration/tests/0002_fn_without_fn_keyword.ol
 create mode 100644 tests/integration/tests/0003_fn_without_type.ol
 create mode 100644 tests/integration/tests/0004_fn_without_type_colon.ol
 create mode 100644 tests/integration/tests/0005_fn_without_open_curly_brackets.ol
 create mode 100644 tests/integration/tests/0006_fn_without_close_curly_brackets.ol
 create mode 100644 tests/integration/tests/0007_fn_without_return.ol

-- 
2.34.1

[PATCH olang 1/4] tests: add comment based integration tests mechanism

[Carlos Maniero]

The old munit based integration tests was replaced by a new mechanism
that makes testing more tasteful and easy to perform.

To create an integration tests you just need to create a olang file at
*tests/integration/tests* directory. The assertions are performed
thought a couple of comments.

spec: some text to describe what are you testing
compiler_exit: the compilers exit code
compiler_output: the compilers output, %empty% for empty output
program_exit: the program exit code
ast: the ast output
tokens: the tokens

Since the language has the line_feed token it is impossible to test the
EOF once when added a comment the EOF will be located at line + 1. So
that the tokens test the first n-lines defined. The AST compares the
entire AST tough.

Signed-off-by: Carlos Maniero <carlos@maniero.me>
---
 Makefile                                  |   1 -
 tests/integration/Makefile                |  33 +---
 tests/integration/test.sh                 | 182 ++++++++++++++++++++++
 tests/integration/tests/0001_main_exit.ol |  27 ++++
 4 files changed, 217 insertions(+), 26 deletions(-)
 create mode 100755 tests/integration/test.sh
 create mode 100644 tests/integration/tests/0001_main_exit.ol

diff --git a/Makefile b/Makefile
index cdfc8e1..a0130f5 100644
--- a/Makefile
+++ b/Makefile
@@ -19,7 +19,6 @@ $(BUILD_DIR):
 .PHONY: linter
 linter: $(SRCS) $(HEADERS)
 	clang-format --dry-run --Werror $?
-	$(MAKE) -C tests/integration/ linter
 	$(MAKE) -C tests/unit/ linter
 
 .PHONY: linter-fix
diff --git a/tests/integration/Makefile b/tests/integration/Makefile
index db2b7d9..567ce88 100644
--- a/tests/integration/Makefile
+++ b/tests/integration/Makefile
@@ -1,31 +1,14 @@
-SRCS        := $(wildcard *_test.c)
-OBJS        := $(patsubst %_test.c, %_test.o, $(SRCS))
-CFLAGS      := -I../../src -I../shared
-TESTS       := $(patsubst %_test.c, %_test, $(SRCS))
-EXEC_TESTS  := $(patsubst %_test, ./%_test, $(TESTS))
-MUNIT_SRC   := ../shared/munit.c
-MUNIT       := ./munit.o
+TESTER_SRC := ./test.sh
+TESTS := $(wildcard ./tests/*.ol)
 
 .PHONY: all
-all: $(MUNIT) proc_exec.o cli_runner.o $(TESTS)
-	@for file in $(EXEC_TESTS); do \
-                ./"$$file"; \
+all:
+	@set -e; \
+	for file in $(TESTS); do \
+                $(TESTER_SRC) "$$file"; \
+		echo; \
         done
 
 .PHONY: clean
 clean:
-	$(RM) *.o *_test
-
-.PHONY: linter
-linter: $(SRCS)
-	clang-format --dry-run --Werror $?
-
-.PHONY: linter-fix
-linter-fix: $(SRCS)
-	clang-format -i $?
-
-cli_test: $(MUNIT) proc_exec.o cli_runner.o cli_test.o
-	$(CC) $? $(CFLAGS) -o $@
-
-$(MUNIT):
-	$(CC) -c $(MUNIT_SRC) $(CFLAGS) -o $(MUNIT)
+	@rm -rf ./tests/*.ol.test*
diff --git a/tests/integration/test.sh b/tests/integration/test.sh
new file mode 100755
index 0000000..6ab03de
--- /dev/null
+++ b/tests/integration/test.sh
@@ -0,0 +1,182 @@
+COMPILER_PATH="../../olang"
+TEST_FILE="$1"
+
+PROGRAM="$TEST_FILE.test"
+
+COMPILER_OUTPUT_FILE="$TEST_FILE.test.compiler_output"
+COMPILER_EXPECT_OUTPUT_FILE="$TEST_FILE.test.expected_compiler_output"
+
+AST_OUTPUT_FILE="$TEST_FILE.test.ast_output"
+AST_EXPECT_OUTPUT_FILE="$TEST_FILE.test.expected_ast_output"
+
+TOKENS_OUTPUT_FILE="$TEST_FILE.test.tokens_output"
+TOKENS_EXPECT_OUTPUT_FILE="$TEST_FILE.test.expected_tokens_output"
+
+TOKENS_OUTPUT_FILE="$TEST_FILE.test.tokens_output"
+TOKENS_EXPECT_OUTPUT_FILE="$TEST_FILE.test.expected_tokens_output"
+
+PROGRAM_OUTPUT="$TEST_FILE.test.program_output"
+
+extract_comment() {
+  tag="$1"
+  comment="# $tag: " 
+  cat $TEST_FILE | grep "$comment" | sed -e "s/$comment//"
+}
+
+# UI
+COLOR_RED=1
+COLOR_GREEN=2
+COLOR_YELLOW=3
+COLOR_BLUE=4
+COLOR_CYAN=6
+COLOR_GRAY=7
+
+colored() {
+    text="$1"
+    color="$2"
+    if [ -t 1 ]; then
+      if tput setaf 1 &> /dev/null; then
+        printf "$(tput setaf $color)$text$(tput sgr0)"
+        return
+      fi
+    fi
+
+    printf "$text"
+}
+
+print_test_description() {
+  colored "$TEST_FILE: \n" $COLOR_CYAN
+  colored "$(extract_comment spec)\n\n" $COLOR_GRAY
+}
+
+print_passed() {
+  context="$1"
+  printf "$context: "
+  colored "passed" $COLOR_GREEN
+  echo
+}
+
+print_failed() {
+  context="$1"
+  printf "$context: "
+  colored "failed" $COLOR_RED
+  echo
+}
+
+print_skiped() {
+  context="$1"
+  printf "$context: "
+  colored "not set" $COLOR_GRAY
+  echo
+}
+
+# Assertions
+expect_exit_code() {
+  context="$1"
+  expected="$2"
+  actual="$3"
+  
+  if [ -z "$expected" ]; then
+    print_skiped "$context"
+    echo "no expected exit code set, it exited with \"$actual\"."
+  else
+    if [ "$expected" = "$actual" ]; then
+      print_passed "$context"
+    else
+      print_failed "$context"
+      colored "Expected exit code: " $COLOR_YELLOW
+      colored "$expected" $COLOR_GRAY
+      colored ", actual: " $COLOR_YELLOW
+      colored "$actual\n" $COLOR_GRAY
+      exit 1
+    fi
+  fi
+}
+
+expect_output() {
+  context="$1"
+  expected_file="$2"
+  actual_file="$3"
+
+  if [ "$(cat $expected_file | wc -l)" = "0" ]; then
+    print_skiped "$context"
+    return
+  fi
+
+  if cmp -s "$expected_file" "$actual_file"; then
+    print_passed "$context"
+  else
+    if [ "$(cat "$expected_file")" = "%empty%" ]; then
+      if [ "$(cat $actual_file | wc -c)" = "0" ]; then
+        print_passed "$context"
+        return
+      fi
+    fi
+    print_failed "$context"
+    colored "$context not match:\n" $COLOR_YELLOW
+    diff "$actual_file" "$expected_file" -u --color
+    exit 1
+  fi
+}
+
+cleanup() {
+  rm -f $PROGRAM*
+}
+
+test_compiler() {
+  $COMPILER_PATH $TEST_FILE -o $PROGRAM > "$COMPILER_OUTPUT_FILE" 2>&1
+
+  EXIT=$?
+  EXPECTED_EXIT=$(extract_comment compiler_exit)
+
+  expect_exit_code compiler_exit "$EXPECTED_EXIT" "$EXIT"
+
+  extract_comment compiler_output > $COMPILER_EXPECT_OUTPUT_FILE
+  expect_output compiler_output "$COMPILER_EXPECT_OUTPUT_FILE" "$COMPILER_OUTPUT_FILE"
+
+  if [ "$EXIT" = "1" ]; then
+    echo "program execution skiped since the compiler failed as expected."
+    cleanup
+    exit 0
+  fi
+}
+
+test_ast() {
+  $COMPILER_PATH $TEST_FILE --dump-ast > "$AST_OUTPUT_FILE" 2>&1
+
+  extract_comment ast > $AST_EXPECT_OUTPUT_FILE
+  expect_output "ast" "$AST_EXPECT_OUTPUT_FILE" "$AST_OUTPUT_FILE"
+}
+
+test_tokens() {
+  extract_comment tokens > $TOKENS_EXPECT_OUTPUT_FILE
+  TOKEN_LINES=$(cat $TOKENS_EXPECT_OUTPUT_FILE | wc -l)
+
+  $COMPILER_PATH $TEST_FILE --dump-tokens | head -n $TOKEN_LINES > "$TOKENS_OUTPUT_FILE" 2>&1
+
+  expect_output tokens $TOKENS_OUTPUT_FILE $TOKENS_EXPECT_OUTPUT_FILE
+
+  print_passed tokens
+}
+
+test_program() {
+  ./$PROGRAM
+
+  EXIT=$?
+  EXPECTED_EXIT=$(extract_comment program_exit)
+  expect_exit_code "program_exit" "$EXPECTED_EXIT" "$EXIT"
+}
+
+
+main() {
+  print_test_description
+
+  test_compiler
+  test_program
+  test_tokens
+  test_ast
+
+  cleanup
+}
+
+main
diff --git a/tests/integration/tests/0001_main_exit.ol b/tests/integration/tests/0001_main_exit.ol
new file mode 100644
index 0000000..c30492f
--- /dev/null
+++ b/tests/integration/tests/0001_main_exit.ol
@@ -0,0 +1,27 @@
+# spec: A minimal olang program
+fn main(): u32 {
+  return 0
+}
+# compiler_exit: 0
+# compiler_output: %empty%
+# program_exit: 0
+# 
+# ast: Translation_Unit
+# ast: `-Function_Definition <name:main> <return:0>
+# ast:   `-Block
+# ast:     `-Return_Statement
+# ast:       `-Literal <kind:u32> <value:0>
+#
+# tokens: ./tests/0001_main_exit.ol:1:32: <line_feed>
+# tokens: ./tests/0001_main_exit.ol:2:1: <fn>
+# tokens: ./tests/0001_main_exit.ol:2:4: <identifier>
+# tokens: ./tests/0001_main_exit.ol:2:8: <(>
+# tokens: ./tests/0001_main_exit.ol:2:9: <)>
+# tokens: ./tests/0001_main_exit.ol:2:10: <:>
+# tokens: ./tests/0001_main_exit.ol:2:12: <identifier>
+# tokens: ./tests/0001_main_exit.ol:2:16: <{>
+# tokens: ./tests/0001_main_exit.ol:2:17: <line_feed>
+# tokens: ./tests/0001_main_exit.ol:3:3: <return>
+# tokens: ./tests/0001_main_exit.ol:3:10: <number>
+# tokens: ./tests/0001_main_exit.ol:3:11: <line_feed>
+# tokens: ./tests/0001_main_exit.ol:4:1: <}>
-- 
2.34.1

[PATCH olang 2/4] tests: remove previous integration tests structure

[Carlos Maniero]

Signed-off-by: Carlos Maniero <carlos@maniero.me>
---
 tests/integration/cli_runner.c | 100 -------------------------
 tests/integration/cli_runner.h |  32 --------
 tests/integration/cli_test.c   | 132 ---------------------------------
 tests/integration/proc_exec.c  |  63 ----------------
 tests/integration/proc_exec.h  |  37 ---------
 tests/integration/test.sh      |  62 ++++++++--------
 6 files changed, 33 insertions(+), 393 deletions(-)
 delete mode 100644 tests/integration/cli_runner.c
 delete mode 100644 tests/integration/cli_runner.h
 delete mode 100644 tests/integration/cli_test.c
 delete mode 100644 tests/integration/proc_exec.c
 delete mode 100644 tests/integration/proc_exec.h

diff --git a/tests/integration/cli_runner.c b/tests/integration/cli_runner.c
deleted file mode 100644
index 636abfc..0000000
--- a/tests/integration/cli_runner.c
+++ /dev/null
@@ -1,100 +0,0 @@
-/*
- * Copyright (C) 2024 olang maintainers
- *
- * This program is free software: you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation, either version 3 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program.  If not, see <https://www.gnu.org/licenses/>.
- */
-#include "cli_runner.h"
-#include "proc_exec.h"
-#include <assert.h>
-#include <stdbool.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <sys/wait.h>
-#include <unistd.h>
-
-#define OLANG_COMPILER_PATH "../../olang"
-
-static int compiler_exists_already_checked = 0;
-
-static void
-assert_compiler_exists()
-{
-    {
-        if (compiler_exists_already_checked == 1) {
-            return;
-        }
-
-        compiler_exists_already_checked = 1;
-    }
-
-    FILE *file = fopen(OLANG_COMPILER_PATH, "r");
-
-    if (file != NULL) {
-        fclose(file);
-        return;
-    }
-
-    perror("Build the compiler before executing tests");
-    exit(1);
-}
-
-void
-create_tmp_file_name(char *file_name)
-{
-    sprintf(file_name, "%s/olang_programXXXXXX", P_tmpdir);
-    int fd = mkstemp(file_name);
-
-    if (fd == -1) {
-        perror("Could not create a tmp file. Check your P_tmpdir permission.");
-        exit(1);
-    }
-    close(fd);
-}
-
-void
-cli_runner_compiler(cli_result_t* result, char *args[])
-{
-    assert_compiler_exists();
-
-    proc_exec_command_t command = {
-        .path = OLANG_COMPILER_PATH,
-        .args = args
-    };
-
-    proc_exec(&command);
-
-    result->exec = command.result;
-}
-
-cli_result_t
-cli_runner_compiler_dump_tokens(char *src)
-{
-    cli_result_t result = { 0 };
-
-    char *program_args[] = { "olang", "--dump-tokens", src, NULL };
-    cli_runner_compiler(&result, program_args);
-    return result;
-}
-
-cli_result_t
-cli_runner_compiler_compile(char *src)
-{
-    cli_result_t result = { 0 };
-    create_tmp_file_name(result.binary_path);
-
-    char *program_args[] = { "olang", src, "-o", result.binary_path, NULL };
-    cli_runner_compiler(&result, program_args);
-    return result;
-}
diff --git a/tests/integration/cli_runner.h b/tests/integration/cli_runner.h
deleted file mode 100644
index 785cd34..0000000
--- a/tests/integration/cli_runner.h
+++ /dev/null
@@ -1,32 +0,0 @@
-/*
- * Copyright (C) 2024 olang maintainers
- *
- * This program is free software: you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation, either version 3 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program.  If not, see <https://www.gnu.org/licenses/>.
- */
-#ifndef CLI_RUNNER_H
-#define CLI_RUNNER_H
-#include "proc_exec.h"
-
-typedef struct cli_result_t
-{
-    char binary_path[255];
-    proc_exec_result_t exec;
-} cli_result_t;
-
-cli_result_t
-cli_runner_compiler_dump_tokens(char *src);
-
-cli_result_t
-cli_runner_compiler_compile(char *src);
-#endif
diff --git a/tests/integration/cli_test.c b/tests/integration/cli_test.c
deleted file mode 100644
index e7ae059..0000000
--- a/tests/integration/cli_test.c
+++ /dev/null
@@ -1,132 +0,0 @@
-/*
- * Copyright (C) 2024 olang maintainers
- *
- * This program is free software: you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation, either version 3 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program.  If not, see <https://www.gnu.org/licenses/>.
- */
-#define MUNIT_ENABLE_ASSERT_ALIASES
-#include "cli_runner.h"
-#include "munit.h"
-#include <stdio.h>
-
-static MunitResult
-test_cli_dump_tokens_example_main_exit(const MunitParameter params[], void *user_data_or_fixture)
-{
-    cli_result_t compilation_result = cli_runner_compiler_dump_tokens("../../examples/main_exit.ol");
-    munit_assert_int(compilation_result.exec.exit_code, ==, 0);
-    munit_assert_string_equal(compilation_result.exec.stdout_buf,
-                              "../../examples/main_exit.ol:1:12: <line_feed>\n"
-                              "../../examples/main_exit.ol:2:16: <line_feed>\n"
-                              "../../examples/main_exit.ol:3:1: <line_feed>\n"
-                              "../../examples/main_exit.ol:4:1: <fn>\n"
-                              "../../examples/main_exit.ol:4:4: <identifier>\n"
-                              "../../examples/main_exit.ol:4:8: <(>\n"
-                              "../../examples/main_exit.ol:4:9: <)>\n"
-                              "../../examples/main_exit.ol:4:10: <:>\n"
-                              "../../examples/main_exit.ol:4:12: <identifier>\n"
-                              "../../examples/main_exit.ol:4:16: <{>\n"
-                              "../../examples/main_exit.ol:4:17: <line_feed>\n"
-                              "../../examples/main_exit.ol:5:3: <return>\n"
-                              "../../examples/main_exit.ol:5:10: <number>\n"
-                              "../../examples/main_exit.ol:5:11: <line_feed>\n"
-                              "../../examples/main_exit.ol:6:1: <}>\n"
-                              "../../examples/main_exit.ol:6:2: <line_feed>\n"
-                              "../../examples/main_exit.ol:7:1: <EOF>\n");
-    return MUNIT_OK;
-}
-
-static MunitResult
-test_cli_dump_tokens_example_expression(const MunitParameter params[], void *user_data_or_fixture)
-{
-    cli_result_t compilation_result = cli_runner_compiler_dump_tokens("../../examples/expression.ol");
-    munit_assert_int(compilation_result.exec.exit_code, ==, 0);
-    munit_assert_string_equal(compilation_result.exec.stdout_buf,
-                              "../../examples/expression.ol:1:1: <fn>\n"
-                              "../../examples/expression.ol:1:4: <identifier>\n"
-                              "../../examples/expression.ol:1:8: <(>\n"
-                              "../../examples/expression.ol:1:9: <)>\n"
-                              "../../examples/expression.ol:1:10: <:>\n"
-                              "../../examples/expression.ol:1:12: <identifier>\n"
-                              "../../examples/expression.ol:1:16: <{>\n"
-                              "../../examples/expression.ol:1:17: <line_feed>\n"
-                              "../../examples/expression.ol:2:3: <return>\n"
-                              "../../examples/expression.ol:2:10: <(>\n"
-                              "../../examples/expression.ol:2:11: <number>\n"
-                              "../../examples/expression.ol:2:14: <+>\n"
-                              "../../examples/expression.ol:2:16: <number>\n"
-                              "../../examples/expression.ol:2:18: <*>\n"
-                              "../../examples/expression.ol:2:20: <number>\n"
-                              "../../examples/expression.ol:2:21: <)>\n"
-                              "../../examples/expression.ol:2:23: <->\n"
-                              "../../examples/expression.ol:2:25: <(>\n"
-                              "../../examples/expression.ol:2:26: <number>\n"
-                              "../../examples/expression.ol:2:29: <->\n"
-                              "../../examples/expression.ol:2:31: <(>\n"
-                              "../../examples/expression.ol:2:32: <number>\n"
-                              "../../examples/expression.ol:2:34: <+>\n"
-                              "../../examples/expression.ol:2:36: <number>\n"
-                              "../../examples/expression.ol:2:37: <)>\n"
-                              "../../examples/expression.ol:2:39: </>\n"
-                              "../../examples/expression.ol:2:41: <number>\n"
-                              "../../examples/expression.ol:2:42: <)>\n"
-                              "../../examples/expression.ol:2:43: <line_feed>\n"
-                              "../../examples/expression.ol:3:1: <}>\n"
-                              "../../examples/expression.ol:3:2: <line_feed>\n"
-                              "../../examples/expression.ol:4:1: <EOF>\n");
-    return MUNIT_OK;
-}
-
-static MunitResult
-test_cli_compile_minimal_program(const MunitParameter params[], void *user_data_or_fixture)
-{
-    cli_result_t compilation_result = cli_runner_compiler_compile("../../examples/main_exit.ol");
-    munit_assert_int(compilation_result.exec.exit_code, ==, 0);
-
-    char *command_args[] = { compilation_result.binary_path, NULL };
-
-    proc_exec_command_t command = { .path = command_args[0], .args = command_args };
-
-    proc_exec(&command);
-
-    remove(command_args[0]);
-
-    munit_assert_int(command.result.exit_code, ==, 0);
-
-    return MUNIT_OK;
-}
-
-static MunitTest tests[] = {
-    { "/test_cli_dump_tokens_example_main_exit",
-      test_cli_dump_tokens_example_main_exit,
-      NULL,
-      NULL,
-      MUNIT_TEST_OPTION_NONE,
-      NULL },
-    { "/test_cli_dump_tokens_example_expression",
-      test_cli_dump_tokens_example_expression,
-      NULL,
-      NULL,
-      MUNIT_TEST_OPTION_NONE,
-      NULL },
-    { "/test_cli_compile_minimal_program", test_cli_compile_minimal_program, NULL, NULL, MUNIT_TEST_OPTION_NONE, NULL },
-    { NULL, NULL, NULL, NULL, MUNIT_TEST_OPTION_NONE, NULL }
-};
-
-static const MunitSuite suite = { "/cli_test", tests, NULL, 1, MUNIT_SUITE_OPTION_NONE };
-
-int
-main(int argc, char *argv[])
-{
-    return munit_suite_main(&suite, NULL, argc, argv);
-    return EXIT_SUCCESS;
-}
diff --git a/tests/integration/proc_exec.c b/tests/integration/proc_exec.c
deleted file mode 100644
index c22dd1e..0000000
--- a/tests/integration/proc_exec.c
+++ /dev/null
@@ -1,63 +0,0 @@
-/*
- * Copyright (C) 2024 olang maintainers
- *
- * This program is free software: you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation, either version 3 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program.  If not, see <https://www.gnu.org/licenses/>.
- */
-#include "proc_exec.h"
-#include <assert.h>
-#include <stdbool.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <sys/wait.h>
-#include <unistd.h>
-
-
-void
-proc_exec(proc_exec_command_t* command)
-{
-    int fd_pipe[2];
-
-    if (pipe(fd_pipe) == -1) {
-        perror("pipe error.");
-        exit(1);
-    }
-
-    pid_t pid = fork();
-
-    if (pid == -1) {
-        perror("fork error.");
-        exit(1);
-    }
-
-    if (pid == 0) {
-        dup2(fd_pipe[1], STDOUT_FILENO);
-        close(fd_pipe[0]);
-        close(fd_pipe[1]);
-
-        execv(command->path, command->args);
-        perror("execl error.");
-        exit(127);
-    } else {
-        close(fd_pipe[1]);
-        // TODO: stop truncating the output.
-        if (read(fd_pipe[0], command->result.stdout_buf, sizeof(command->result.stdout_buf)) == -1) {
-            perror("read error.");
-            exit(1);
-        }
-        int status;
-        waitpid(pid, &status, 0);
-        command->result.exit_code = WEXITSTATUS(status);
-    }
-}
diff --git a/tests/integration/proc_exec.h b/tests/integration/proc_exec.h
deleted file mode 100644
index 45c2977..0000000
--- a/tests/integration/proc_exec.h
+++ /dev/null
@@ -1,37 +0,0 @@
-/*
- * Copyright (C) 2024 olang maintainers
- *
- * This program is free software: you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation, either version 3 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program.  If not, see <https://www.gnu.org/licenses/>.
- */
-#ifndef PROC_EXEC_H
-#define PROC_EXEC_H
-#include <stdlib.h>
-
-typedef struct proc_exec_result
-{
-    int exit_code;
-    // FIXME: output buffer shouldn't be fixed size
-    char stdout_buf[2048];
-} proc_exec_result_t;
-
-typedef struct proc_exec_command
-{
-    char* path;
-    char** args;
-    proc_exec_result_t result;
-} proc_exec_command_t;
-
-void
-proc_exec(proc_exec_command_t* command);
-#endif
diff --git a/tests/integration/test.sh b/tests/integration/test.sh
index 6ab03de..9fdabef 100755
--- a/tests/integration/test.sh
+++ b/tests/integration/test.sh
@@ -1,3 +1,4 @@
+#!/bin/sh
 COMPILER_PATH="../../olang"
 TEST_FILE="$1"
 
@@ -15,57 +16,63 @@ TOKENS_EXPECT_OUTPUT_FILE="$TEST_FILE.test.expected_tokens_output"
 TOKENS_OUTPUT_FILE="$TEST_FILE.test.tokens_output"
 TOKENS_EXPECT_OUTPUT_FILE="$TEST_FILE.test.expected_tokens_output"
 
-PROGRAM_OUTPUT="$TEST_FILE.test.program_output"
-
+# Misc
 extract_comment() {
   tag="$1"
-  comment="# $tag: " 
-  cat $TEST_FILE | grep "$comment" | sed -e "s/$comment//"
+  comment="# $tag:"
+  grep "$comment" "$TEST_FILE" | sed -e "s/$comment //"
+}
+
+cleanup() {
+  rm -f "$PROGRAM*"
 }
 
 # UI
 COLOR_RED=1
 COLOR_GREEN=2
 COLOR_YELLOW=3
-COLOR_BLUE=4
 COLOR_CYAN=6
 COLOR_GRAY=7
 
 colored() {
     text="$1"
-    color="$2"
+    color=$(tput setaf "$2")
+    reset=$(tput sgr0)
+
     if [ -t 1 ]; then
-      if tput setaf 1 &> /dev/null; then
-        printf "$(tput setaf $color)$text$(tput sgr0)"
+      if tput setaf 1 > /dev/null 2>&1; then
+        printf "%s%s%s" "$color" "$text" "$reset"
         return
       fi
     fi
 
-    printf "$text"
+    printf "%s" "$text"
 }
 
 print_test_description() {
-  colored "$TEST_FILE: \n" $COLOR_CYAN
-  colored "$(extract_comment spec)\n\n" $COLOR_GRAY
+  colored "$TEST_FILE: " $COLOR_CYAN
+  printf "\n"
+  colored "$(extract_comment spec)" $COLOR_GRAY
+  printf "\n\n"
 }
 
 print_passed() {
   context="$1"
-  printf "$context: "
+  printf "%s: " "$context"
   colored "passed" $COLOR_GREEN
   echo
 }
 
 print_failed() {
   context="$1"
-  printf "$context: "
+  printf "%s: " "$context"
   colored "failed" $COLOR_RED
   echo
 }
 
 print_skiped() {
   context="$1"
-  printf "$context: "
+  printf "%s: " "$context"
   colored "not set" $COLOR_GRAY
   echo
 }
@@ -98,7 +105,7 @@ expect_output() {
   expected_file="$2"
   actual_file="$3"
 
-  if [ "$(cat $expected_file | wc -l)" = "0" ]; then
+  if [ "$(wc -l < "$expected_file")" = "0" ]; then
     print_skiped "$context"
     return
   fi
@@ -107,7 +114,7 @@ expect_output() {
     print_passed "$context"
   else
     if [ "$(cat "$expected_file")" = "%empty%" ]; then
-      if [ "$(cat $actual_file | wc -c)" = "0" ]; then
+      if [ "$(wc -c < "$actual_file")" = "0" ]; then
         print_passed "$context"
         return
       fi
@@ -119,19 +126,16 @@ expect_output() {
   fi
 }
 
-cleanup() {
-  rm -f $PROGRAM*
-}
-
+# Tests
 test_compiler() {
-  $COMPILER_PATH $TEST_FILE -o $PROGRAM > "$COMPILER_OUTPUT_FILE" 2>&1
+  $COMPILER_PATH "$TEST_FILE" -o "$PROGRAM" > "$COMPILER_OUTPUT_FILE" 2>&1
 
   EXIT=$?
   EXPECTED_EXIT=$(extract_comment compiler_exit)
 
   expect_exit_code compiler_exit "$EXPECTED_EXIT" "$EXIT"
 
-  extract_comment compiler_output > $COMPILER_EXPECT_OUTPUT_FILE
+  extract_comment compiler_output > "$COMPILER_EXPECT_OUTPUT_FILE"
   expect_output compiler_output "$COMPILER_EXPECT_OUTPUT_FILE" "$COMPILER_OUTPUT_FILE"
 
   if [ "$EXIT" = "1" ]; then
@@ -142,25 +146,25 @@ test_compiler() {
 }
 
 test_ast() {
-  $COMPILER_PATH $TEST_FILE --dump-ast > "$AST_OUTPUT_FILE" 2>&1
+  $COMPILER_PATH "$TEST_FILE" --dump-ast > "$AST_OUTPUT_FILE" 2>&1
 
-  extract_comment ast > $AST_EXPECT_OUTPUT_FILE
+  extract_comment ast > "$AST_EXPECT_OUTPUT_FILE"
   expect_output "ast" "$AST_EXPECT_OUTPUT_FILE" "$AST_OUTPUT_FILE"
 }
 
 test_tokens() {
-  extract_comment tokens > $TOKENS_EXPECT_OUTPUT_FILE
-  TOKEN_LINES=$(cat $TOKENS_EXPECT_OUTPUT_FILE | wc -l)
+  extract_comment tokens > "$TOKENS_EXPECT_OUTPUT_FILE"
+  TOKEN_LINES=$(wc -l < "$TOKENS_EXPECT_OUTPUT_FILE")
 
-  $COMPILER_PATH $TEST_FILE --dump-tokens | head -n $TOKEN_LINES > "$TOKENS_OUTPUT_FILE" 2>&1
+  $COMPILER_PATH "$TEST_FILE" --dump-tokens | head -n "$TOKEN_LINES" > "$TOKENS_OUTPUT_FILE" 2>&1
 
-  expect_output tokens $TOKENS_OUTPUT_FILE $TOKENS_EXPECT_OUTPUT_FILE
+  expect_output tokens "$TOKENS_OUTPUT_FILE" "$TOKENS_EXPECT_OUTPUT_FILE"
 
   print_passed tokens
 }
 
 test_program() {
-  ./$PROGRAM
+  "./$PROGRAM"
 
   EXIT=$?
   EXPECTED_EXIT=$(extract_comment program_exit)
-- 
2.34.1

[PATCH olang 3/4] tests: include integration tests for function parser errors

[Carlos Maniero]

This new tests covers the parser errors we support.

Signed-off-by: Carlos Maniero <carlos@maniero.me>
---
 tests/integration/tests/0002_fn_without_fn_keyword.ol     | 8 ++++++++
 tests/integration/tests/0003_fn_without_type.ol           | 8 ++++++++
 tests/integration/tests/0004_fn_without_type_colon.ol     | 8 ++++++++
 .../tests/0005_fn_without_open_curly_brackets.ol          | 8 ++++++++
 .../tests/0006_fn_without_close_curly_brackets.ol         | 7 +++++++
 tests/integration/tests/0007_fn_without_return.ol         | 8 ++++++++
 6 files changed, 47 insertions(+)
 create mode 100644 tests/integration/tests/0002_fn_without_fn_keyword.ol
 create mode 100644 tests/integration/tests/0003_fn_without_type.ol
 create mode 100644 tests/integration/tests/0004_fn_without_type_colon.ol
 create mode 100644 tests/integration/tests/0005_fn_without_open_curly_brackets.ol
 create mode 100644 tests/integration/tests/0006_fn_without_close_curly_brackets.ol
 create mode 100644 tests/integration/tests/0007_fn_without_return.ol

diff --git a/tests/integration/tests/0002_fn_without_fn_keyword.ol b/tests/integration/tests/0002_fn_without_fn_keyword.ol
new file mode 100644
index 0000000..f1ad57e
--- /dev/null
+++ b/tests/integration/tests/0002_fn_without_fn_keyword.ol
@@ -0,0 +1,8 @@
+# spec: An invalid program with no fn keyword
+main(): u32 {
+  return 0
+}
+# compiler_exit: 1
+# compiler_output: ./tests/0002_fn_without_fn_keyword.ol:2:1: error: got 'main' token but expect <fn>
+# compiler_output: main(): u32 {
+# compiler_output: ^
diff --git a/tests/integration/tests/0003_fn_without_type.ol b/tests/integration/tests/0003_fn_without_type.ol
new file mode 100644
index 0000000..32716e1
--- /dev/null
+++ b/tests/integration/tests/0003_fn_without_type.ol
@@ -0,0 +1,8 @@
+# spec: An invalid program with no type
+fn main(): {
+  return 0
+}
+# compiler_exit: 1
+# compiler_output: ./tests/0003_fn_without_type.ol:2:12: error: got '{' token but expect <identifier>
+# compiler_output: fn main(): {
+# compiler_output:            ^
diff --git a/tests/integration/tests/0004_fn_without_type_colon.ol b/tests/integration/tests/0004_fn_without_type_colon.ol
new file mode 100644
index 0000000..bd04c37
--- /dev/null
+++ b/tests/integration/tests/0004_fn_without_type_colon.ol
@@ -0,0 +1,8 @@
+# spec: An invalid program with no type colon
+fn main() u32 {
+  return 0
+}
+# compiler_exit: 1
+# compiler_output: ./tests/0004_fn_without_type_colon.ol:2:11: error: got 'u32' token but expect <:>
+# compiler_output: fn main() u32 {
+# compiler_output:           ^
diff --git a/tests/integration/tests/0005_fn_without_open_curly_brackets.ol b/tests/integration/tests/0005_fn_without_open_curly_brackets.ol
new file mode 100644
index 0000000..8e13946
--- /dev/null
+++ b/tests/integration/tests/0005_fn_without_open_curly_brackets.ol
@@ -0,0 +1,8 @@
+# spec: An invalid program with no open curly brackets
+fn main(): u32
+  return 0
+}
+# compiler_exit: 1
+# compiler_output: ./tests/0005_fn_without_open_curly_brackets.ol:3:3: error: got 'return' token but expect <{>
+# compiler_output:   return 0
+# compiler_output:   ^
diff --git a/tests/integration/tests/0006_fn_without_close_curly_brackets.ol b/tests/integration/tests/0006_fn_without_close_curly_brackets.ol
new file mode 100644
index 0000000..d43c9f9
--- /dev/null
+++ b/tests/integration/tests/0006_fn_without_close_curly_brackets.ol
@@ -0,0 +1,7 @@
+# spec: An invalid program with no close curly brackets
+fn main(): u32 {
+  return 0
+# compiler_exit: 1
+# compiler_output: ./tests/0006_fn_without_close_curly_brackets.ol:8:1: error: got '' token but expect <}>
+# compiler_output: 
+# compiler_output: ^
diff --git a/tests/integration/tests/0007_fn_without_return.ol b/tests/integration/tests/0007_fn_without_return.ol
new file mode 100644
index 0000000..fe33d69
--- /dev/null
+++ b/tests/integration/tests/0007_fn_without_return.ol
@@ -0,0 +1,8 @@
+# spec: An invalid program without return statment
+# TODO: Improve the compiler error message
+fn main(): u32 {
+}
+# compiler_exit: 1
+# compiler_output: ./tests/0007_fn_without_return.ol:4:1: error: got '}' token but expect <return>
+# compiler_output: }
+# compiler_output: ^
-- 
2.34.1

[PATCH olang 4/4] tests: print integration tests TODOs

[Carlos Maniero]

Sometimes people use to ask how to contribute with olang. This is a
great way to help them, we can just create TODOS on the integration
tests as we already have some parts of the language specified.

Signed-off-by: Carlos Maniero <carlos@maniero.me>
---
 tests/integration/test.sh | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tests/integration/test.sh b/tests/integration/test.sh
index 9fdabef..c8535a7 100755
--- a/tests/integration/test.sh
+++ b/tests/integration/test.sh
@@ -56,6 +56,14 @@ print_test_description() {
   printf "\n\n"
 }
 
+print_todo() {
+  TODOS=$(extract_comment TODO)
+
+  if [ -n "$TODOS" ]; then
+    colored "TODO: $(extract_comment TODO)\n\n" $COLOR_YELLOW
+  fi
+}
+
 print_passed() {
   context="$1"
   printf "%s: " "$context"
@@ -174,6 +182,7 @@ test_program() {
 
 main() {
   print_test_description
+  print_todo
 
   test_compiler
   test_program
-- 
2.34.1

[olang/patches/.build.yml] build success

[builds.sr.ht]

olang/patches/.build.yml: SUCCESS in 51s

[comment based integration tests][0] from [Carlos Maniero][1]

[0]: https://lists.sr.ht/~johnnyrichard/olang-devel/patches/51804
[1]: mailto:carlos@maniero.me

✓ #1219016 SUCCESS olang/patches/.build.yml https://builds.sr.ht/~johnnyrichard/job/1219016

Re: [PATCH olang 0/4] comment based integration tests

[Johnny Richard]

Thanks a lot for this contribution, I think it will improve the
development experience and definitely will help us to move forward
faster.

I have only a small adjustment to suggest, actually a nitpick.

On Sun, May 12, 2024 at 11:30:29AM GMT, Carlos Maniero wrote:
> The old munit based integration tests was replaced by a new mechanism
> that makes testing more tasteful and easy to perform.
> 
> To create an integration tests you just need to create a olang file at
> *tests/integration/tests* directory. The assertions are performed
> thought a couple of comments.

The folder *tests/integration/tests* feels a little bit redundant. What
do you think about changing the folder structure to a smaller one like
*test/integration*?

Re: [PATCH olang 0/4] comment based integration tests

[Johnny Richard]

This patch has been rejected.  Actually there is another patch which
has superseded this one.