From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id MBWNAJ3SQGbG3wAAe85BDQ:P1 (envelope-from ) for ; Sun, 12 May 2024 16:30:53 +0200 Received: from aspmx1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id MBWNAJ3SQGbG3wAAe85BDQ (envelope-from ) for ; Sun, 12 May 2024 16:30:53 +0200 X-Envelope-To: patches@johnnyrichard.com Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=lists.sr.ht header.s=20240113 header.b=Ib2uVwwo; dkim=pass header.d=maniero.me header.s=hostingermail1 header.b=VetUneoY; arc=pass ("mailchannels.net:s=arc-2022:i=1"); dmarc=pass (policy=none) header.from=maniero.me; spf=pass (aspmx1.migadu.com: domain of lists@sr.ht designates 46.23.81.152 as permitted sender) smtp.mailfrom=lists@sr.ht ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=johnnyrichard.com; s=key1; t=1715524252; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=1l9PcOfx1EVSZPw9F5ZUHcuEmymuuLNilpBsp0Yo9Iw=; b=Fna8JcYGGgYOsGhVABTBH79c1LZ1xJvnUEEU4wteKsUDUKsisXF0BPA0Q5FL9yqLQ6O/ip Lv6dDdg1kBN4muUj7EYok0f+ImFe1P/9TG9kkzFky89sgTy5P8tkkpePnbhvOR6d43LLHd GyfknP8ozgeiwx+cptplyGndmcpmP86v12436NWowmjcb1b8+iIk+sRYpZoWUJts0k4fcs QUx6HMstQvsIeyIxdZhg8/NX7si07j9EJyPEt1TllGsCoiPmDDw9RWFYdQrXNxvnob/ARk vjs3YTYqxmLrWw3P2pJBvrTzVnDV5E93VjpuC7h604fz0+8trQ53Sq9HqVRyAw== ARC-Seal: i=2; s=key1; d=johnnyrichard.com; t=1715524252; a=rsa-sha256; cv=pass; b=XIoc0I9RQuxzLZtyIUKwO+cPzI+fCPfuwDT2gjz3Ga0GJoxeHQZghw95VXln89+e63tL8q iT3YiiybYSr489wKAka/UUFdaWSc9dY+YmpWfMiHPu/kDjk/Rm+JXFYI0Vi4IqfvnUvcf5 RZxwnSIo0FXELYequ/p+6Fv6qLIkCWqqT5wnaQ42LETaDCkjpvsQnP70ZsNQzTHZwrvVYJ +iA9kNGiM5q5XTmwQ5w/+ku1ctsxauGVsh2zW7yYo+fbqKFFA5lQ51ZDCc9nKnORnjn2+P qTg2a3utZjwBSM2OKI97wufHJYER0SnCeEigPFoaks6VQ6NUjhB5uizEpUbKTg== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=pass header.d=lists.sr.ht header.s=20240113 header.b=Ib2uVwwo; dkim=pass header.d=maniero.me header.s=hostingermail1 header.b=VetUneoY; arc=pass ("mailchannels.net:s=arc-2022:i=1"); dmarc=pass (policy=none) header.from=maniero.me; spf=pass (aspmx1.migadu.com: domain of lists@sr.ht designates 46.23.81.152 as permitted sender) smtp.mailfrom=lists@sr.ht Received: from mail-a.sr.ht (mail-a.sr.ht [46.23.81.152]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E66A23DF86 for ; Sun, 12 May 2024 16:30:49 +0200 (CEST) DKIM-Signature: a=rsa-sha256; bh=KXuk/Rdu1diBEMoBe7v5iWYwwFrLEMw2QZVUVMJwfa4=; c=simple/simple; d=lists.sr.ht; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-Unsubscribe:List-Subscribe:List-Archive:List-Post:List-ID; q=dns/txt; s=20240113; t=1715524249; v=1; b=Ib2uVwwoQE+rG8jz9Y31EDb5h7Byd0G8bN5LccoCuEkKqo+XZpgSJ/uyNlnqM+APs2/s0eUi yuXREpnvXcVtWDGlAtBqCAxUTP7IpP6MrEaAOzfcDJ7k+Vdj7GqVtuPBbhq8PR2QAKNkFm81k7q r8Mtn46IH4XdYos6li4CJXmho6q6mvFJdncxn9PuXb2xGF1Dsao3dX9P28IQJ7vU4JM6gQigRnh 59j5KA4bFjvwi4QeDw3CzTuiO54Fa5qvbUdk0Dm769kbkFBUTLo6TEN5p2HmiWPAzkMsLVtA62R pliuq2Rpc60Y43JZoyV9c1F7AiH7n2V9rpgJwTnJ+AAww== Received: from lists.sr.ht (unknown [IPv6:2a03:6000:1813:1337::154]) by mail-a.sr.ht (Postfix) with ESMTPSA id AC60320130 for ; Sun, 12 May 2024 14:30:49 +0000 (UTC) Received: from rusty.tulip.relay.mailchannels.net (rusty.tulip.relay.mailchannels.net [23.83.218.252]) by mail-a.sr.ht (Postfix) with ESMTPS id B958120119 for <~johnnyrichard/olang-devel@lists.sr.ht>; Sun, 12 May 2024 14:30:48 +0000 (UTC) X-Sender-Id: hostingeremail|x-authuser|carlos@maniero.me Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id 4935E541ADB for <~johnnyrichard/olang-devel@lists.sr.ht>; Sun, 12 May 2024 14:30:46 +0000 (UTC) Received: from uk-fast-smtpout5.hostinger.io (unknown [127.0.0.6]) (Authenticated sender: hostingeremail) by relay.mailchannels.net (Postfix) with ESMTPA id 573BC541843 for <~johnnyrichard/olang-devel@lists.sr.ht>; Sun, 12 May 2024 14:30:45 +0000 (UTC) ARC-Seal: i=1; s=arc-2022; d=mailchannels.net; t=1715524245; a=rsa-sha256; cv=none; b=bkf/+QE6coOCT0XP13bA7bZqIzX+qJ2WbiWv5XMO3Ttb6fwIfr/vhee12ZDQFbipB3R3il w53XmWHuPurcTzzU8WJfz/KRVtduxctSuqxat2fOlOdIiVyJSU9gAtrgJ7PCl2h3h5ju6L eWVouMSemxDJoyIYF9Qp8vnoyd9TSkDtmw+xnPQlOnIvHnocnKnIeMULuY6QKkL0fX9U2H iiWmQoBZsjAmhHLFn7ATxaTQVddPrpe+MRu5x0ppUW+8kp2eTAJGenrIk+IJkyRhJ0C0l7 UmKwUIc/gZ5qr/9O4SJKWiDG2OsKJclKaP8Ero2QlI/1Vqugky89mmFU2en0SA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=mailchannels.net; s=arc-2022; t=1715524245; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1l9PcOfx1EVSZPw9F5ZUHcuEmymuuLNilpBsp0Yo9Iw=; b=H479BKkB12qkxmpaLVhnUokfYcwrmmZkaMsafSYOtHBVIXxgOmf9CaG5YJwt9AZQYuISQ7 58E0kudTjuqViIPHrITp5kGuKYlVdSAgKdiT/ax18vJqNQQE/TNYOrmI46/ziBjDYeLcvi xKq6qdGJruptXp+xIzbj3YnrPFcBtH5gKjFibR/CpcJ/uaNIMFYVLDiA0HoQN+7Kp2XiCs IBnrzfijVGBzANYAg/6V1NLZdon3iPqguSd1a2GKrgL/I/WIL7c1290p1njG6PghS/zHGJ v7WiV3PPFL1S0x1mNnKcM2v4crvNeUS+Z3G6B/ptl1rHVrOa6Zx8Y/eBmkkYMA== ARC-Authentication-Results: i=1; rspamd-68bbddc7f5-j7p6w; auth=pass smtp.auth=hostingeremail smtp.mailfrom=carlos@maniero.me X-Sender-Id: hostingeremail|x-authuser|carlos@maniero.me X-MC-Relay: Neutral X-MailChannels-SenderId: hostingeremail|x-authuser|carlos@maniero.me X-MailChannels-Auth-Id: hostingeremail X-Ski-Trail: 55f7551c0386440e_1715524246073_2348085844 X-MC-Loop-Signature: 1715524246073:3281312885 X-MC-Ingress-Time: 1715524246073 Received: from uk-fast-smtpout5.hostinger.io ([UNAVAILABLE]. [31.220.23.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384) by 100.116.218.200 (trex/6.9.2); Sun, 12 May 2024 14:30:46 +0000 From: Carlos Maniero DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=maniero.me; s=hostingermail1; t=1715524243; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1l9PcOfx1EVSZPw9F5ZUHcuEmymuuLNilpBsp0Yo9Iw=; b=VetUneoY/k4nVWCIGAc0lQWnBzxRm/IX7iEKxcdhyAhLe5NYrE8mvnAddrjIaemA6KLR6d LErUBtFmxJrsRYgnTYlbddac0SDUw+CL0pu+GA1KCOEJk1yUSEEDi2YuaU8cCm4TLO8oVa rzjeyB9etbdlPglDWTCMXu8ta9SfP2zFJVsAhbk9v1+BHvEWzQokNLBEQjnennCnqdMPCH AiDKggfLurb7hus3X8e9bGVniRTwdF5kVn/s+WsUnSsptnA38hQlH45IRlsRSOTIcu83LP MPp+pb3VwIeI+vb+wjxcjMuQxqTWwGZ7ieGaHJZldPuiugUi101WKkEdF5opVQ== To: ~johnnyrichard/olang-devel@lists.sr.ht Cc: Carlos Maniero Subject: [PATCH olang 1/4] tests: add comment based integration tests mechanism Date: Sun, 12 May 2024 11:30:30 -0300 Message-Id: <20240512143033.229961-2-carlos@maniero.me> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240512143033.229961-1-carlos@maniero.me> References: <20240512143033.229961-1-carlos@maniero.me> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-CM-Envelope: MS4xfHnlQJNHX2O2WziVDieUv/LZVahGHKOkp1DXUK09eg6TyTV/QVGy81DDyhIZu4e/Z6+EyteDyHDlIAkdVWvVRaMbSSvpmmBfqGIQ11zK9JRdNLOhtogk ay5v8JZ+p/v0VHB3fNeiOo+KE2MtGmGlv/z0KUQCXyPUJIJCQbe8IJNbXxjYn94mFuGf7aRTcCGL9ykQesks+X9LPvsP1BjPiFktZtawaVCFbshMjJZ2mcrR X-CM-Analysis: v=2.4 cv=ctHJkE4i c=1 sm=1 tr=0 ts=6640d293 a=WwxFCuf3mf1fs3oSi6/dng==:117 a=WwxFCuf3mf1fs3oSi6/dng==:17 a=MKtGQD3n3ToA:10 a=1oJP67jkp3AA:10 a=1sCFLnRHbVogvLVTsK8A:9 a=o_yoq9tSdnaDG1nzyfYJ:22 X-AuthUser: carlos@maniero.me X-Sourcehut-Patchset-Status: UNKNOWN List-Unsubscribe: List-Subscribe: List-Archive: Archived-At: List-Post: List-ID: ~johnnyrichard/olang-devel <~johnnyrichard/olang-devel.lists.sr.ht> Sender: ~johnnyrichard/olang-devel <~johnnyrichard/olang-devel@lists.sr.ht> X-Migadu-Flow: FLOW_IN X-Migadu-Country: NL X-Migadu-Spam-Score: -6.58 X-Spam-Score: -6.58 X-Migadu-Queue-Id: E66A23DF86 X-Migadu-Scanner: mx11.migadu.com X-TUID: soGVG7bYiAk2 The old munit based integration tests was replaced by a new mechanism that makes testing more tasteful and easy to perform. To create an integration tests you just need to create a olang file at *tests/integration/tests* directory. The assertions are performed thought a couple of comments. spec: some text to describe what are you testing compiler_exit: the compilers exit code compiler_output: the compilers output, %empty% for empty output program_exit: the program exit code ast: the ast output tokens: the tokens Since the language has the line_feed token it is impossible to test the EOF once when added a comment the EOF will be located at line + 1. So that the tokens test the first n-lines defined. The AST compares the entire AST tough. Signed-off-by: Carlos Maniero --- Makefile | 1 - tests/integration/Makefile | 33 +--- tests/integration/test.sh | 182 ++++++++++++++++++++++ tests/integration/tests/0001_main_exit.ol | 27 ++++ 4 files changed, 217 insertions(+), 26 deletions(-) create mode 100755 tests/integration/test.sh create mode 100644 tests/integration/tests/0001_main_exit.ol diff --git a/Makefile b/Makefile index cdfc8e1..a0130f5 100644 --- a/Makefile +++ b/Makefile @@ -19,7 +19,6 @@ $(BUILD_DIR): .PHONY: linter linter: $(SRCS) $(HEADERS) clang-format --dry-run --Werror $? - $(MAKE) -C tests/integration/ linter $(MAKE) -C tests/unit/ linter .PHONY: linter-fix diff --git a/tests/integration/Makefile b/tests/integration/Makefile index db2b7d9..567ce88 100644 --- a/tests/integration/Makefile +++ b/tests/integration/Makefile @@ -1,31 +1,14 @@ -SRCS := $(wildcard *_test.c) -OBJS := $(patsubst %_test.c, %_test.o, $(SRCS)) -CFLAGS := -I../../src -I../shared -TESTS := $(patsubst %_test.c, %_test, $(SRCS)) -EXEC_TESTS := $(patsubst %_test, ./%_test, $(TESTS)) -MUNIT_SRC := ../shared/munit.c -MUNIT := ./munit.o +TESTER_SRC := ./test.sh +TESTS := $(wildcard ./tests/*.ol) .PHONY: all -all: $(MUNIT) proc_exec.o cli_runner.o $(TESTS) - @for file in $(EXEC_TESTS); do \ - ./"$$file"; \ +all: + @set -e; \ + for file in $(TESTS); do \ + $(TESTER_SRC) "$$file"; \ + echo; \ done .PHONY: clean clean: - $(RM) *.o *_test - -.PHONY: linter -linter: $(SRCS) - clang-format --dry-run --Werror $? - -.PHONY: linter-fix -linter-fix: $(SRCS) - clang-format -i $? - -cli_test: $(MUNIT) proc_exec.o cli_runner.o cli_test.o - $(CC) $? $(CFLAGS) -o $@ - -$(MUNIT): - $(CC) -c $(MUNIT_SRC) $(CFLAGS) -o $(MUNIT) + @rm -rf ./tests/*.ol.test* diff --git a/tests/integration/test.sh b/tests/integration/test.sh new file mode 100755 index 0000000..6ab03de --- /dev/null +++ b/tests/integration/test.sh @@ -0,0 +1,182 @@ +COMPILER_PATH="../../olang" +TEST_FILE="$1" + +PROGRAM="$TEST_FILE.test" + +COMPILER_OUTPUT_FILE="$TEST_FILE.test.compiler_output" +COMPILER_EXPECT_OUTPUT_FILE="$TEST_FILE.test.expected_compiler_output" + +AST_OUTPUT_FILE="$TEST_FILE.test.ast_output" +AST_EXPECT_OUTPUT_FILE="$TEST_FILE.test.expected_ast_output" + +TOKENS_OUTPUT_FILE="$TEST_FILE.test.tokens_output" +TOKENS_EXPECT_OUTPUT_FILE="$TEST_FILE.test.expected_tokens_output" + +TOKENS_OUTPUT_FILE="$TEST_FILE.test.tokens_output" +TOKENS_EXPECT_OUTPUT_FILE="$TEST_FILE.test.expected_tokens_output" + +PROGRAM_OUTPUT="$TEST_FILE.test.program_output" + +extract_comment() { + tag="$1" + comment="# $tag: " + cat $TEST_FILE | grep "$comment" | sed -e "s/$comment//" +} + +# UI +COLOR_RED=1 +COLOR_GREEN=2 +COLOR_YELLOW=3 +COLOR_BLUE=4 +COLOR_CYAN=6 +COLOR_GRAY=7 + +colored() { + text="$1" + color="$2" + if [ -t 1 ]; then + if tput setaf 1 &> /dev/null; then + printf "$(tput setaf $color)$text$(tput sgr0)" + return + fi + fi + + printf "$text" +} + +print_test_description() { + colored "$TEST_FILE: \n" $COLOR_CYAN + colored "$(extract_comment spec)\n\n" $COLOR_GRAY +} + +print_passed() { + context="$1" + printf "$context: " + colored "passed" $COLOR_GREEN + echo +} + +print_failed() { + context="$1" + printf "$context: " + colored "failed" $COLOR_RED + echo +} + +print_skiped() { + context="$1" + printf "$context: " + colored "not set" $COLOR_GRAY + echo +} + +# Assertions +expect_exit_code() { + context="$1" + expected="$2" + actual="$3" + + if [ -z "$expected" ]; then + print_skiped "$context" + echo "no expected exit code set, it exited with \"$actual\"." + else + if [ "$expected" = "$actual" ]; then + print_passed "$context" + else + print_failed "$context" + colored "Expected exit code: " $COLOR_YELLOW + colored "$expected" $COLOR_GRAY + colored ", actual: " $COLOR_YELLOW + colored "$actual\n" $COLOR_GRAY + exit 1 + fi + fi +} + +expect_output() { + context="$1" + expected_file="$2" + actual_file="$3" + + if [ "$(cat $expected_file | wc -l)" = "0" ]; then + print_skiped "$context" + return + fi + + if cmp -s "$expected_file" "$actual_file"; then + print_passed "$context" + else + if [ "$(cat "$expected_file")" = "%empty%" ]; then + if [ "$(cat $actual_file | wc -c)" = "0" ]; then + print_passed "$context" + return + fi + fi + print_failed "$context" + colored "$context not match:\n" $COLOR_YELLOW + diff "$actual_file" "$expected_file" -u --color + exit 1 + fi +} + +cleanup() { + rm -f $PROGRAM* +} + +test_compiler() { + $COMPILER_PATH $TEST_FILE -o $PROGRAM > "$COMPILER_OUTPUT_FILE" 2>&1 + + EXIT=$? + EXPECTED_EXIT=$(extract_comment compiler_exit) + + expect_exit_code compiler_exit "$EXPECTED_EXIT" "$EXIT" + + extract_comment compiler_output > $COMPILER_EXPECT_OUTPUT_FILE + expect_output compiler_output "$COMPILER_EXPECT_OUTPUT_FILE" "$COMPILER_OUTPUT_FILE" + + if [ "$EXIT" = "1" ]; then + echo "program execution skiped since the compiler failed as expected." + cleanup + exit 0 + fi +} + +test_ast() { + $COMPILER_PATH $TEST_FILE --dump-ast > "$AST_OUTPUT_FILE" 2>&1 + + extract_comment ast > $AST_EXPECT_OUTPUT_FILE + expect_output "ast" "$AST_EXPECT_OUTPUT_FILE" "$AST_OUTPUT_FILE" +} + +test_tokens() { + extract_comment tokens > $TOKENS_EXPECT_OUTPUT_FILE + TOKEN_LINES=$(cat $TOKENS_EXPECT_OUTPUT_FILE | wc -l) + + $COMPILER_PATH $TEST_FILE --dump-tokens | head -n $TOKEN_LINES > "$TOKENS_OUTPUT_FILE" 2>&1 + + expect_output tokens $TOKENS_OUTPUT_FILE $TOKENS_EXPECT_OUTPUT_FILE + + print_passed tokens +} + +test_program() { + ./$PROGRAM + + EXIT=$? + EXPECTED_EXIT=$(extract_comment program_exit) + expect_exit_code "program_exit" "$EXPECTED_EXIT" "$EXIT" +} + + +main() { + print_test_description + + test_compiler + test_program + test_tokens + test_ast + + cleanup +} + +main diff --git a/tests/integration/tests/0001_main_exit.ol b/tests/integration/tests/0001_main_exit.ol new file mode 100644 index 0000000..c30492f --- /dev/null +++ b/tests/integration/tests/0001_main_exit.ol @@ -0,0 +1,27 @@ +# spec: A minimal olang program +fn main(): u32 { + return 0 +} +# compiler_exit: 0 +# compiler_output: %empty% +# program_exit: 0 +# +# ast: Translation_Unit +# ast: `-Function_Definition +# ast: `-Block +# ast: `-Return_Statement +# ast: `-Literal +# +# tokens: ./tests/0001_main_exit.ol:1:32: +# tokens: ./tests/0001_main_exit.ol:2:1: +# tokens: ./tests/0001_main_exit.ol:2:4: +# tokens: ./tests/0001_main_exit.ol:2:8: <(> +# tokens: ./tests/0001_main_exit.ol:2:9: <)> +# tokens: ./tests/0001_main_exit.ol:2:10: <:> +# tokens: ./tests/0001_main_exit.ol:2:12: +# tokens: ./tests/0001_main_exit.ol:2:16: <{> +# tokens: ./tests/0001_main_exit.ol:2:17: +# tokens: ./tests/0001_main_exit.ol:3:3: +# tokens: ./tests/0001_main_exit.ol:3:10: +# tokens: ./tests/0001_main_exit.ol:3:11: +# tokens: ./tests/0001_main_exit.ol:4:1: <}> -- 2.34.1