https://ocamlpro.com/blog/2023_03_18_autofonce/ OCamlPro * Home * Services + Domain Specific Languages + OCaml Language + Rust Language + Cobol Language + Blockchains + Audit and Formal Verification + Alt-Ergo Users' Club + Research & Development * Team + Our Team + Internships + Join Us * Software + Dedicated Languages (DSLs) o M Language o Optal Language o Liquidity Language + Formal Methods o Alt-Ergo Prover o Colibri2 Prover + OCaml Tooling o OPAM Package Manager o Flambda Optimizer o Try OCaml o Learn OCaml o Drom Project Manager o Opam Bin Plugin o OCaml Canvas o ocp-indent o OCaml Docs + COBOL Tooling o GnuCOBOL Compiler o Autofonce Tests + + Blockchain Tooling o Tezos Blockchain o Liquidity Language + Scilab Tooling o Scilab Online o Scilint Analyzer + ------------------------------------------------------------- + OCamlPro on Github * Trainings * Blog * Lang: en + English + Francais Table of contents * Introduction * The Gnu Autoconf Testsuite of GnuCOBOL * Main Features of Autofonce * Conclusion Autofonce, GNU Autotests Revisited [icon_perso] Authors: Fabrice Le Fessant [icon_calen] Date: 2023-06-27 [icon_categ] Category: OCaml [icon_tags] Tags: ocaml, tests, gnucobol, autoconf, autofonce, cobol ----------------------------------------------------------------- Since 2022, OCamlPro has been contributing to GnuCOBOL, the only fully open-source compiler for the COBOL language. To speed-up our contributions to the compiler, we developed a new tool, autofonce, to be able to easily run and modify the testsuite of the compiler, originally written as a GNU Autoconf testsuite. This article describes this tool, that could be useful for other project testsuites. Table of contents + Introduction + The Gnu Autoconf Testsuite of GnuCOBOL + Main Features of Autofonce + Conclusion Autofonce is a modern runner for GNU Autoconf Testsuite Autofonce is a modern runner for GNU Autoconf Testsuite Introduction[?] Since 2022, OCamlPro has been involved in a big modernization project for the French state: the goal is to move a large COBOL application, running on a former Bull mainframe (GCOS) to a cluster of Linux computers. The choice was made to use the most open-source compiler, GnuCOBOL, that had already been used in such projects. One of the main problems in such migration projects is that most COBOL proprietary compilers provide extensions to the COBOL language standard, that are not supported by other compilers. Fortunately, GnuCOBOL has good support for several mainstream COBOL dialects, such as IBM or Micro-Focus ones. Unfortunately, GnuCOBOL had no support at the time for the GCOS COBOL dialect developed by Bull for its mainframes. As a consequence, OCamlPro got involved in the project to extend GnuCOBOL with the support for the GCOS dialect needed for the application. This work implied a lot of (sometimes very deep) modifications of the compiler and its runtime library, both of them written in the C language. And of course, our modifications had first to pass the large existing testsuite of COBOL examples, and then extend it with new tests, so that the new dialect would continue to work in the future. This work lead us to develop autofonce, a modern open-source runner for GNU Autoconf Testsuites, the framework used in GnuCOBOL to manage its testsuite. Our tool is available on Github, with Linux and Windows binaries on the release page. The GNU Autoconf Testsuite of GnuCOBOL[?] GNU Autoconf is a set of powerful tools, developed to help developers of open-source projects to manage their projects, from configuration steps to testing and installation. As a very old technology, GNU Autoconf relies heavily on M4 macros both as its own development language, and as its extension language, typically for tests. In GnuCOBOL, the testsuite is in a sub-directory tests/, containing a file testsuite.at, itself including other files from a sub-directory testsuite.src/. As an example, a typical test from syn_copy.at looks like: AT_SETUP([INITIALIZE constant]) AT_KEYWORDS([misc]) AT_DATA([prog.cob], [ IDENTIFICATION DIVISION. PROGRAM-ID. prog. DATA DIVISION. WORKING-STORAGE SECTION. 01 CON CONSTANT 10. 01 V PIC 9. 78 C78 VALUE 'A'. PROCEDURE DIVISION. INITIALIZE CON. INITIALIZE V. INITIALIZE V, 9. INITIALIZE C78, V. ]) AT_CHECK([$COMPILE_ONLY prog.cob], [1], [], [prog.cob:10: error: invalid INITIALIZE statement prog.cob:12: error: invalid INITIALIZE statement prog.cob:13: error: invalid INITIALIZE statement ]) AT_CLEANUP Actually, we were quite pleased by the syntax of tests, it is easy to generate test files (using AT_DATA macro) and to test the execution of commands (using AT_CHECK macro), checking its exit code, its standard output and error output separately. It is even possible to combine checks to run additional checks in case of error or success. In general, the testsuite is easy to read and complete. However, there were still some issues: + At every update of the code or the tests, the testsuite runner has to be recompiled; + Running the testsuite requires to be in the correct sub-directory, typically within the _build/ sub-directory; + By default, tests are ran sequentially, even when many cores are available. + The output is pretty verbose, showing all tests that have been executed. Failed tests are often lost in the middle of other successful tests, and you have to wait for the end of the run to start investigating them; ## -------------------------------------------- ## ## GnuCOBOL 3.2-dev test suite: GnuCOBOL Tests. ## ## -------------------------------------------- ## General tests of used binaries 1: compiler help and information ok 2: compiler warnings ok 3: compiler outputs (general) ok 4: compiler outputs (file specified) ok 5: compiler outputs (path specified) ok 6: compiler outputs (assembler) ok 7: source file not found ok 8: temporary path invalid ok 9: use of full path for cobc ok 10: C Compiler optimizations ok 11: invalid cobc option ok 12: cobcrun help and information ok 13: cobcrun validation ok 14: cobcrun -M DSO entry argument ok 15: cobcrun -M directory/ default ok [...] + There is no automatic way to update tests, when their output has changed. Every test has to be updated manually. + In case of error, it is not always easy to rerun a specific test within its directory. With autofonce, we tried to solve all of these issues... Main Features of Autofonce[?] autofonce is written in a modern language, OCaml, so that it can handle a large testsuite much faster than GNU Autoconf. Since we do not expect users to have an OCaml environment set up, we provide binary versions of autofonce for both Linux (static executable) and Windows (cross-compiled executable) on Github. autofonce does not use m4, instead, it has a limited support for a small set of predefined m4 macros, typically supporting m4 escape sequences (quadrigraphs), but not the addition of new m4 macros, and neither the execution of shell commands outside of these macros (yes, testsuites in GNU Autoconf are actually sh shell scripts with m4 macros...). In the case of GnuCOBOL, we were lucky enough that the testsuite was well written and avoided such problems (we had to fix only a few of them, such as including shell commands into AT_CHECK macros). The syntax of tests is documented here. Some interesting features of autofonce are : + autofonce executes the tests in parallel by default, using as many cores as available. Only failed tests are printed, so that the developer can immediately start investigating them; + autofonce can be run from any directory in the project. A .autofonce file has to be present at the root of the project, to describe where the tests are located and in which environment they should be executed; + autofonce makes it easy to re-execute a specific test that failed, by generating, within the test sub-directory, a script for every step of the test; + autofonce provides many options to filter which tests should be executed. Tests can be specified by number, range of numbers, keywords, or negative keywords. The complete list of options is easily printable using autofonce run --help for example; Additionnally, autofonce implements a powerful promotion mechanism to update tests, with the autofonce promote sub-command . For example, if you update a warning message in the compiler, you would like all tests where this message appears to be modified. With autofonce, it is as easy as: # Run all tests at least once autofonce run # Print the patch that would be applied in case of promotion autofonce promote # Apply the patch above autofonce promote --apply # Combine running and promotion 10 times: autofonce run --auto-promote 10 The last command iterates promotion 10 times: indeed, since a test may have multiple checks, and only the first failed check of the test will be updated during one iteration (because the test aborts at the first failed check), as many iterations as the maximal number of failed checks within a test may be needed. Also, as for GNU Autoconf, autofonce generates a final log file containing the results with a full log of errors and files needed to reproduce the error. This file can be uploaded into the artefacts of a CI system to easily debug errors after a CI failure. Conclusion[?] During our work on GnuCOBOL, autofonce improved a lot our user experience of running the testsuite, especially using the auto-promotion feature to update tests after modifications. We hope autofonce could be used for other open-source projects that already use the GNU Autoconf testsuite. Of course, it requires that the testsuite does not make heavy use of shell features and mainly relies on standard m4 macros. We found that the format of GNU Autoconf tests to be quite powerful to easily check exit codes, standard outputs and error outputs of shell commands. autofonce could be used to help using this format in projects, that do not want to rely on an old tool like GNU Autoconf, and are looking for a much more modern test framework. ----------------------------------------------------------------- About OCamlPro: OCamlPro is a R&D lab founded in 2011, with the mission to help industrial users benefit from experts with a state-of-the-art knowledge of programming languages theory and practice. o We provide audit, support, custom developer tools and training for both the most modern languages, such as Rust , WASM and OCaml, and for legacy languages, such as COBOL or even home-made domain-specific languages; o We design, create and implement software with great added-value for our clients. High complexity is not a problem for our PhD-level experts. For example, we developed the prototype of the Tezos proof-of-stake blockchain. o We have a long history of creating open-source projects, such as the Opam package manager, the LearnOCaml web platform, and contributing to other ones, such as the Flambda optimizing compiler, or the GnuCOBOL compiler. o We are also experts of Formal Methods, developing tools such as our SMT Solver Alt-Ergo (check our Alt-Ergo Users' Club) and using them to prove safety or security properties of programs. Please reach out, we'll be delighted to discuss your challenges: contact@ocamlpro.com or book a quick discussion. Most Recent Articles 2023 + Autofonce, GNU Autotests Revisited + Sub-single-instruction Peano to machine integer conversion + Statically guaranteeing security properties on Java bytecode: Paper presentation at VMCAI 23 + Release of ocplib-simplex, version 0.5 + The Growth of the OCaml Distribution 2022 + WebAssembly/Wasm and OCaml + Alt-Ergo: the SMT solver with model generation + Let's Encrypt Wildcard Certificates Made Easy with Agnos + opam 2.1.3 is released! + OCamlPro at the JFLA2022 Conference + 2021 at OCamlPro 2021 + Verification for Dummies: SMT and Induction + Generating static and portable executables with OCaml + opam 2.1.0 is released! + opam 2.0.9 release + Detecting identity functions in Flambda + Detection de fonctions d'identite dans Flambda + opam 2.1.0~rc2 released + Tutorial: Format Module of OCaml + Reunion annuelle du Club des utilisateurs d'Alt-Ergo 2021 [Show all] ----------------------------------------------------------------- (c) 2011-2022 OCamlPro * Twitter * LinkedIn * Email * Phone * 21 rue de Chatillon, 75014 Paris, France * Legal Notice * [icon_atom_]