Significance of failed tests in Emacs 28.1 ("make check" et al.) reddit / emacs ---------------------------------------------------------------------- colemacsen · 2022-07-11 12:26 For the past few months, I've used Emacs 28.1 (compiled from the official release tarball) with no *apparent* bugs. I've primarily used Debian Unstable, but I've also tried this on Fedora 36 and Arch. Changing from gcc11 to gcc12 doesn't change the situation. It seems that no matter what dependencies I install or which configuration options I provide, my compiled Emacs will never pass a handful of the tests provided by `make check-all`. Why is this the case? Has anyone managed to run 'make check-all' without any resulting failures, and if so, how? Are the failed tests in levels above 'make check' telling me anything significant? --Editing to fix formatting and (hopefully) simplify my question-- Here's the result of running make check-all after compiling with the `--with-xwidgets` option: SUMMARY OF TEST RESULTS ----------------------- Files examined: 375 Ran 5568 tests, 1 failed to run, 5456 results as expected, 13 unexpected, 99 skipped 1 files did not finish: misc/test-custom-noloads.log 7 files contained unexpected results: src/thread-tests.log lisp/progmodes/python-tests.log lisp/mail/rmailmm-tests.log lisp/emacs-lisp/cconv-tests.log lisp/cedet/srecode/fields-tests.log lisp/cedet/srecode/document-tests.log lisp/autorevert-tests.log make[2]: *** [Makefile:336: check-doit] Error 2 make[2]: Leaving directory '/home/src/emacs/emacs-28.1-tarball/emacs-28.1/test' make[1]: *** [Makefile:315: check-all] Error 2 make[1]: Leaving directory '/home/src/emacs/emacs-28.1-tarball/emacs-28.1/test' make: *** [Makefile:994: check-all] Error 2 All the tests that fail are performed only when I run `make check- all`. They aren't run for `make check` or `make check-expensive`. I'll post more information in the comments shortly. ---------------------------------------------------------------------- colemacsen · 2022-07-11 13:18 For the life of me I couldn't get the code to display properly on [old.reddit.com](https://old.reddit.com), but it should work correctly on the 'new' reddit. ---------------------------------------------------------------------- TheKrister2 · 2022-07-11 13:54 It is because you're using backticks instead of indenting the text you want to be in a code block. Both methods work on new reddit, but only indenting works on old reddit. ---------------------------------------------------------------------- eli-zaretskii · 2022-07-11 17:59 First, at least one of the results above says "0 unexpected", which means success. And you don't say which tests failed, so it's hard to say anything intelligent about the possible reasons. ---------------------------------------------------------------------- 7890yuiop · 2022-07-11 18:06 As /u/TheKrister2 says, the *only* code formatting method which works for all of reddit is to switch the reddit editor to markdown mode and indent the code by 4 spaces (you can use `M-4 C-x C-i` on a region followed by `M-x untabify` to achieve this in Emacs). Remove backticks and/or any other code formatting syntax. ---------------------------------------------------------------------- colemacsen · 2022-07-11 19:03 I think my initial post was overly complicated. My intent was to show that the failed tests are only performed when I run `make check-all`, and not when I run `make check-expensive` or `make check`. That's why I included multiple test summaries, of which some showed '0 unexpected'. It should be easier to follow now. Here are two questions that I hope will help me understand the situation: * When building Emacs from source, should a user expect to only get 'expected' results for each test run by `make check-all`? * Does the fact that these tests are included only in `check-all` say anything about their importance? Or does it say only that they take a long time to run? Below is information about which tests failed. Happy to provide any other info that might prove useful. From src/thread-tests.log: 1 unexpected results: FAILED threads-join-error From lisp/progmodes/python-tests.log: 1 unexpected results: FAILED python-tests--python-nav-end-of-statement--infloop From lisp/mail/rmailmm-tests.log: 2 unexpected results: FAILED rmailmm-test-bulk-handler FAILED rmailmm-test-multipart-handler From lisp/emacs-lisp/cconv-tests.log: 2 unexpected results: PASSED cconv-tests-cl-iter-defun-:documentation PASSED cconv-tests-iter-defun-:documentation From lisp/cedet/srecode/fields-tests.log: 1 unexpected results: FAILED srecode-field-utest-impl From lisp/cedet/srecode/document-tests.log: 1 unexpected results: FAILED srecode-document-function-comment-extract-test From lisp/autorevert-tests.log: 5 unexpected results: FAILED auto-revert-test01-auto-revert-several-files-remote FAILED auto-revert-test02-auto-revert-deleted-file-remote FAILED auto-revert-test04-auto-revert-mode-dired-remote FAILED auto-revert-test05-global-notify-remote FAILED auto-revert-test06-write-file-remote ---------------------------------------------------------------------- colemacsen · 2022-07-11 19:05 Thanks. I think it was the need for an empty line before the indented lines that was throwing me off. Should now be correctly formatted. ---------------------------------------------------------------------- colemacsen · 2022-07-11 19:05 Thanks. I think it was the need for an empty line before the indented lines that was throwing me off. Should now be correctly formatted. ---------------------------------------------------------------------- 7890yuiop · 2022-07-11 20:02 > the need for an empty line before the indented lines Thanks, I'll add that to my standard reply for this issue :) ---------------------------------------------------------------------- rpluim · 2022-07-12 10:05 `make check-all` runs *all* the tests, including ones that might randomly fail. The normal thing to try is `make check`, which runs 5444 tests for me here. ---------------------------------------------------------------------- colemacsen · 2022-07-12 15:52 This is really helpful, thanks. That resolves my concerns about the unexpected results from `make check-all`. Does your advice also apply to tests run by `make check-expensive`, but not `make check`? I'd like to understand an unexpected test result that appears when I add `--with-native-compilation` to my configuration flags and then run `make check-expensive`. Again, this unexpected test result appears consistently on all distributions I've tried. SUMMARY OF TEST RESULTS ----------------------- Files examined: 375 Ran 5699 tests, 5602 results as expected, 1 unexpected, 96 skipped 1 files contained unexpected results: lisp/net/tramp-archive-tests.log make[2]: *** [Makefile:336: check-doit] Error 1 make[2]: Leaving directory '/home/src/emacs/emacs-28.1-tarball/emacs-28.1/test' make[1]: *** [Makefile:310: check-expensive] Error 2 make[1]: Leaving directory '/home/src/emacs/emacs-28.1-tarball/emacs-28.1/test' make: *** [Makefile:994: check-expensive] Error 2 Unfortunately, I'm not yet able to understand the backtrace in `tramp- archive-tests.log`. Here are the relevant lines from that file: Test tramp-archive-test45-auto-load backtrace: signal(ert-test-failed (((should (string-match (format "tramp- archiv ert-fail(((should (string-match (format "tramp-archive loaded: %s[[: (if (unwind-protect (setq value-910 (apply fn-908 args-909)) (setq f (let (form-description-912) (if (unwind-protect (setq value-910 (app (let ((value-910 'ert-form-evaluation-aborted-911)) (let (form- descr (let* ((fn-908 #'string-match) (args-909 (condition-case err (let (( (let ((file (car --dolist-tail--))) (let* ((fn-908 #'string- match) ( (while --dolist-tail-- (let ((file (car --dolist-tail--))) (let* ((f (let ((--dolist-tail-- (list "/mock::foo" (concat tramp-archive- test (let ((default-directory (car --dolist-tail--))) (let ((--dolist-tai (while --dolist-tail-- (let ((default-directory (car --dolist- tail-- (let ((--dolist-tail-- (list temporary-file-directory (file- name-as- (let ((code "(progn \11 (message \"tramp-archive loaded: %%s\" . (let ((lexical-binding t)) (let ((value-901 (gensym "ert-form- evalua (closure (tramp-copy-size-limit t) nil (let ((lexical-binding t)) (l ert--run-test-internal(#s(ert--test-execution-info :test #s(ert- test ert-run-test(#s(ert-test :name tramp-archive-test45-auto-load :docum ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m ert-run-tests((not (tag :unstable)) #f(compiled-function (event- type ert-run-tests-batch((not (tag :unstable))) ert-run-tests-batch-and-exit((not (tag :unstable))) command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/net/tramp- archive-te command-line() normal-top-level() Test tramp-archive-test45-auto-load condition: (ert-test-failed ((should (string-match (format "tramp-archive loaded: %s[[:ascii:]]+tramp-archive loaded: %s" ... ...) (shell-command-to-string ...))) :form (string-match "tramp-archive loaded: t[[:ascii:]]+tramp- archive loaded: t" "Cannot open load file: Too many open files, debug\nError in kill-emacs-hook (#[0 \"\\301\\300\\302\\\"\\207\" [\"/home/src/emacs/emacs-28.1-tarball/emacs-28.1/test/lisp/net/tramp- archive-resources/foo.iso/emacs-testsuite-gH3jR3\" delete-directory t] 3]): (file-error \"Cannot open load file\" \"Too many open files\" \"tramp-archive\")\n") :value nil)) FAILED 21/22 tramp-archive-test45-auto-load (11.039797 sec) ... Ran 22 tests, 21 results as expected, 1 unexpected (2022-07-12 11:27:11-0400, 172.927142 sec) ... 1 unexpected results: FAILED tramp-archive-test45-auto-load Do you have any insight into why this test fails for builds compiled using `--with-native-compilation`? ---------------------------------------------------------------------- eli-zaretskii · 2022-07-12 17:34 You should only ever run `make check`. All the rest is for those pointy-hairy developers. ---------------------------------------------------------------------- colemacsen · 2022-07-12 17:45 That pretty much settles it, then. Much appreciated.