OCaml Forge

Detail: [#1392] Backwards compatibility of return code using run_test_tt_main

Bugs: Browse | Download .csv | Monitor

[#1392] Backwards compatibility of return code using run_test_tt_main

Date:
2014-05-13 13:48
Priority:
3
State:
Open
Submitted by:
Si Beaumont (sjbx)
Assigned to:
Sylvain Le Gall (gildor-admin)
Severity:
critical
Component:
Core
Product:
OUnit library
Resolution:
Accepted As Bug
Version:
v2.0.0
 
URL:
Status:
Open
 
Summary:
Backwards compatibility of return code using run_test_tt_main

Detailed description
Here is a simple example of a failing case in the documented format:
-----
https://gist.github.com/simonjbeaumont/9dbb0771baf616c3493d#file-test-ml
-----
open OUnit

let test1 () = assert_failure "Test failure"

let suite = "suite" >::: ["test1" >:: test1]

let _ =
run_test_tt_main suite
-----

When this fails I expect a non-zero exit code from the executable. This worked in OUnit 1.1.2.

We recently upgraded the OUnit package to 2.0.0 in our environment and found that the exit code for the above is always 0. It correctly reports in the textual output that there is a failure but returns 0.

This silent failure is major for us because any failure won't be picked up by our automated build system.

I can get this to work as desired by opening OUnit2 instead of OUnit and replacing the () parameter of the test1 to _ to match against the test_ctxt type. However, I would need to do this for all of our existing test base.

But we maintain old build environments and we'd like to be able to keep backwards compatibility.

I would have understood this change in behaviour if OUnit2 updates had been made to the OUnit module itself but the separation of the two namespaces implies to me that I should be able to open the OUnit module and expect the same behaviour regardless of whether I have OUnit package version 2.0.0 or 1.x.

Followup

Message
Date: 2014-05-13 15:34
Sender: Sylvain Le Gall

The fixe is pretty straightforward (check result and exit with error code), but I encourage you to upgrade to OUnit2.

The reason of the split of interface was that there are some huge changes (i.e. test_ctxt) but the code is a lot more clean than before.... Not even considering the possible improvement of speed you'll get.

I need to do a new release but I have no ETA for that.

Attached Files:

Changes:

Field Old Value Date By
assigned_tonone2014-05-13 14:40gildor-admin
ResolutionNone2014-05-13 14:40gildor-admin