est-arranger is a tool for arranging data in tests. It was designed on top of an approach
driven by randomness and several test design patterns described in xUnit Test Patterns.
There is a brief introduction to the solution on the project’s GitHub page
(https://github.com/ocadotechnology/test-arranger) and a more comprehensive one in the
“On testing in DDD” post (https://medium.com/@marian.jureczko/on-testing-in-ddd-f250482b5717)
In short, the test-arranger instantiates classes initialized with random data. The solution has
proven successful in a couple of projects that I run at Ocado Technology so I have decided
to share my experience with a wider audience. So far, my talks about the solution generated
positive interest and were followed by engaging questions to which answers may be very
informative for everyone considering test-arranger adoption.
What should I do when I need instances with a different set of parameters for different
test cases?
The custom arranger comes with an instance method that should be considered the default
recipe for a certain class. It will be used each time while the test-arranger is creating an
instance of the class. However, the custom arranger is also the place where all the factory
methods for the certain class should be located. Let’s assume that a particular test case
needs an instance carrying values that the custom arranger instance method is not able to
deliver. To address this challenge, we should ask the test-arranger to create the random
instance for use and then adjust the instance to the test cases needs (we can use whatever
the instantiated class offers: setters, toBuilder, Kotlin’s copy method… and if the class offers
nothing we can rely on test-arranger some method with the overrides parameter
(https://www.javadoc.io/doc/com.ocadotechnology.gembus/test-arranger/latest/com/ocadote
chnology/gembus/test/Arranger.html). If there is even a slight possibility that the adjustment
can be reused in a different test case, we should pack it into a factory method and locate in
the arranger class to make it easy to find (and reuse) in the future.
Is there a way to repeat a test with the same values?
Test-arranger has a fixed randomization seed. So, when launching a single test, again and
again, it always gets precisely the same values. Nonetheless, in the scale of a whole
software system, many moving parts can affect the randomization, like tests execution order
or the details of classpath processing. Therefore, when executing an entire test suite, the
tests may receive different values at different runs.
Is it possible to generate a set of values and execute a test for each of them?
Test-arranger is not aware of the context it is executed, and consequently, it cannot control
the test execution engine. So it is not possible out of the box. Nonetheless, you can create a
parameterized test (e.g. https://junit.org/junit5/docs/current/user-guide/#writing-tests-parameterized-tests) and in the values source call test-arranger to generate a set of random data.
Can I use test-arranger for property based testing?
Test-arranger is not designed for property based testing. Property testing is one of the three
solutions to the problem of lack of deterministic test oracle
(https://en.wikipedia.org/wiki/Test_oracle). They are used to generate a cohort of different
random values to a variable used as the object of a test and to check that the system is
responding correctly for each of the values. While test-arranger is focused on the data that is
not crucial for the test, it must nonetheless, be there to make it possible to execute the test.
So, those are complementary approaches that are focused on optimizing two separate parts
of the test.
Does test-arranger support javax validation annotations?
Test-arranger uses EasyRandom (https://github.com/j-easy/easy-random) under the hood
which has the Bean Validation support (https://github.com/j-easy/easy-random/wiki/bean-val
idation-support).
Maybe the noise that test-arranger eliminates from tests is a consequence of a badly
designed domain and instead of optimizing the process of test data creation, we
should refactor the domain.
Indeed test-arranger eliminates a noise that comes from low cohesion between the test
cases and the system’s domain under test. It lets us eliminate the code that is required by
the domain but has nothing to do with the test objective. So, if our domain is perfectly
adjusted to the test cases (a perfect cohesion – test uses all fields of the entity that is used in
the test) there would be no place for random data and, as a consequence, test-arranger.
Unfortunately, that is a theoretical possibility as the domain primarily must be adjusted to
business use cases. There are always test cases that use only a part of the entities. While
the remaining parts, which are not needed for the test cases, need to be somehow arranged
to satisfy the domain invariants and that is the place where the test-arranger comes to the
rescue.
Does test-arranger lead to blinking tests?
There is no direct link between test-arranger and blinking tests. Nonetheless, test-arranger
uses random data and randomness has a lot to do with blinking tests. But when considering
random data as test input, we should be aware that there are two kinds of blinking tests:
false positives and false negatives and there is a huge difference between them. The false
positives are tests that sometimes fail despite the production code’s correctness. Such tests
do much harm as they undermine our confidence in our test suite. The false negatives are
tests that only sometimes (usually under conditions we are not aware of) are able to detect a
defect. They do nothing severe with our confidence as the failing test still means a defect.
There are only challenges in reproducing the issue and identifying the defect preconditions.
Random test data may result in both scenarios. However, based on my own
experience, I can say that both are seldom, and the second one is more frequent than the
first one. To sum up, introducing the test-arranger creates an additional opportunity (but not
a big one) for detecting defects that otherwise would be overlooked.
Does the test-arranger work only in Kotlin?
I believe that Kotlin is a great language for writing tests and test-arranger has a Kotlin API,
but it also has a JAVA API and it is usable in most of the JVM languages.
When is the test-arranger approach not the way to go?
It is hard to point to a category of projects where the test-arranger should not be used –
nonetheless, the more complex data to arrange the more significant benefit it gives. Hence,
if a system is not focused on data, but rather on algorithms (e.g. a parser), the benefits of
using test-arranger will be small.
I already use Test Data Builders. Will be using test-arranger an improvement for me?
Test Data Builder is a concept that is with us for quite a long time, e.g.
http://www.natpryce.com/articles/000714.html, and it is a great improvement over ad hoc test
data creation. Let’s consider two possibilities to compare the Builder with the test-arranger
approach.
- The builder is part of the production code.
It is convenient to have the builders (or at least a copy constructor) in the production code
regardless of the test-arranger presence.
Anyway, the given section of our tests may differ when there are domain invariants to satisfy.
A builder should check the invariants in the build() method and throw an exception when
they are not satisfied. As a consequence, if our Product class requires a Category, then
in every test requiring the Product there will be a .withCategory(…) line (it can be
wrapped in a factory method to avoid code duplication, but there will be still the additional
complexity in our code ). With the test-arranger we can satisfy this requirement once in the
ProductArranger and have it automatically applied to every Product instantiation.
Additionally, it will introduce a separation between test cases and domain invariant support.
There is also a difference in what happens under the hood. The builder does not initialize
unmentioned fields (unless you implement an initialization). While having all fields initialized
creates an additional opportunity for detecting unexpected defects. - The builder needs to be delivered in the testing code.
When implementing the builder in your testing code, you need to call a constructor or a
factory method from the production code which creates coupling. Consequently, with further
development, the builder needs to reflect changes from the production code. That is a
maintenance cost you can avoid with the test-arranger.
Test Data Builders are really helpful, and replacing them with the test-arranger will
not bring such big benefits as restructuring unorganized tests. Nonetheless, there are the
listed above benefits and it makes sense to at least give the test-arranger a try. You may
replace a single builder with an arranger to find out if the reduction in test maintenance
justifies the effort needed for replacement.
How the random data generator handles complex, nested data structures?
Test-arranger uses EasyRandom (https://github.com/j-easy/easy-random) under the hood.
The EasyRandom is configured with collectionSizeRange equal (1,4) and
randomizationDepth equal 4. It controls how far the randomization will go in nested data
structures and recursive structures (e.g. when a Product has a list of other, nested products,
which has a list of other products…) You can find more details about the consequences of
the configuration at https://github.com/j-easy/easy-random/wiki/Randomization-parameters.
It is noteworthy that on top of the solution test-arranger goes through the deepest nesting
level and sets empty values there (e.g. empty List, Set…) to minimize the negative effect
of nulls left by EasyRandom.
Marian Jureczko
O autorze:
Programista Java z kilkunastoletnim doświadczeniem i pasjonat tematów związanych z jakością oprogramowania. Współtwórca narzędzia test-arranger oraz autor szeregu artykułów dotyczących testowania. Od ponad 6 lat w Ocado Technology.