Code-based test generation

Overview

In software testing one of the challenging tasks is to select the test inputs. There are several methods that are capable of automatically generating test input based on the source code of the program under test. (However, these methods usually cannot generate the expected output for these inputs, as they do not have a specification or model). These methods usually try to satisfy kind of coverage (e.g. different branches and paths of the code under test).

Note: these tools use the code/binary of the program under test; see this list for model-based test generation tools.

Several methods can be used for code-based test generation, for example:

Random selection: random or adaptive random test generation could also yield good results.
Code annotations: if the code is annotated with pre/post conditions, these can drive test input selection.
Search-based techniques: search algorithms (e.g. genetic algorithms) are used guided by a fitness function representing some kind of coverage.
Symbolic execution based techniques: these include dynamic symbolic execution, concolic testing, etc.

There are several good overviews and surveys for the topic:

[A+13]: a general survey presenting symbolic execution techniques, model-based testing, combinatiorial testing, random testing, combinatorial testing and search-based testing.
[GA14]: a survey about gray-box and white-box test generation tools, and the capabilities of 7 tools.
[PV09]: a survey about the techniques applied in tools using symbolic execution.
symexbib: online list of papers on symbolic execution (includes not just testing usages)

Comparison: for an evaluation and comparison of some tools see our SETTE project.

Tools

A far from complete list of code-based test generation tools. (If you know about a tool that is not in the list, feel free to send me an email.)

Mod.: Last modification was checked on 2012.11.
Methods: ANN -- annotations, RAND -- random, SB -- search-based; SE -- symbolic execution;
Type: ACAD -- academic or research, COMM -- commercial, OSS -- open source software

Tool	AutoTest
Mod.	2021
Ref.	[M+09]
Input	Eiffel
Method	ANN
Type	COMM, OSS
Desc.	AutoTest uses assertions in Design by Contract to generate test cases. It automates test creation, test minimization. It is now integrated into the EiffelStudio IDE.

Tool	EvoSuite
Mod.	2021
Ref.	[FA13]
Input	Java
Method	SB
Type	ACAD, OSS
Desc.	EvoSuite is a tool that automatically generates test cases with assertions for classes written in Java code. To achieve this, EvoSuite applies a novel hybrid approach that generates and optimizes whole test suites towards satisfying a coverage criterion.

Tool	Jtest
Mod.	2021
Ref.
Input	Java
Method	?
Type	COMM
Desc.	Jtest is an automated Java software testing and static analysis product that is made by Parasoft. The product includes technology for Unit test-case generation and execution, static analysis, regression testing, runtime error detection, code review, and Design by contract.

Tool	KLEE
Mod.	2022
Ref.	[CDE08]
Input	C (LLVM bitcode)
Method	SE
Type	OSS
Desc.	KLEE is capable of automatically generating tests that achieve high coverage on a diverse set of complex and environmentally-intensive programs. KLEE is the successor of EXE.

Tool	Pex
Mod.	2021
Ref.	[TH08]
Input	.NET code
Method	SE
Type	ACAD, COMM
Desc.	With Pex testers create parametrized unit tests and then Pex finds input parameters for these test that result in high coverage, boundary conditions, exceptions and assertion failures. Pex applies dynamic symbolic execution and an underlying constraint solver. Pex is now included in Visual Studio as IntelliTest.

Tool	QuickCheck
Mod.	2022
Ref.	[CH11]
Input	Haskell
Method	RAND
Type	OSS
Desc.	QuickCheck is a library for random testing of program properties. The programmer provides a specification of the program, in the form of properties which functions should satisfy, and QuickCheck then tests that the properties hold in a large number of randomly generated cases. There are commercial versions for C, Erlang developed by QuviQ

Tool	SymFlower
Mod.	2022
Ref.
Input	Java, Go
Method	SE
Type	COMM
Desc.	Symflower utilizes symbolic execution to generate manageable test suites that contain only essential tests. It supports several modern IDEs.

Tool	Randoop
Mod.	2022
Ref.	[P+07]
Input	Java
Method	RAND
Type	OSS
Desc.	Randoop generates unit tests using feedback-directed random test generation. Randoop executes the sequences it creates, using the results of the execution to create assertions that capture the behavior or the program and to catch bugs.

Tool	Symbolic PathFinder
Mod.	2022
Ref.	[P+13]
Input	Java
Method	SE
Type	OSS
Desc.	Symbolic PathFinder (SPF) combines symbolic execution with model checking and constraint solving for test case generation.

Tool	Testrecorder
Mod.	2020
Ref.
Input	Java
Method	Capture - Replay
Type	OSS
Desc.	Testrecorder is a tool that runs as an agent while executing an application, recording the state before and after annotated methods. The output of such a recording are executable JUnit-Tests that replay and verify all recorded samples.

Tools not developed any more

Tool	AgitarOne
Mod.	2015
Ref.	[BDS06]
Input	Java
Method
Type	COMM
Desc.	AgitarOne is a commercial tool for Java combining dynamic invariant detection and test-input generation. It exercises classes with various input data and create 'observations' that represent the behavior of the unit, which developers can convert to assertions.

Tool	CATG
Mod.	2015
Ref.
Input	Java
Method	SE
Type	OSS
Desc.	CATG is a concolic unit testing engine for Java programs (built by the developers of jCUTE).

Tool	CAUT
Mod.	2014
Ref.	[S+15]
Input	C
Method	SE
Type	ACAD
Desc.	CAUT is a DSE (dynamic symbolic execution)-based tool to automatically generate test data for C program at unit/program testing level. It currently supports the coverage-driven testing on both control flow and data flow coverage, e.g., branch, MC/DC and all def-use criteria.

Tool	CREST
Mod.	2014
Ref.	[BS08]
Input	C
Method	SE
Type	OSS
Desc.	CREST is an automatic test generation tool for C. It instruments the source code to perform symbolic execution, and uses a constraint solver to search for unexplored program paths.

Tool	CodePro Analytix
Mod.	2009
Ref.
Input	Java
Method
Type	COMM
Desc.	The CodePro JUnit Test Case Generation facility allows you to automate the creation of comprehensive JUnit regression test cases. The tool was acquired by Google, its development stopped.

Tool	GRT
Mod.	2015
Ref.	[M+15]
Input	Java
Method	RAND
Type	ACAD
Desc.	GRT (Guided Random Testing) is an automatic test generation tool for Java code, which leverages static and dynamic program analysis to guide run-time test generation. It uses feedback-directed random testing and is based on Randoop.

Tool	GUITAR
Mod.	2015
Ref.	[NRBM13]
Input	GUI application (JFC, Web)
Method
Type	OSS
Desc.	GUI Testing frAmewoRk: "rips" the GUI of an application and creates an event-flow graph model from it. GUITAR is open-source, there is a lot of publications and case studies with it. Beta versions are available for testing other types of applications (e.g. iPhone).

Tool	Jalangi
Mod.	2015
Ref.	[SKBG13]
Input	JavaScript
Method	SE
Type	OSS
Desc.	Jalangi is a framework for writing heavy-weight dynamic analyses for JavaScript, like concolic testing or taint analysis.

Tool	JSeft
Mod.	2014
Ref.	[MMP15]
Input	JavaScript
Method
Type	OSS, ACAD
Desc.	JSeft generates function-level unit tests and DOM-level event-based test cases for JavaScript code.

Tool	CUTE, jCUTE
Mod.	2006
Ref.	[SA06]
Input	C, Java
Method	SE
Type	ACAD
Desc.	A concolic unit testing engine for C (CUTE) and Java (jCUTE), which could handle multi-threaded programs and pointer operations.

Tool	DART
Mod.	2005
Ref.	[GKS05]
Input	C
Method	SE
Type	ACAD
Desc.	DART (Directed Automated Random Testing) was one of the first tools to use dynamic symbolic execution. (Tool was not publicly available.)

Tool	DOTgEAr
Mod.	2007
Ref.	[OS06]
Input	Java
Method	SB
Type	ACAD
Desc.	Dataflow-oriented Test Case Generation for Object-oriented Software Systems by Evolutionary Algorithm.

Tool	DSD-Crasher
Mod.	2007
Ref.	[SC07]
Input	Java
Method
Type	ACAD
Desc.	DSD-Crasher is the successor of JCrasher. It adds two steps before the test generation: runs existing tests to capture their behaviour, then uses static analysis to explore the paths of the program.

Tool	Euclide
Mod.	2009
Ref.	[G09]
Input	C
Method	SE
Type	ACAD
Desc.	Test generator for C, implemented as a web application. Uses contraint solvers to generate a test reaching a given point in the program.

Tool	EXE
Mod.	2006
Ref.	[C+08]
Input	C
Method	SE
Type	ACAD
Desc.	EXecution generated Executions (EXE) is a bug-finding tool that uses symbolic and concrete execution by turns. EXE was later redesigned in KLEE.

Tool	JCrasher
Mod.	2007
Ref.	[CS04]
Input	Java
Method
Type	ACAD
Desc.	JCrasher is an automatic robustness tester for Java. JCrasher examines the type information of a set of Java classes and constructs code fragments that will create instances of different types to test the behavior of public methods under random data.

Tool	JTExpert
Mod.	2015
Ref.	[SPG15]
Input	Java
Method	SB
Type	ACAD
Desc.	JTExport can automatically generate a whole test suite towards satisfying the branch coverage criterion on a given Java class.

Tool	JET
Mod.	2010
Ref.	[CR07]
Input	Java annotated with JML
Method	ANN
Type	ACAD
Desc.	JET is an automated unit testing tool for Java classes annotated with Java Modeling Language (JML) specifications. It generates random tests and uses the JML constraints as test oracles.

Tool	jFuzz
Mod.	2009
Ref.	[JHGK09]
Input	Java
Method	SE
Type	ACAD
Desc.	jFuzz is a concolic whitebox fuzzer, built on the NASA Java PathFinder.

Tool	Korat
Mod.	2011
Ref.	[BKM02]
Input	Java
Method
Type	OSS
Desc.	Korat is a tool for constraint-based generation of structurally complex test inputs for Java programs. Korat requires (1) an imperative predicate that specifies the desired structural constraints and (2) a finitization that bounds the desired test input size.

Tool	LCT
Mod.	2012
Ref.	[K+11]
Input	Java
Method	SE
Type	ACAD
Desc.	The LIME testbench is a collection of tools that allows concolic testing of Java programs.

Tool	Mika
Mod.	2011
Ref.
Input	Ada
Method	?
Type	COMM
Desc.	Mika is an entirely automatic tool that analyses your Ada code and generates test inputs that will exercise all the branches, decisions or MC/DCs within your code at a level suitable for integration testing.

Tool	MergePoint/Mayhem
Mod.	2013
Ref.	[A+14]
Input	binary (32bit, Linux)
Method	SE
Type	COMM?
Desc.	MergePoint is a tool that combines dynamic and static symbolic execution. It is built on top of Mayhem (a tool that is used for exploit generation). MergePoint was run on more than 30000 binaries from Debian.

Tool	PathCrawler
Mod.	2013?
Ref.	[WMMR05]
Input	C
Method
Type	ACAD
Desc.	PathCrawler is a method to automatically generate path tests for the all-feasible-paths criterion with a user-defined limit on the number of loop iterations in the covered path. It has an online version and a version integrated into Frama-C.

Tool	Palus
Mod.	2012
Ref.	[Z+11]
Input	Java
Method	SE
Type	OSS
Desc.	Palus is an automated test generation tool, which combines various static and dynamic analysis to create tests for Java programs in the JUnit form.

Tool	PET, jPET
Mod.	2011
Ref.	[AGP10]
Input	Java
Method
Type	ACAD
Desc.	Partial Evaluation-based Test Case Generator for Bytecode (PET) transforms Java bytecode to equivalent CLP code, and generates test cases from it. It is implemented in Prolog, jPET is an Eclipse GUI for it.

Tool	SAGE
Mod.	2013?
Ref.	[GLM12]
Input	x86 binary
Method	SE
Type	COMM
Desc.	Scalable Automated Guided Execution (SAGE) is a whitebox fuzzing tool, which extends systematic dynamic test generation to whole-program security testing. It is internally used in Microsoft.

Tool	T3
Mod.	2015
Ref.	[PVB08]
Input	Java
Method	RAND
Type	OSS
Desc.	T3 (previously T2) is able to randomly generate sequences of calls to Java class' methods to test it.

Tool	TAO
Mod.	2004
Ref.
Input	C
Method	ANN
Type	ACAD
Desc.	TAO can generate test cases based on structural coverage, or from pre- and postconditions (with the CAVEAT tool). (Tool web page not available any more.)

Further tools not developed any more: GenUTest [PTY09], Klover [LGR11], TestGen4J, TestEra

References

[A+13]	S. Anand et al. (2013) "An orchestrated survey of methodologies for automated software test case generation," Journal of Systems and Software, 86:8, pp. 1978–2001. DOI: j.jss.2013.02.061
[AGP10]	E. Albert, M. Gomez-Zamalloa, and G. Puebla (2010) "PET: A Partial Evaluation-based Test Case Generation Tool for Java Bytecode." In Proc. PEPM, ACM, pp. 25-28. DOI: 10.1145/1706356.1706363
[A+14]	T. Avgerinos, A. Rebert, S. K. Cha, and D. Brumley (2014) "Enhancing Symbolic Execution with Veritesting", In Proc. ICSE'14, pp. 1083-1094. DOI: 10.1145/2568225.2568293
[BDS06]	M. Boshernitsan, R. Doong, A. Savoia (2006) "From Daikon to Agitator: lessons and challenges in building a commercial tool for developer testing", In Proc. of ISSTA '06, pp. 169-180. DOI: 10.1145/1146238.1146258
[BKM02]	C. Boyapati, S. Khurshid, and D. Marinov (2002) "Korat: Automated testing based on Java predicates", In Proc. ISSTA'02, pp. 123-133. DOI: 10.1145/566172.566191
[BS08]	J. Burnim and K. Sen (2008) "Heuristics for Scalable Dynamic Test Generation," In Proc. ASE'08, IEEE, pp. 443-446. DOI: 10.1109/ASE.2008.69
[C+08]	C. Cadar et al. (2008) "EXE: Automatically Generating Inputs of Death." ACM Trans. Inf. Syst. Secur. 12:2. DOI: 10.1145/1455518.1455522
[C+11]	C. Cadar et al. (2011) "Symbolic execution for software testing in practice: preliminary assessment". In Proc. of ICSE '11, ACM, 1066-1071. DOI: 10.1145/1985793.1985995
[CDE08]	C. Cadar, D. Dunbar, D. Engler (2008) "KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs." In Proc. of OSDI'08, pp. 209-224.
[CH11]	K. Claessen, J. Hughes (2011) "QuickCheck: a lightweight tool for random testing of Haskell programs". ACM SIGPLAN Notices 46:4, pp. 53-64. DOI: 10.1145/1988042.1988046
[CR07]	Y. Cheon and C. E. Rubio-Medrano (2007) "Random Test Data Generation for Java Classes Annotated with JML Specifications." In Proc. of ICSERP, pages 385-392.
[CS04]	C. Csallner and Y. Smaragdakis (2004) "JCrasher: An automatic robustness tester for Java," Software -- Practice ∧ Experience, vol. 34, no. 11, pp. 1025-1050. DOI: 10.1002/spe.602
[FA13]	G. Fraser and A. Arcuri (2013) "Whole Test Suite Generation," IEEE Trans. on Software Engineering, vol. 39, iss. 2, pp. 276-291. DOI: 10.1109/TSE.2012.14
[G09]	A. Gotlieb (2009) "EUCLIDE: A Constraint-Based Testing platform for critical C programs." In Proc. ICST'09, IEEE, pp. 151-160. DOI: 10.1109/ICST.2009.10
[GA14]	S. Galler, B. Aichernig (2014) "Survey on test data generation tools" STTT 16:6, Springer, pp. 727-751. DOI: 10.1007/s10009-013-0272-3
[GKS05]	P. Godefroid, N. Klarlund, and K. Sen. (2005) "DART: directed automated random testing." In Proc. PLDI '05. ACM, pp. 213-223. DOI: 10.1145/1065010.1065036
[GLM12]	P. Godefroid, M. Y. Levin, and D. Molnar. (2012) "SAGE: Whitebox Fuzzing for Security Testing." ACM Queue 10:1. DOI: 10.1145/2090147.2094081
[JHGK09]	K. Jayaraman, D. Harvison, V. Ganesh, A. Kiezun (2009) "jFuzz: A Concolic Whitebox Fuzzer for Java," NASA Formal Methods Workshop.
[K+11]	K. Kahkonen et al. (2011) "LCT: An Open Source Concolic Testing Tool for Java Programs." In Proc. of BYTECODE'2011, pp. 75-80, Saarbrucken, Germany.
[LGR11]	G. Li, I. Ghosh, S. P. Rajan (2011) "KLOVER: A Symbolic Execution and Automatic Test Generation Tool for C++ Programs". In CAV'11, pp. 609-615. DOI: 10.1007/978-3-642-22110-1_49
[M+09]	B. Meyer et al. (2009) "Programs That Test Themselves," Computer, vol. 42, 2009, pp. 46-55. DOI: 10.1109/MC.2009.296
[M+15]	Lei Ma et al. (2015) "GRT: Program-Analysis-Guided Random Testing ," in ASE'15, pp. 212-223. DOI: 10.1109/ASE.2015.49
[MMP15]	S. Mirshokraie, A. Mesbah, and K. Pattabiraman (2015) "JSEFT: Automated JavaScript Unit Test Generation". In Proc. ICST'15. DOI: 10.1109/ICST.2015.7102595
[NRBM13]	B. N. Nguyen, B. Robbins, I. Banerjee, A. Memon (2013) "GUITAR: an innovative tool for automated testing of GUI-driven software" Automated Software Engineering, Springer, pp. 1-41. DOI: 10.1007/s10515-013-0128-9
[OS06]	N. Oster and F. Saglietti (2006) "Automatic Test Data Generation by Multi-objective Optimisation," In Proc SAFECOMP'06, Springer, pp. 426-438. DOI: 10.1007/11875567_32
[P+07]	C. Pacheco et al. (2007) "Feedback-directed random test generation," In Proc. of ICSE'07, IEEE, pp. 75-84. DOI: 10.1109/ICSE.2007.37
[P+13]	C. S. Pasareanu et al. (2013) "Symbolic PathFinder: integrating symbolic execution with model checking for Java bytecode analysis," Automated Software Engineering 20:3, pp 391-425. DOI: 10.1007/s10515-013-0122-2
[PTY09]	B. Pasternak, S. Tyszberowicz, A. Yehudai (2009) "GenUTest: a unit test and mock aspect generation tool". STTT 11:4, pp. 273-290. DOI: 10.1007/s10009-009-0115-4
[PV09]	C. S. Pasareanu, W. Visser (2009) "A survey of new trends in symbolic execution for software testing and analysis". STTT, 11(4): 339-353. DOI: 10.1007/s10009-009-0118-1
[PVB08]	I.S.W.B. Prasetya, T.E.J. Vos, A.I. Baars (2008) "Trace-based Reflexive Testing of OO Programs with T2". ICST'08, pp. 151-160. DOI: 10.1109/ICST.2008.12
[QR11]	X. Qu, B. Robinson (2011) "A Case Study of Concolic Testing Tools and their Limitations". In Proc. ESEM'11, IEEE, pp.117-126. DOI: 10.1109/ESEM.2011.20
[S+15]	T. Su et al. "Combining symbolic execution and model checking for data flow testing". In Proc. ICSE 2015, pp. 654-665. DOI: 10.1109/ICSE.2015.81
[SA06]	K. Sen and G. Agha (2006) "CUTE and jCUTE : Concolic Unit Testing and Explicit Path Model-Checking Tools," In Proc. CAV'06, Springer. DOI: 10.1007/11817963_38
[SC07]	Y. Smaragdakis and C. Csallner (2007) "Combining static and dynamic reasoning for bug detection" In Proc. TAP'07, Springer, pp. 1-16. DOI: 10.1007/978-3-540-73770-4_1
[SKBG13]	K. Sen, S. Kalasapur, T. Brutch, and S. Gibbs (2013) "Jalangi: A Selective Record-Replay and Dynamic Analysis Framework for JavaScript." In Proc. ESEC/FSE'13. DOI: 10.1145/2491411.2491447
[SPG15]	A. Sakti, G. Pesant, Y.-G. Guéhéneuc (2015) "Instance Generator and Problem Representation to Improve Object Oriented Code Coverage," in Software Engineering, IEEE Transactions on, vol.41, no.3, pp.294-313. DOI: 10.1109/TSE.2014.2363479
[TH08]	N. Tillmann and J. de Halleux (2008) "Pex -- White Box Test Generation for .NET," In Prof. TAP'08, Springer, pp. 134-153. DOI: 10.1007/978-3-540-79124-9_10
[WMMR05]	N. Williams, B. Marre, P. Mouy, M. Roger. (2005) "PathCrawler: automatic generation of path tests by combining static and dynamic analysis." In Proc. EDCC-5, Springer, pp. 281-292. DOI: 10.1007/11408901_21
[Z+11]	S. Zhang et al. (2011) "Combined static and dynamic automated test generation," In Proc. ISSTA'11, ACM, pp. 353-363. DOI: 10.1145/2001420.2001463
	DOI:

Last modified: 2022. 11. 24.