Test-first
When should we start testing a system? In the traditional approaches, testing begins as soon as the developers finish their modules. This has raised several problems, like the fact that testers are idle during a significant part of development. To cope with this it is usually recommended that testers review the requirements descriptions. This activity prepares them do better understand what to test and to identify possible sources of errors. Ambiguous requirements are a powerful hint about what should be tested.
Another approach is to write the tests before the code. The development workflow is to write a test, run the test to verify that it fails, write the functionality that implements the test, and finally run the test to check that it passes. By following this approach you come out with the code and the tests of the implemented functionality. This is a well-known XP rule, named test-first. In test-first the goal of tests is two-fold: in the one hand they assess the quality of the code description, but, on the other hand they describe the problem that has to be implemented.
One of the criticisms on test-first is that tests are biased by the code. Actually, tests are not there to identify bugs but as a specification process, prior to the coding activity. However, this can be seen as an advantage, because it forces developers to make the problem explicit before writing the solution. The quality of the code results from the process used to write it. Anyway, test-first is not enough, and the quality assurance team and the clients also need to test the system.
But there is an advantage of test-first that surpasses the context of testing and deeply impacts on the development process; it promotes code modifiability. Developers are cautious about changing code because it easily breaks, and people do not like to loose functionality that is already implemented. With test-first we can run regression tests to verify if any functionality is lost by the last changes. Whenever a change is made the suite of test is run again. The hard part of debugging is the conceptual and temporal distance between the fault introduction and its failure detection. Regression testing reduces this gap.
Once developers are not afraid of changing the code it is possible to have another perspective on software design, a bottom-up approach based on code refactorings.
JUnit is an object-oriented framework that simplifies the process of testing. It automatically runs a suite of tests and collects their results. Each test is independent and its execution does not impact on others. Therefore, a test failure does not stop the execution of the test suite. JUnit is instrumental in the success of test-first approaches because developers can focus on the “business logic” of tests, ignoring the management of their execution.
Incomplete Specification
Software has bugs, and it cannot be proved that a program does not have any. On the other hand, several qualities of descriptions can be automatically proved. There is some controversy between different communities on the role, cost and extent of the application of formal techniques in software development.
In test-first we have seen the role of tests as a specification language. However, a set of test cases does not completely specify the problem. It may describe a set of situations considered relevant, but it does not address all situations. The test cases are an incomplete specification of the problem.
Design by contract was popularized by the Eiffel language. It allows to define pre- and post-conditions to specify the methods behavior. The execution of a method should guarantee that if it occurs in a state where the pre-conditions holds, the post-conditions will hold when the method terminates. Design-by-contract provides a complete specification of the problem.
In face of design-by-contract there is some debate on the role of Specification by Example as a useful technique to describe requirements due to its lack of rigor.
A way to define a boundary is to think in terms of S, P and E-systems. Although we aim to have complete specifications of problems, in P and E systems, their complexity cannot be captured by a model, or the cost of doing so it too high. In these cases the specifications are incomplete de facto. However, if we have a S-system, then it would be possible, and cost effective, to have a complete description of the problem.
Testing only proves that there are not bugs in what is tested. It is incomplete by definition.
Note that testing may also be useful in a S-system to show that the implementation conforms to the problem specification. Nevertheless, it is possible to prove the correctness of a program using formal methods, like Hoare Logic. The B-Method allows to correctly implement a formal specification.