Jim,
In advance, i would like to apologize for the length of the following, as i lack to time to reduce to a smaller explanation.
Below i will attempt to answer some of the items you raised, but while you read it you have to remember that the same term may have different meanings in different contexts. Or said another way, there are various degrees to which we undertake to construction of software. Who we are, which industry we work for and our understanding of activities shapes the way we look at "programming" hence what we mean by "testing".
To start we need to define a few ideas:
1) Why do we test?
2) What do we test?
3) When do we test?
4) What is testing?
*** be prepared for some major oversimplication to set the stage for the overall view ***
1) We test to:
a) provide some level of confidence that tested item meets some pre-determined specification,
b) to reduce costs of reworking errors, and
c) provide feedback that we understand the task to be completed.
2) We test requirements, specifications, designs, interfaces, code, builds, sub-systems and systems (basically everything).
3) We test as soon as possible.
4) Testing is the formal to semi-formal process of comparing the pre-determined, expected results to the actual results at any point in the software development process.
? OK so far ?
Given the above, i now need to provide an overview of a typical project. This project will be defined using a more heavy-weight (read: formal) approach. This assumption for this project is that we a mid to large company, so that we have some user departments and we have a software development organization. (we can easily extend this to vendor supplied/developed, but we do need a separation of departments for this to make sense). This example project will be developed using the waterfall approach (because it is so much easier to discuss), but the concepts can easily be transformed into any approach.
As a very high overview, what we are going to do is define a system, going from the most general to the most specific, then build the system going from the most specific to the most general. A great analogy is that we are going to dig a hole for ourselves one level at a time and then fill it up one level at time, testing to insure we filled the level correctly.
Phase 0 - Actualization that a need exists.
The company realizes that we have a process that currently costs too much. Employees of the company discuss the issue and want to replace process A with a process B that saves 50%.
We have a Business Requirement and a test specification: ie when all is said and done, process B will save 50%
Phase 1 - Each user department defines requirements.
Given the Business Requirement, the effected user departments generates a list of "features" necessary for the system, such as:
User Department - D1 has features { f1, f2, f4, f6}
User Department - D2 has features { f1, f3, f5, f6}
User Department - D3 has features { f4, f6, f7 }
For each of these features (or requirements) each User Department generates a list of test specifications that will determine if the system meets the needs. (here is where we have the user requirements (or better still functional requirements)).
Phase 2 - The software development organization defines what features the system to be developed will have, such as:
System will have features - { f1, f2, f3, f4, f6, f7 } and will start detailing what each feature will do, in addition to creating test cases for each feature. (System Requirements).
Phase 3 - The system definition is formalized to an "engineering specification" of precisely what the system will (not how) do.
System will have critical factors { cf1......cfn } where each of these critical factors map to one or more system features and each system feature is fully defined by one or more critical factors. For each critical factor, there is a test specification (System Specifications).
Phase 4 - Generation of the High-Level Design
The high-level design defines the module public interface, the system architecture and defines the builds. At this point all modules, methods, parameters should be defined. for each build, test specifications are generated (Build and Integration Specifications).
Phase 5 - Generation of the Detailed Design
Each method is defined down to the level of how the method is to perform it's task and a set of test cases to run against the method. (Code Specification).
Phase 6 - Code and unit test.
The code specification (from phase 5) is given to a programmer. That programmer writes the code and executes the code specification. <First real test in code>
Phase 7 - Build and integration test.
The code from phase 6 is gathered together into builds and is tested against the build and integration specifications <from Phase 4>.
Phase 8 - System Test.
The code from Phase 7 is subjected to testing from the system specifications <from Phase 3>. This test should be performed under the guidance of another project manager and is "usually" called QC (quality control) testing.
Phase 9 - Formal System Test or Quality Assurance Test
The code from Phase 8 is tested against the Systems Requirements <from Phase 2>. This should be managed by someone outside the software development organization.
Phase 10 - The User Departments test the system against the User Specifications <Phase 1> in what some call the acceptance test.
Phase 11 - After some time in production, the system is reviewed against the Business Specifications <Phase 0>.
*** Well that's how it's supposed to work in theory ***
