On my way to the latest TMF (Test Management Forum) meeting, I was running late for my train. So, I attempted to use the automatic ticket machine at my station—only to give up when it decided to time out rather than process my Visa card, and without telling me what I'd done wrong. Luckily, I just had time to buy my ticket from the First Great Western Wetware Interface....

When you're in a real hurry, a guarantee of getting it right first time can be better than several attempts at something potentially much faster.

Which gets me onto software development; getting it right first time may just beat a lot of agile prototyping attempts (if you take a "whole lifecycle" view of the metrics). And testing is part of this—if you see testing as part of the whole lifecycle of a system, not just the final barrier to going live.

To my mind, testing the business case for an automated system; questioning the competence and experience of the domain expert the team has access to (is s/he the person the business least wants to be operate the business process, in which case their information may well be unreliable); and testing any requirements specification (for completeness and consistency) is where testing starts. That is, long before you have any code. Imagine the testing (and other) resources you might save if you discovered that the system you're starting in on will never make any money! Unless you work for the sort of organisation which says "don't question TPTB—build it anyway".

It's worth looking at ways of making testing more effective, because I think we're we're approaching at least one testing crisis, probably more (see my article here, which mentions the shortage of skilled testers; I'm now equally worried about the complexity of testing "dark path" compensation in SOA environments). James Whittaker (Principle Architect, Visual Studio Team Test at Microsoft) told the TMF about the future of Testing as Microsoft (presumably) sees it—for this month at least—in an extremely entertaining lecture (although possibly targeted on an audience a little wider and less informed than the testing managers at TMF). Software is getting more complex (at Microsoft certainly) and more critical to business survival, and we are not getting all that better at making it work reliably; as James put it, we now have the ability to screw up more things than ever before.

You can hold the jokes about Microsoft being in a good position to know (James himself had a good selection of Microsoft horror stories to tell)—it really is right about the criticality of software today and the importance of testing it. On the other hand lumping all software together is a bit unfair; we can produce reliable software if we want to (and can do without the "go faster stripes" and bloatware add-ins which often add unnecessary complexity and contribute to the problem). Praxis, for instance, has a process which the US NSA found to produce "zero defect" software (if a defect is defined as a departure from the spec), without any possibility of many kinds of bugs, but you have to program in a subset of Ada (and validate the requirements spec, of course). However, if it helps produce reliable software, perhaps dumping Web 2.0 scripting is worth it (see here, for a recognition of the Praxis approach).

Anyway, James suggested that we needed radical new approaches to testing next generation software: using Virtualisation; better Information feedback; and better Visualisation. I'm not sure that second two (rich information feedback and visualisation of test processes) aren't already available in some environments, at least in crude forms, but James is right to suggest that they need to become ubiquitous, and lots slicker. James envisages "heads up" displays for testers, displaying information for testers in real-time, rather like the interface you build for "cheating" in video games—why shouldn't testers be allowed to "cheat" when finding bugs so that (for example) they can see the faulty code hiding behind a GUI failure. What James calls "X-ray Vision for Testers". Testers need to be able to see, and use, all the development assets, from Visio business models and PowerPoint presentations to the bits and bytes behind the high level code. If necessary.

And the people managing the testers (as well as the testers themselves) apparently need graphical "heat maps" (for example), which let them visualise test coverage and code complexity in the same view—so that sub-optimal use of resources becomes obvious. However, I'm sure that this won't be new to many of you, although you might find James' envisaged implementations exciting, if they ever reach product..

The Virtualisation thing is genuinely radical, I think. As James sees it, we can now press a button and capture a "virtual machine" (VM) which encapsulates a system failure together with all its environment, in compact form. This VM can be analysed at leisure and stored for all time as a record of a particular failure scenario. More radically, this VM could be released into the "cloud" for "cloud testing"—for all and sundry to test their own data against, for financial reward (traditional beta tests are largely useless, in my opinion, because voluntary beta testers have other work to do and tend only to play with bits of the code which pretty much work—and don't often remember to tell people about what they find anyway. James sees a market in these "testing VMs" growing up.

It's an intriguing idea and addresses a serious issue: code is so complex these days that no organisation has the internal resources to completely test it. But the devil, of course, is in the detail. How do you generalise these VMs so that they are useful as code and data formats change? How do you manage (find and classify) these VMs? How do you produce risk metrics for the results of "cloud testing"—perhaps no-one tests the boring bits of code (although testing visualisation tools could help here)? What about security/privacy (and does "sanitising" the VM and its data damage its value as a test)? And, perhaps most important, isn't testing only after you have code to test leaving things a bit late—mistakes in Requirements are the real killer.

Still, I do think that we do need to radically rethink the process of testing. Possibly people might like to check out process improvement for testing with the TMMi Foundation; although I know some people don't care for Maturity Models. I do, although using them appropriately needs some maturity to start with (they shouldn't be thought of as "process improvement by numbers", and implementation should be mentored by experienced practitioners/appraisers). They are also often "reassuringly expensive"—but rework isn't cheap either—and it really isn't possible to get all the benefits of maturity in one process area (such as testing) unless any other process areas interacting with it are at a similar level of maturity too (which helps to make process improvement quite a big undertaking).

Roger Halton (Programme Manager at Amsphere, a sponsor of TMF) took us through the complexity of testing PCI-DSS rule compliance for credit card processing, in an informal workshop. Lots of potential issues were raised, even though the rules themselves seemed comparatively easy to test. One conclusion was that looking at business process around the environment in which the rules would be used was critical; another was that involvement with the development teams in the external organisations coding the software that would support credit card processing was vital; in other words, proper scoping is really important.

Finally Paul Gerrard of Gerrard Consulting (the other TMF sponsor this time) proposed Axioms for Testing as a framework for the testing process. Axioms are "self-evident truths" that help clarify thinking—Paul, for example, criticised the oft-held view that "testing finds defects in software"—no, he said, testing finds failures, from which people (not necessarily testers) infer defects in software, specifications etc. (this is a dig at the ‘experts' who preach exactitude but then use sloppy language in their books and the Certification Scheme Syllabuses). Anyway, Paul's Axioms are a "work in progress". Currently, they look something like this:

1. Stakeholder: Testing needs stakeholders.

2. Test Basis: Test needs a source of [domain] knowledge to help you select things to test.

3. Test Oracle: Test needs a source of [domain] knowledge to evaluate actual behaviour.

4. Fallibility: Your sources of knowledge are fallible and incomplete.

5. Scope Management: If you don't manage scope, you may never meet stakeholder expectations.

6. Coverage Testing: requires a coverage model or models.

7. Delivery: The usefulness of the intelligence produced by test determines the value of testing.

8. Environment: Test execution requires a known, controlled environment.

9. Events Testing: never goes as planned.

10. Prioritisation: The most important tests are those that uncover the best intelligence, fast.

11. Execution Sequencing: Run your most important tests first - you may not have time to run them later.

12. Design: Test design is based on models.

13. Repeat-Test: Repeated tests (and regression tests) are inevitable.

14. Good-Enough: Acceptance is always a compromise.

15. Never-Finished: Testing never finishes; it stops.

16. Value: The value of intelligence is independent of who produces it.

Readers can join in the discussion at Paul's blog—he wants community feedback on these testing axioms. If you are at all involved in testing or software quality management, the TMF site (and registration with the TMF) is well worthwhile.