Personally, I find that most people don’t pay attention to the passing of test and don’t correlate test cases to test coverage very well so the result numbers are pretty ambiguous anyhow with traditional methods of testing.
Almost like saying 3 out of 4 dentist prefer this toothpaste over that. Where did they find these dentists? After the company bribed them with money?
What do I mean? You could have tons of test cases covering UI but not necessarily the business logic on how something is suppose to work. etc. etc. Of course, with good testers that won’t happen; they would try to get a broad range of tests. Unless you dig into the tests it’s hard to get a view of all the tests. One could say that it’s documented. How many people actually bother looking at the percent coverage though? Or question it? “There’s x many tests, of course there must be good coverage!” I’ve heard that too line so many times yet then there’s those escaped bugs… Escaped bugs will happen. I’m just saying that the assumption that you have 100 % test coverage based on x number of tests is just… wrong. Even the thought of the more tests you have the closer you are to 100%. Well, that might be true; x number of tests doesn’t tell you how close you are to that 100%.
And as I stated, before running the same tests over and over doesn’t prove as fruitful as running new tests.
I’m trying to come up with a way to display coverage using a flow diagram as way of saying this is what I’ve been looking at. I’ve also contemplated merging it with James Bach stuff on session based testing. He says to take notes. I’m expanding on the note taking to fast diagrams.
The thing with developing software is … is that it keeps changing and evolving. Those changes have to be reflected in the tests as well. Until the software gets to a place of “code freeze/string freeze”, it’s hard to lock down test cases. You’ll spend more cycles maintaining than testing. In which case, exploratory testing/session based testing has the advantage.
The problem is that it’s very faith based that the tester will do well and it only works w/ good testers. (or testers that are trained to exploratory test)… The comparison is like do you send a novice mushroom hunter out or do you send a professional mushroom hunter out and how much more mushroom do you think each will come back with in a shorter amount of time?
Management of course wants some documentation, and rather than making it, “faith based like a religion”, we want it more “fact based like science” in an easy to read, charts/diagrams. Hence the UI user-flow diagram concept I’m trying to do.