[Feature-dev] Test quality metrics and recovery plan

[Mike Wong]

I see your points, and they are very insightful.

There are a lot of methods that (1) are not used, and (2) will go away after standardizing on the STL. I'd rather not spend time writing tests for these methods. This is what I mean by waiting for the API to finalize. I see your point that while we refactor, we are gaining valuable knowledge about each method and risk forgetting how they work (and how to test them) as time moves forward.

For methods that we absolutely know will remain, then we should write tests for them as we refactor. However, I'm a huge believer in keeping thing simple; do one thing at a time, do it well, and come back to the code and do the next thing. I've been following this philosophy throughout the project, and it's really paid off. For example, see how I removed the HashTable class. This would ordinarily be a two-week job if I did it all at once, but because I was smart about breaking the job down into very small tasks, I got it working in two days.

So this is what I recommend: if you see a method that you know is important, don't write the tests now, but write a test plan in the documentation. Something like:

/**
 * @test {your test plan description goes here; maybe include pseudo code, but not too much detail}
 **/

 
We can come back and use the test plans to write the tests later. This way we focus on refactoring and we'll have posted signs so our death-march will be more like a guided tour.

We've been using the Perl Test::Harness framework. I've literally used it for over a decade and it's grown in sophistication as the years go by and is still easy-to-use and rock-solid. Mark picked it up over a weekend and liked it. 

I think that looking at other test frameworks is an excellent idea and Mauricio and I will definitely look at CppTest; I'm really glad you brought it to our attention.

Too bad the mailing list doesn't have karma or I'd mod you up if I could. Thanks Gurgen!

- m.

__________________________________
Mike Wong, Staff Research Associate
Center for Computing for Life Sciences (CCLS)
San Francisco State University
1600 Holloway Avenue, Hensill Hall 301, SF, CA 94132
(415) 405-2119
mikewong at sfsu.edu




________________________________
From: Gurgen Tumanian <tumanian at gmail.com>
To: Mike Wong <mikewong at sfsu.edu>; feature-dev at simtk.org
Sent: Wednesday, June 24, 2009 11:59:09 PM
Subject: Re: [Feature-dev] Test quality metrics and recovery plan

Hi Mike. 

A small comment from my side.
>>As we continue to eliminate unused code and understand the interactions between >>classes, we need to finalize the API for each class and then write tests (i.e. unit ??tests) to test each method.

I think that it is better not to wait for the API to finalize, but to start writing unit tests now. I think it will be easier to do it now in the process of refactoring the code, then to do a test-writing-death march later. 

BTW, what do you use for unit tests? Is that still the mini executables that are in the unit-test dir? I propose to have a look at the existing unit-testing frameworks and choose the one that suits our tasks best. CppTest could be one solution.

Regards
Gurgen


On Wed, Jun 24, 2009 at 5:47 PM, Mike Wong <mikewong at sfsu.edu> wrote:

Hi all,

Just to confirm, there is an unresolved bug in our code base and Mauricio and I will address it.

I've written some software to perform our regression test against our submissions to the FEATURE Subversion server, and the results are good and bad. As we increase coverage in our tests there are more failures. This is good because we know that our tests are pushing the envelope of the utility of FEATURE and our testing process is improving. This is bad because somewhere along the line we destroyed previously existing functionality in FEATURE and didn't know about it (because the test for the functionality didn't exist).

Now, of course we can always roll back to r418 and restore all the old functionality and try to re-implement our changes. However, my plan is to move forward, find the error that caused the loss of functionality, patch that error, and apply the patch to the latest revision.

Good news is that r454 passes all tests, so we know that r454 has equivalent functionality to r418 (for all functions that we test).

Bottom line: As a team we need to make a commitment to software testing, and we need to share the QA duties. As we continue to eliminate unused code and understand the interactions between classes, we need to finalize the API for each class and then write tests (i.e. unit tests) to test each method.

Best regards,

- m.

 __________________________________
Mike Wong, Staff Research Associate
Center for Computing for Life Sciences (CCLS)
San Francisco State University
1600 Holloway Avenue, Hensill Hall 301, SF, CA 94132
(415) 405-2119
mikewong at sfsu.edu

_______________________________________________
Feature-dev mailing list
Feature-dev at simtk.org
https://simtk.org/mailman/listinfo/feature-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://simtk.org/pipermail/feature-dev/attachments/20090625/31030bc7/attachment.html