Test-driven development
A long-term goal of MIT Kerberos is to move toward test-driven development. It is not necessary to implement exactly the Extreme Programming or Agile ideals of test-driven development. One aspect of test-driven development as usually defined is that programmers write tests prior to writing actual implementation code. While this may be a worthy ideal, achieving it is more likely if done in stages.
A general observation is that instituting practices for "some testing" are better than having "no testing". This is not to imply that "more testing" is always better than "less testing", but for most reasonable scenarios, "more testing" is usually better. Automated tests are far more likely to be run than non-automated tests. Manual tests tend not to be run except in special circumstances, such as pre-arranged interoperability events.
Several possible levels of testing:
- No testing
- Written outline of manual testing procedures
- Written detailed descriptions of manual testing procedures
- Testing scripts not yet fully integrated into an automated testing framework
- Automated test cases fully integrated into an automated testing framework
- Fully implement a "test-first" methodology: no new features without an automated test being written first
An outside contributor may have difficulty producing an automated test case fully integrated into an automatic testing framework. The existing automated tests are not very well integrated and many are extremely ad-hoc in nature. A long-term goal will be to regularize the automated tests into a single integrated framework. This has advantages such as the capability of reviewing test results in a form optimized for differential measurements of success and regression.
Timeline
For the krb5-1.8 release, MITKC staff must provide testing scripts for every new feature they implement. Contributors must provide at least a written outline of manual testing procedures for their contributions, and should provide a written detailed description of the manual testing procedures. Contributors should strongly consider writing higher levels of tests.
During the 1.8 and 1.9 development timeframes, we will work to regularize our automated testing framework and decrease the barriers to entry for outside contributors. This will probably also involve substantial amounts of documentation work.
