Software Test AutomationView this Special Issue
Software Test Automation in Practice: Empirical Observations
The objective of this industry study is to shed light on the current situation and improvement needs in software test automation. To this end, 55 industry specialists from 31 organizational units were interviewed. In parallel with the survey, a qualitative study was conducted in 12 selected software development organizations. The results indicated that the software testing processes usually follow systematic methods to a large degree, and have only little immediate or critical requirements for resources. Based on the results, the testing processes have approximately three fourths of the resources they need, and have access to a limited, but usually sufficient, group of testing tools. As for the test automation, the situation is not as straightforward: based on our study, the applicability of test automation is still limited and its adaptation to testing contains practical difficulties in usability. In this study, we analyze and discuss these limitations and difficulties.
E. Kit, Software Testing in the Real World: Improving the Process, Addison-Wesley, Reading, Mass, USA, 1995.
G. Tassey, “The economic impacts of inadequate infrastructure for software testing,” RTI Project 7007.011, U.S. National Institute of Standards and Technology, Gaithersburg, Md, USA, 2002.View at: Google Scholar
R. Ramler and K. Wolfmaier, “Observations and lessons learned from automated testing,” in Proceedings of the International Workshop on Automation of Software Testing (AST '06), pp. 85–91, Shanghai, China, May 2006.View at: Google Scholar
K. Karhu, T. Repo, O. Taipale, and K. Smolander, “Empirical observations on software testing automation,” in Proceedings of the 2nd International Conference on Software Testing, Verification, and Validation (ICST '09), pp. 201–209, Denver, Colo, USA, April 2009.View at: Publisher Site | Google Scholar
O. Taipale and K. Smolander, “Improving software testing by observing causes, effects, and associations from practice,” in Proceedings of the International Symposium on Empirical Software Engineering (ISESE '06), Rio de Janeiro, Brazil, September 2006.View at: Google Scholar
B. Shea, “Sofware testing gets new respect,” InformationWeek, July 2000.View at: Google Scholar
E. Dustin, J. Rashka, and J. Paul, Automated Software Testing: Introduction, Management, and Performance, Addison-Wesley, Boston, Mass, USA, 1999.
S. Berner, R. Weber, and R. K. Keller, “Observations and lessons learned from automated testing,” in Proceedings of the 27th International Conference on Software Engineering (ICSE '05), pp. 571–579, St. Louis, Mo, USA, May 2005.View at: Google Scholar
L. J. Osterweil, “Software processes are software too, revisited: an invited talk on the most influential paper of ICSE 9,” in Proceedings of the 19th IEEE International Conference on Software Engineering, pp. 540–548, Boston, Mass, USA, May 1997.View at: Google Scholar
ISO/IEC and ISO/IEC 29119-2, “Software Testing Standard—Activity Descriptions for Test Process Diagram,” 2008.View at: Google Scholar
O. Taipale, K. Smolander, and H. Kälviäinen, “Cost reduction and quality improvement in software testing,” in Proceedings of the 14th International Software Quality Management Conference (SQM '06), Southampton, UK, April 2006.View at: Google Scholar
O. Taipale, K. Smolander, and H. Kälviäinen, “A survey on software testing,” in Proceedings of the 6th International SPICE Conference on Software Process Improvement and Capability dEtermination (SPICE '06), Luxembourg, May 2006.View at: Google Scholar
N. C. Dalkey, The Delphi Method: An Experimental Study of Group Opinion, RAND, Santa Monica, Calif, USA, 1969.
S. P. Ng, T. Murnane, K. Reed, D. Grant, and T. Y. Chen, “A preliminary survey on software testing practices in Australia,” in Proceedings of the Australian Software Engineering Conference (ASWEC '04), pp. 116–125, Melbourne, Australia, April 2004.View at: Google Scholar
R. Torkar and S. Mankefors, “A survey on testing and reuse,” in Proceedings of IEEE International Conference on Software—Science, Technology and Engineering (SwSTE '03), Herzlia, Israel, November 2003.View at: Google Scholar
C. Ferreira and J. Cohen, “Agile systems development and stakeholder satisfaction: a South African empirical study,” in Proceedings of the Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists (SAICSIT '08), pp. 48–55, Wilderness, South Africa, October 2008.View at: Publisher Site | Google Scholar
J. Bach, “Test automation snake oil,” in Proceedings of the 14th International Conference and Exposition on Testing Computer Software (TCS '99), Washington, DC, USA, June 1999.View at: Google Scholar
M. Fewster, Common Mistakes in Test Automation, Grove Consultants, 2001.
A. Hartman, M. Katara, and A. Paradkar, “Domain specific approaches to software test automation,” in Proceedings of the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE '07), pp. 621–622, Dubrovnik, Croatia, September 2007.View at: Publisher Site | Google Scholar
C. Persson and N. Yilmaztürk, “Establishment of automated regression testing at ABB: industrial experience report on ‘avoiding the pitfalls’,” in Proceedings of the 19th International Conference on Automated Software Engineering (ASE '04), pp. 112–121, Linz, Austria, September 2004.View at: Publisher Site | Google Scholar
A. Cavarra, J. Davies, T. Jeron, L. Mournier, A. Hartman, and S. Olvovsky, “Using UML for automatic test generation,” in Proceedings of the International Symposium on Software Testing and Analysis (ISSTA '02), Roma, Italy, July 2002.View at: Google Scholar
D. Kreuer, “Applying test automation to type acceptance testing of telecom networks: a case study with customer participation,” in Proceedings of the 14th IEEE International Conference on Automated Software Engineering, pp. 216–223, Cocoa Beach, Fla, USA, October 1999.View at: Publisher Site | Google Scholar
M. Blackburn, R. Busser, and A. Nauman, “Why model-based test automation is different and what you should know to get started,” in Proceedings of the International Conference on Practical Software Quality, Braunschweig, Germany, September 2004.View at: Google Scholar
ISO/IEC and ISO/IEC 15504-1, “Information Technology—Process Assessment—Part 1: Concepts and Vocabulary,” 2002.View at: Google Scholar
EU and European Commission, “The new SME definition: user guide and model declaration,” 2003.View at: Google Scholar
G. Paré and J. J. Elam, “Using case study research to build theories of IT implementation,” in Proceedings of the IFIP TC8 WG 8.2 International Conference on Information Systems and Qualitative Research, pp. 542–568, Chapman & Hall, Philadelphia, Pa, USA, May-June 1997.View at: Google Scholar
A. Strauss and J. Corbin, Basics of Qualitative Research: Grounded Theory Procedures and Techniques, SAGE, Newbury Park, Calif, USA, 1990.
ATLAS.ti, The Knowledge Workbench, Scientific Software Development, 2005.
M. B. Miles and A. M. Huberman, Qualitative Data Analysis, SAGE, Thousand Oaks, Calif, USA, 1994.
C. B. Seaman, “Qualitative methods in empirical studies of software engineering,” IEEE Transactions on Software Engineering, vol. 25, no. 4, pp. 557–572, 1999.View at: Google Scholar
C. Robson, Real World Research, Blackwell, Oxford, UK, 2nd edition, 2002.
N. K. Denzin, The Research Act: A Theoretical Introduction to Sociological Methods, McGraw-Hill, New York, NY, USA, 1978.
A. Fink and J. Kosecoff, How to Conduct Surveys: A Step-by-Step Guide, SAGE, Beverly Hills, Calif, USA, 1985.
ISO/IEC and ISO/IEC 25010-2, “Software Engineering—Software product Quality Requirements and Evaluation (SQuaRE) Quality Model,” 2008.View at: Google Scholar
Y. Baruch, “Response rate in academic studies—a comparative analysis,” Human Relations, vol. 52, no. 4, pp. 421–438, 1999.View at: Google Scholar
T. Koomen and M. Pol, Test Process Improvement: A Practical Step-by-Step Guide to Structured Testing, Addison-Wesley, Reading, Mass, USA, 1999.
P. Kruchten, The Rational Unified Process: An Introduction, Addison-Wesley, Reading, Mass, USA, 2nd edition, 1998.
K. Schwaber and M. Beedle, Agile Software Development with Scrum, Prentice-Hall, Upper Saddle River, NJ, USA, 2001.
K. Beck, Extreme Programming Explained: Embrace Change, Addison-Wesley, Reading, Mass, USA, 2000.
B. Glaser and A. L. Strauss, The Discovery of Grounded Theory: Strategies for Qualitative Research, Aldine, Chicago, Ill, USA, 1967.
C. Kaner, “Improving the maintainability of automated test suites,” Software QA, vol. 4, no. 4, 1997.View at: Google Scholar
D. J. Mosley and B. A. Posey, Just Enough Software Test Automation, Prentice-Hall, Upper Saddle River, NJ, USA, 2002.
D. Foray, Economics of Knowledge, MIT Press, Cambridge, Mass, USA, 2004.