Software Test AutomationView this Special Issue
Research Article | Open Access
Software Test Automation in Practice: Empirical Observations
The objective of this industry study is to shed light on the current situation and improvement needs in software test automation. To this end, 55 industry specialists from 31 organizational units were interviewed. In parallel with the survey, a qualitative study was conducted in 12 selected software development organizations. The results indicated that the software testing processes usually follow systematic methods to a large degree, and have only little immediate or critical requirements for resources. Based on the results, the testing processes have approximately three fourths of the resources they need, and have access to a limited, but usually sufficient, group of testing tools. As for the test automation, the situation is not as straightforward: based on our study, the applicability of test automation is still limited and its adaptation to testing contains practical difficulties in usability. In this study, we analyze and discuss these limitations and difficulties.
- E. Kit, Software Testing in the Real World: Improving the Process, Addison-Wesley, Reading, Mass, USA, 1995.
- G. Tassey, “The economic impacts of inadequate infrastructure for software testing,” RTI Project 7007.011, U.S. National Institute of Standards and Technology, Gaithersburg, Md, USA, 2002.
- R. Ramler and K. Wolfmaier, “Observations and lessons learned from automated testing,” in Proceedings of the International Workshop on Automation of Software Testing (AST '06), pp. 85–91, Shanghai, China, May 2006.
- K. Karhu, T. Repo, O. Taipale, and K. Smolander, “Empirical observations on software testing automation,” in Proceedings of the 2nd International Conference on Software Testing, Verification, and Validation (ICST '09), pp. 201–209, Denver, Colo, USA, April 2009.
- O. Taipale and K. Smolander, “Improving software testing by observing causes, effects, and associations from practice,” in Proceedings of the International Symposium on Empirical Software Engineering (ISESE '06), Rio de Janeiro, Brazil, September 2006.
- B. Shea, “Sofware testing gets new respect,” InformationWeek, July 2000.
- E. Dustin, J. Rashka, and J. Paul, Automated Software Testing: Introduction, Management, and Performance, Addison-Wesley, Boston, Mass, USA, 1999.
- S. Berner, R. Weber, and R. K. Keller, “Observations and lessons learned from automated testing,” in Proceedings of the 27th International Conference on Software Engineering (ICSE '05), pp. 571–579, St. Louis, Mo, USA, May 2005.
- J. A. Whittaker, “What is software testing? And why is it so hard?” IEEE Software, vol. 17, no. 1, pp. 70–79, 2000.
- L. J. Osterweil, “Software processes are software too, revisited: an invited talk on the most influential paper of ICSE 9,” in Proceedings of the 19th IEEE International Conference on Software Engineering, pp. 540–548, Boston, Mass, USA, May 1997.
- ISO/IEC and ISO/IEC 29119-2, “Software Testing Standard—Activity Descriptions for Test Process Diagram,” 2008.
- O. Taipale, K. Smolander, and H. Kälviäinen, “Cost reduction and quality improvement in software testing,” in Proceedings of the 14th International Software Quality Management Conference (SQM '06), Southampton, UK, April 2006.
- O. Taipale, K. Smolander, and H. Kälviäinen, “Factors affecting software testing time schedule,” in Proceedings of the Australian Software Engineering Conference (ASWEC '06), pp. 283–291, Sydney, Australia, April 2006.
- O. Taipale, K. Smolander, and H. Kälviäinen, “A survey on software testing,” in Proceedings of the 6th International SPICE Conference on Software Process Improvement and Capability dEtermination (SPICE '06), Luxembourg, May 2006.
- N. C. Dalkey, The Delphi Method: An Experimental Study of Group Opinion, RAND, Santa Monica, Calif, USA, 1969.
- S. P. Ng, T. Murnane, K. Reed, D. Grant, and T. Y. Chen, “A preliminary survey on software testing practices in Australia,” in Proceedings of the Australian Software Engineering Conference (ASWEC '04), pp. 116–125, Melbourne, Australia, April 2004.
- R. Torkar and S. Mankefors, “A survey on testing and reuse,” in Proceedings of IEEE International Conference on Software—Science, Technology and Engineering (SwSTE '03), Herzlia, Israel, November 2003.
- C. Ferreira and J. Cohen, “Agile systems development and stakeholder satisfaction: a South African empirical study,” in Proceedings of the Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists (SAICSIT '08), pp. 48–55, Wilderness, South Africa, October 2008.
- J. Li, F. O. Bjørnson, R. Conradi, and V. B. Kampenes, “An empirical study of variations in COTS-based software development processes in the Norwegian IT industry,” Empirical Software Engineering, vol. 11, no. 3, pp. 433–461, 2006.
- W. Chen, J. Li, J. Ma, R. Conradi, J. Ji, and C. Liu, “An empirical study on software development with open source components in the Chinese software industry,” Software Process Improvement and Practice, vol. 13, no. 1, pp. 89–100, 2008.
- R. Dossani and N. Denny, “The Internet's role in offshored services: a case study of India,” ACM Transactions on Internet Technology, vol. 7, no. 3, 2007.
- K. Y. Wong, “An exploratory study on knowledge management adoption in the Malaysian industry,” International Journal of Business Information Systems, vol. 3, no. 3, pp. 272–283, 2008.
- J. Bach, “Test automation snake oil,” in Proceedings of the 14th International Conference and Exposition on Testing Computer Software (TCS '99), Washington, DC, USA, June 1999.
- M. Fewster, Common Mistakes in Test Automation, Grove Consultants, 2001.
- A. Hartman, M. Katara, and A. Paradkar, “Domain specific approaches to software test automation,” in Proceedings of the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE '07), pp. 621–622, Dubrovnik, Croatia, September 2007.
- C. Persson and N. Yilmaztürk, “Establishment of automated regression testing at ABB: industrial experience report on ‘avoiding the pitfalls’,” in Proceedings of the 19th International Conference on Automated Software Engineering (ASE '04), pp. 112–121, Linz, Austria, September 2004.
- M. Auguston, J. B. Michael, and M.-T. Shing, “Test automation and safety assessment in rapid systems prototyping,” in Proceedings of the 16th IEEE International Workshop on Rapid System Prototyping (RSP '05), pp. 188–194, Montreal, Canada, June 2005.
- A. Cavarra, J. Davies, T. Jeron, L. Mournier, A. Hartman, and S. Olvovsky, “Using UML for automatic test generation,” in Proceedings of the International Symposium on Software Testing and Analysis (ISSTA '02), Roma, Italy, July 2002.
- M. Vieira, J. Leduc, R. Subramanyan, and J. Kazmeier, “Automation of GUI testing using a model-driven approach,” in Proceedings of the International Workshop on Automation of Software Testing, pp. 9–14, Shanghai, China, May 2006.
- Z. Xiaochun, Z. Bo, L. Juefeng, and G. Qiu, “A test automation solution on gui functional test,” in Proceedings of the 6th IEEE International Conference on Industrial Informatics (INDIN '08), pp. 1413–1418, Daejeon, Korea, July 2008.
- D. Kreuer, “Applying test automation to type acceptance testing of telecom networks: a case study with customer participation,” in Proceedings of the 14th IEEE International Conference on Automated Software Engineering, pp. 216–223, Cocoa Beach, Fla, USA, October 1999.
- W. D. Yu and G. Patil, “A workflow-based test automation framework for web based systems,” in Proceedings of the 12th IEEE Symposium on Computers and Communications (ISCC '07), pp. 333–339, Aveiro, Portugal, July 2007.
- A. Bertolino, “Software testing research: achievements, challenges, dreams,” in Proceedings of the Future of Software Engineering (FoSE '07), pp. 85–103, Minneapolis, Minn, USA, May 2007.
- M. Blackburn, R. Busser, and A. Nauman, “Why model-based test automation is different and what you should know to get started,” in Proceedings of the International Conference on Practical Software Quality, Braunschweig, Germany, September 2004.
- P. Santos-Neto, R. Resende, and C. Pádua, “Requirements for information systems model-based testing,” in Proceedings of the ACM Symposium on Applied Computing, pp. 1409–1415, Seoul, Korea, March 2007.
- ISO/IEC and ISO/IEC 15504-1, “Information Technology—Process Assessment—Part 1: Concepts and Vocabulary,” 2002.
- K. M. Eisenhardt, “Building theories from case study research,” The Academy of Management Review, vol. 14, no. 4, pp. 532–550, 1989.
- EU and European Commission, “The new SME definition: user guide and model declaration,” 2003.
- G. Paré and J. J. Elam, “Using case study research to build theories of IT implementation,” in Proceedings of the IFIP TC8 WG 8.2 International Conference on Information Systems and Qualitative Research, pp. 542–568, Chapman & Hall, Philadelphia, Pa, USA, May-June 1997.
- A. Strauss and J. Corbin, Basics of Qualitative Research: Grounded Theory Procedures and Techniques, SAGE, Newbury Park, Calif, USA, 1990.
- ATLAS.ti, The Knowledge Workbench, Scientific Software Development, 2005.
- M. B. Miles and A. M. Huberman, Qualitative Data Analysis, SAGE, Thousand Oaks, Calif, USA, 1994.
- C. B. Seaman, “Qualitative methods in empirical studies of software engineering,” IEEE Transactions on Software Engineering, vol. 25, no. 4, pp. 557–572, 1999.
- C. Robson, Real World Research, Blackwell, Oxford, UK, 2nd edition, 2002.
- N. K. Denzin, The Research Act: A Theoretical Introduction to Sociological Methods, McGraw-Hill, New York, NY, USA, 1978.
- A. Fink and J. Kosecoff, How to Conduct Surveys: A Step-by-Step Guide, SAGE, Beverly Hills, Calif, USA, 1985.
- B. A. Kitchenham, S. L. Pfleeger, L. M. Pickard et al., “Preliminary guidelines for empirical research in software engineering,” IEEE Transactions on Software Engineering, vol. 28, no. 8, pp. 721–734, 2002.
- T. Dybå, “An instrument for measuring the key factors of success in software process improvement,” Empirical Software Engineering, vol. 5, no. 4, pp. 357–390, 2000.
- ISO/IEC and ISO/IEC 25010-2, “Software Engineering—Software product Quality Requirements and Evaluation (SQuaRE) Quality Model,” 2008.
- Y. Baruch, “Response rate in academic studies—a comparative analysis,” Human Relations, vol. 52, no. 4, pp. 421–438, 1999.
- T. Koomen and M. Pol, Test Process Improvement: A Practical Step-by-Step Guide to Structured Testing, Addison-Wesley, Reading, Mass, USA, 1999.
- P. Kruchten, The Rational Unified Process: An Introduction, Addison-Wesley, Reading, Mass, USA, 2nd edition, 1998.
- K. Schwaber and M. Beedle, Agile Software Development with Scrum, Prentice-Hall, Upper Saddle River, NJ, USA, 2001.
- K. Beck, Extreme Programming Explained: Embrace Change, Addison-Wesley, Reading, Mass, USA, 2000.
- B. Glaser and A. L. Strauss, The Discovery of Grounded Theory: Strategies for Qualitative Research, Aldine, Chicago, Ill, USA, 1967.
- C. Kaner, “Improving the maintainability of automated test suites,” Software QA, vol. 4, no. 4, 1997.
- D. J. Mosley and B. A. Posey, Just Enough Software Test Automation, Prentice-Hall, Upper Saddle River, NJ, USA, 2002.
- D. Foray, Economics of Knowledge, MIT Press, Cambridge, Mass, USA, 2004.
Copyright © 2010 Jussi Kasurinen et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.