Advances in Software Engineering

Advances in Software Engineering / 2010 / Article
Special Issue

Software Test Automation

View this Special Issue

Research Article | Open Access

Volume 2010 |Article ID 620836 | 18 pages | https://doi.org/10.1155/2010/620836

Software Test Automation in Practice: Empirical Observations

Academic Editor: Phillip Laplante
Received10 Jun 2009
Revised28 Aug 2009
Accepted05 Nov 2009
Published04 Feb 2010

Abstract

The objective of this industry study is to shed light on the current situation and improvement needs in software test automation. To this end, 55 industry specialists from 31 organizational units were interviewed. In parallel with the survey, a qualitative study was conducted in 12 selected software development organizations. The results indicated that the software testing processes usually follow systematic methods to a large degree, and have only little immediate or critical requirements for resources. Based on the results, the testing processes have approximately three fourths of the resources they need, and have access to a limited, but usually sufficient, group of testing tools. As for the test automation, the situation is not as straightforward: based on our study, the applicability of test automation is still limited and its adaptation to testing contains practical difficulties in usability. In this study, we analyze and discuss these limitations and difficulties.

References

  1. E. Kit, Software Testing in the Real World: Improving the Process, Addison-Wesley, Reading, Mass, USA, 1995.
  2. G. Tassey, “The economic impacts of inadequate infrastructure for software testing,” RTI Project 7007.011, U.S. National Institute of Standards and Technology, Gaithersburg, Md, USA, 2002. View at: Google Scholar
  3. R. Ramler and K. Wolfmaier, “Observations and lessons learned from automated testing,” in Proceedings of the International Workshop on Automation of Software Testing (AST '06), pp. 85–91, Shanghai, China, May 2006. View at: Google Scholar
  4. K. Karhu, T. Repo, O. Taipale, and K. Smolander, “Empirical observations on software testing automation,” in Proceedings of the 2nd International Conference on Software Testing, Verification, and Validation (ICST '09), pp. 201–209, Denver, Colo, USA, April 2009. View at: Publisher Site | Google Scholar
  5. O. Taipale and K. Smolander, “Improving software testing by observing causes, effects, and associations from practice,” in Proceedings of the International Symposium on Empirical Software Engineering (ISESE '06), Rio de Janeiro, Brazil, September 2006. View at: Google Scholar
  6. B. Shea, “Sofware testing gets new respect,” InformationWeek, July 2000. View at: Google Scholar
  7. E. Dustin, J. Rashka, and J. Paul, Automated Software Testing: Introduction, Management, and Performance, Addison-Wesley, Boston, Mass, USA, 1999.
  8. S. Berner, R. Weber, and R. K. Keller, “Observations and lessons learned from automated testing,” in Proceedings of the 27th International Conference on Software Engineering (ICSE '05), pp. 571–579, St. Louis, Mo, USA, May 2005. View at: Google Scholar
  9. J. A. Whittaker, “What is software testing? And why is it so hard?” IEEE Software, vol. 17, no. 1, pp. 70–79, 2000. View at: Publisher Site | Google Scholar
  10. L. J. Osterweil, “Software processes are software too, revisited: an invited talk on the most influential paper of ICSE 9,” in Proceedings of the 19th IEEE International Conference on Software Engineering, pp. 540–548, Boston, Mass, USA, May 1997. View at: Google Scholar
  11. ISO/IEC and ISO/IEC 29119-2, “Software Testing Standard—Activity Descriptions for Test Process Diagram,” 2008. View at: Google Scholar
  12. O. Taipale, K. Smolander, and H. Kälviäinen, “Cost reduction and quality improvement in software testing,” in Proceedings of the 14th International Software Quality Management Conference (SQM '06), Southampton, UK, April 2006. View at: Google Scholar
  13. O. Taipale, K. Smolander, and H. Kälviäinen, “Factors affecting software testing time schedule,” in Proceedings of the Australian Software Engineering Conference (ASWEC '06), pp. 283–291, Sydney, Australia, April 2006. View at: Publisher Site | Google Scholar
  14. O. Taipale, K. Smolander, and H. Kälviäinen, “A survey on software testing,” in Proceedings of the 6th International SPICE Conference on Software Process Improvement and Capability dEtermination (SPICE '06), Luxembourg, May 2006. View at: Google Scholar
  15. N. C. Dalkey, The Delphi Method: An Experimental Study of Group Opinion, RAND, Santa Monica, Calif, USA, 1969.
  16. S. P. Ng, T. Murnane, K. Reed, D. Grant, and T. Y. Chen, “A preliminary survey on software testing practices in Australia,” in Proceedings of the Australian Software Engineering Conference (ASWEC '04), pp. 116–125, Melbourne, Australia, April 2004. View at: Google Scholar
  17. R. Torkar and S. Mankefors, “A survey on testing and reuse,” in Proceedings of IEEE International Conference on Software—Science, Technology and Engineering (SwSTE '03), Herzlia, Israel, November 2003. View at: Google Scholar
  18. C. Ferreira and J. Cohen, “Agile systems development and stakeholder satisfaction: a South African empirical study,” in Proceedings of the Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists (SAICSIT '08), pp. 48–55, Wilderness, South Africa, October 2008. View at: Publisher Site | Google Scholar
  19. J. Li, F. O. Bjørnson, R. Conradi, and V. B. Kampenes, “An empirical study of variations in COTS-based software development processes in the Norwegian IT industry,” Empirical Software Engineering, vol. 11, no. 3, pp. 433–461, 2006. View at: Publisher Site | Google Scholar
  20. W. Chen, J. Li, J. Ma, R. Conradi, J. Ji, and C. Liu, “An empirical study on software development with open source components in the Chinese software industry,” Software Process Improvement and Practice, vol. 13, no. 1, pp. 89–100, 2008. View at: Publisher Site | Google Scholar
  21. R. Dossani and N. Denny, “The Internet's role in offshored services: a case study of India,” ACM Transactions on Internet Technology, vol. 7, no. 3, 2007. View at: Publisher Site | Google Scholar
  22. K. Y. Wong, “An exploratory study on knowledge management adoption in the Malaysian industry,” International Journal of Business Information Systems, vol. 3, no. 3, pp. 272–283, 2008. View at: Publisher Site | Google Scholar
  23. J. Bach, “Test automation snake oil,” in Proceedings of the 14th International Conference and Exposition on Testing Computer Software (TCS '99), Washington, DC, USA, June 1999. View at: Google Scholar
  24. M. Fewster, Common Mistakes in Test Automation, Grove Consultants, 2001.
  25. A. Hartman, M. Katara, and A. Paradkar, “Domain specific approaches to software test automation,” in Proceedings of the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE '07), pp. 621–622, Dubrovnik, Croatia, September 2007. View at: Publisher Site | Google Scholar
  26. C. Persson and N. Yilmaztürk, “Establishment of automated regression testing at ABB: industrial experience report on ‘avoiding the pitfalls’,” in Proceedings of the 19th International Conference on Automated Software Engineering (ASE '04), pp. 112–121, Linz, Austria, September 2004. View at: Publisher Site | Google Scholar
  27. M. Auguston, J. B. Michael, and M.-T. Shing, “Test automation and safety assessment in rapid systems prototyping,” in Proceedings of the 16th IEEE International Workshop on Rapid System Prototyping (RSP '05), pp. 188–194, Montreal, Canada, June 2005. View at: Publisher Site | Google Scholar
  28. A. Cavarra, J. Davies, T. Jeron, L. Mournier, A. Hartman, and S. Olvovsky, “Using UML for automatic test generation,” in Proceedings of the International Symposium on Software Testing and Analysis (ISSTA '02), Roma, Italy, July 2002. View at: Google Scholar
  29. M. Vieira, J. Leduc, R. Subramanyan, and J. Kazmeier, “Automation of GUI testing using a model-driven approach,” in Proceedings of the International Workshop on Automation of Software Testing, pp. 9–14, Shanghai, China, May 2006. View at: Publisher Site | Google Scholar
  30. Z. Xiaochun, Z. Bo, L. Juefeng, and G. Qiu, “A test automation solution on gui functional test,” in Proceedings of the 6th IEEE International Conference on Industrial Informatics (INDIN '08), pp. 1413–1418, Daejeon, Korea, July 2008. View at: Publisher Site | Google Scholar
  31. D. Kreuer, “Applying test automation to type acceptance testing of telecom networks: a case study with customer participation,” in Proceedings of the 14th IEEE International Conference on Automated Software Engineering, pp. 216–223, Cocoa Beach, Fla, USA, October 1999. View at: Publisher Site | Google Scholar
  32. W. D. Yu and G. Patil, “A workflow-based test automation framework for web based systems,” in Proceedings of the 12th IEEE Symposium on Computers and Communications (ISCC '07), pp. 333–339, Aveiro, Portugal, July 2007. View at: Publisher Site | Google Scholar
  33. A. Bertolino, “Software testing research: achievements, challenges, dreams,” in Proceedings of the Future of Software Engineering (FoSE '07), pp. 85–103, Minneapolis, Minn, USA, May 2007. View at: Publisher Site | Google Scholar
  34. M. Blackburn, R. Busser, and A. Nauman, “Why model-based test automation is different and what you should know to get started,” in Proceedings of the International Conference on Practical Software Quality, Braunschweig, Germany, September 2004. View at: Google Scholar
  35. P. Santos-Neto, R. Resende, and C. Pádua, “Requirements for information systems model-based testing,” in Proceedings of the ACM Symposium on Applied Computing, pp. 1409–1415, Seoul, Korea, March 2007. View at: Publisher Site | Google Scholar
  36. ISO/IEC and ISO/IEC 15504-1, “Information Technology—Process Assessment—Part 1: Concepts and Vocabulary,” 2002. View at: Google Scholar
  37. K. M. Eisenhardt, “Building theories from case study research,” The Academy of Management Review, vol. 14, no. 4, pp. 532–550, 1989. View at: Publisher Site | Google Scholar
  38. EU and European Commission, “The new SME definition: user guide and model declaration,” 2003. View at: Google Scholar
  39. G. Paré and J. J. Elam, “Using case study research to build theories of IT implementation,” in Proceedings of the IFIP TC8 WG 8.2 International Conference on Information Systems and Qualitative Research, pp. 542–568, Chapman & Hall, Philadelphia, Pa, USA, May-June 1997. View at: Google Scholar
  40. A. Strauss and J. Corbin, Basics of Qualitative Research: Grounded Theory Procedures and Techniques, SAGE, Newbury Park, Calif, USA, 1990.
  41. ATLAS.ti, The Knowledge Workbench, Scientific Software Development, 2005.
  42. M. B. Miles and A. M. Huberman, Qualitative Data Analysis, SAGE, Thousand Oaks, Calif, USA, 1994.
  43. C. B. Seaman, “Qualitative methods in empirical studies of software engineering,” IEEE Transactions on Software Engineering, vol. 25, no. 4, pp. 557–572, 1999. View at: Google Scholar
  44. C. Robson, Real World Research, Blackwell, Oxford, UK, 2nd edition, 2002.
  45. N. K. Denzin, The Research Act: A Theoretical Introduction to Sociological Methods, McGraw-Hill, New York, NY, USA, 1978.
  46. A. Fink and J. Kosecoff, How to Conduct Surveys: A Step-by-Step Guide, SAGE, Beverly Hills, Calif, USA, 1985.
  47. B. A. Kitchenham, S. L. Pfleeger, L. M. Pickard et al., “Preliminary guidelines for empirical research in software engineering,” IEEE Transactions on Software Engineering, vol. 28, no. 8, pp. 721–734, 2002. View at: Publisher Site | Google Scholar
  48. T. Dybå, “An instrument for measuring the key factors of success in software process improvement,” Empirical Software Engineering, vol. 5, no. 4, pp. 357–390, 2000. View at: Publisher Site | Google Scholar
  49. ISO/IEC and ISO/IEC 25010-2, “Software Engineering—Software product Quality Requirements and Evaluation (SQuaRE) Quality Model,” 2008. View at: Google Scholar
  50. Y. Baruch, “Response rate in academic studies—a comparative analysis,” Human Relations, vol. 52, no. 4, pp. 421–438, 1999. View at: Google Scholar
  51. T. Koomen and M. Pol, Test Process Improvement: A Practical Step-by-Step Guide to Structured Testing, Addison-Wesley, Reading, Mass, USA, 1999.
  52. P. Kruchten, The Rational Unified Process: An Introduction, Addison-Wesley, Reading, Mass, USA, 2nd edition, 1998.
  53. K. Schwaber and M. Beedle, Agile Software Development with Scrum, Prentice-Hall, Upper Saddle River, NJ, USA, 2001.
  54. K. Beck, Extreme Programming Explained: Embrace Change, Addison-Wesley, Reading, Mass, USA, 2000.
  55. B. Glaser and A. L. Strauss, The Discovery of Grounded Theory: Strategies for Qualitative Research, Aldine, Chicago, Ill, USA, 1967.
  56. C. Kaner, “Improving the maintainability of automated test suites,” Software QA, vol. 4, no. 4, 1997. View at: Google Scholar
  57. D. J. Mosley and B. A. Posey, Just Enough Software Test Automation, Prentice-Hall, Upper Saddle River, NJ, USA, 2002.
  58. D. Foray, Economics of Knowledge, MIT Press, Cambridge, Mass, USA, 2004.

Copyright © 2010 Jussi Kasurinen et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

1590 Views | 2739 Downloads | 38 Citations
 PDF  Download Citation  Citation
 Download other formatsMore
 Order printed copiesOrder