Abstract
This paper reviews existing measures used for evaluating the usability of information systems and those used for evaluating the level of the productivity of a company. We discuss the usefulness of the existing measures in the mobile business context, where both mobility and work-context pose specific demands for the mobile business services. The review showed that the existing measures rarely consider the great contextual variation caused by the mobility of the services and the demands this poses on usability; which, in turn, affects productivity. To build a measurement tool that better meets the requirements of mobile business services, we completed case studies on two mobile business services, one used in passenger transport and the other in construction sites. Based on the understanding gained from the case studies, we propose a list of dimensions and items addressing both usability and productivity aspects that work as the basis for a multidisciplinary measurement tool.
1. Introduction
Mobile business services have great potential for benefiting both the individual workers and the companies deploying them. These business-oriented services are used via mobile devices (e.g., mobile phone or PDA), for example, to access corporate information systems and the Internet or for collecting data in the field (e.g., [1, 2]). Both the companies developing and deploying these services would benefit from a tool that measures their success. This tool could be used for benchmarking and evaluating the new versions of the service, for evaluating the success of the service against competitors, and for evaluating the possible benefits the service could provide for the end users and companies deploying the service.
Despite the potential benefits, it remains a challenge to measure how successful the services actually are. One challenge in providing a tool for measuring success lies in the fact that success means different things for the different stakeholders: the users of the service might emphasise the easiness of use while improved productivity is often mentioned as one of the main benefits from the perspective of the company (e.g., [3, 4]). These views on success are not necessarily contradictory; often the easy-to-use service is also more successful financially. For example, if the service is easy to use, the need for training decreases and then users can focus on working instead of taking part in training courses or seeking online help [5]. The bottom line is that the success of a mobile business service use cannot be determined by utilizing one factor alone—both usability and business aspects need to be considered together.
Another challenge in the development of a tool for measuring success lies in the fact that such tool needs to take into account the context. From a technical view, mobile context includes, for example, the network infrastructure, the broader computational system, and the actual physical environment [6]. According to Vartiainen [7], connections (e.g., Internet, WLAN, broadband), devices, as well as applications and services construct virtual spaces that enable communication and collaboration. Moreover, Vartiainen [7] defines physical environments as settings and environments such as home, offices, cars, and hotels, where the mobile services are used. In addition to environmental context (including physical and social contexts), mobile context also includes personal context from the user’s point of view [8]. According to Lee etal. [8], emotion, time, and movement are three important factors related to personal context.
On the basis of examining work in a mobile context, recent studies concluded that there is the so-called “anytime, anywhere” paradox [9], that is, time frames of mobile work are not always negotiable, nor are the places where mobile work take place. On the other hand, Sørensen etal. [10] stated that in the mobile work context, supporting smart work processes is the key in keeping the dynamic work environment in control. This control, in turn, ensures safer and more economic work environment [10]. Moreover, we believe that the complexity of the mobile work context has an effect on measuring the success factors. Mobility is not only “a matter of location in time and space” [11] as the mobility can be spatial, temporal, and contextual [12].
In this paper, we will first present a review on the state of the art of the measurement tools for usability and productivity. To better understand whether the current tools are sufficient for the mobile business context, we will present findings from two case studies. One of the case studies was done on a mobile taxi dispatching system and another on a mobile safety-measurement service used at construction sites. In the case studies, we explored factors that have an effect on the success of mobile business services. In addition, we will highlight the shortcomings of the current way of measuring success—why some measures are not suitable or optimal in this context and why none of the existing measurement tools are sufficient alone. As a result, the paper presents a pool of items that will be used for developing a questionnaire for evaluating some key success factors of mobile business services from the individual user’s point of view.
2. Measuring Usability
According to ISO 9241 [13] standard, usability is an extent to which the intended users of a product achieve specified goals in an effective, efficient and satisfactory manner within a specified context of use. This section presents an overview on the existing tools and metrics for measuring usability and discusses their limitations in the context of mobile business services.
2.1. Measuring User’s Performance
Effectiveness—or the accuracy and completeness with which intended goals are achieved—can be measured by the percent or number of errors, the success-failure ratio, or the percentage of tasks completed. Efficiency is a measure of the amount of human, economic, and temporal resources that are expended in attaining a required level of product effectiveness. Measures of efficiency include time taken to complete tasks, time spent using help or documentation, and estimations of subjective workload (mental effort). Satisfaction refers to the comfort in using the system and its overall functionality. Useful measures of user satisfaction include the number of times the interface misleads the user or the user loses control of the system, the percent of favourable/unfavourable comments, and ratings on post-test questionnaires.
According to Nielsen [14], another way to define usability is through the concepts of learnability, memorability, efficiency, subjective satisfaction, and low rate of errors. For example, learnability can be measured as the beginners’ time to attain the average skills with the system and efficiency by considering how fast and accurately a practiced user can work with the system. In relation to errors, it is important to separate between errors that are repairable (users can recover from them) and errors that lead to a catastrophic failure.
2.2. Measuring User’s Subjective Opinions
Quantitatively, user’s perceived usability ratings are often collected using questionnaires. A number of usability questionnaires have been carefully developed and tested for their psychometric qualities, such as (SUS) [15], Software Usability Measurement Inventory (SUMI) [16], Questionnaire for User Interface Satisfaction (QUIS) [17], The After-Scenario Questionnaire (ASQ), Post-Study System Usability Questionnaire (PSSUQ), The Computer System Usability Questionnaire (CSUQ) (all three by [18]), and measurement scales for perceived usefulness and ease of use [19]. The questionnaires do not only focus on user satisfaction, but also study how users perceive the other attributes of usability.
The questionnaires differ in the number and contents of the constructs they measure. For example, SUS consists of 10 statements that cover several aspects of usability, such as the need for support, training, and complexity; QUIS, on the other hand, studies eleven dimensions, such as screen factors, terminology, system feedback, learning factors, and system capabilities. In both SUS and QUIS, the items focus on the use of the system (e.g., “Overall reactions to software,” “System speed”) and on the details of the user interface or system functionality (e.g., “I found the various functions of the system were well integrated”). On the other hand, the three items in ASQ focus on tasks (e.g., “Overall, I am satisfied with the ease of completing the tasks in this scenario”); PSSUQ and CSUQ include items with system, task, and interface focus. The measurement scales for perceived usefulness and ease of use by Davis’s Technology Acceptance Model (TAM) [19] focus specifically on the use of the system in the context of work tasks.
There are also some models and questionnaires that are specifically developed for mobile devices and services. For example, the applied TAM for Mobile services (TAMM) extends the original TAM with two new perceived product characteristics affecting the intention to use: trust and ease of adoption. In addition, TAMM model redefines the theme of usefulness as value to the user [20]. In addition, the Value-based Adoption Model (VAM) extends TAM by explaining customer’s mobile internet adoption from the perceived value perspective including usefulness, enjoyment, fee, and technicality [21]. The Mobile Phone Usability Questionnaire (MPUQ) evaluates the usability of mobile phones for the purpose of making decisions among competing variations in the end-user market, alternatives of prototypes during the development process, and evolving versions during an iterative design process. In addition, the questionnaire can serve as a tool for identifying diagnostic information to improve specific usability dimensions and related interface elements [22]. Moreover, Wehmeyer [23] has developed an initial instrument for assessing users’ attachment to their mobile devices by three dimensions: symbolism, aesthetics, and perceived necessity. However, all these previously mentioned methods are developed for consumer products and services. Thus, there is still a need for analysing the characteristics of mobile business services that are used in a work context. The networked nature of the mobile work system requires further extensions to user acceptance and usability measures.
2.3. Combining Objective and Subjective Measures
The usability research publications often report results in relation to the three factors of usability mentioned in the ISO standard [13], namely, satisfaction, efficiency, and effectiveness. However, these constructs are often reported separately: one system is reported as more efficient than the other, whereas, in relation to effectiveness, the results do not differ between the interfaces. Subjective satisfaction is often used to give additional testimony on the superiority of one interface over the other. A reasonable question is whether all the different aspects of usability need to be studied or whether one would be sufficient: if effectiveness, efficiency, and subjective satisfaction all point to the same direction, maybe it is enough to simply (and cheaply) ask the user how much they liked the service and use this score as an indication of the usability of the service.
Nielsen and Levy [24] reviewed 57 studies and concluded that although there is a clear association between subjective satisfaction and task performance, there are still many cases where the “users prefer systems that are measurably worse for them” [24]. Later, Frøkjær etal. [25] showed that correlations between the three aspects of usability are dependent on the application domain, the user’s experience, and the use context, and that no simple correlation between the three can be expected. Especially, for complex “nonroutine” tasks, efficiency measures as indicators of usability are useless if the task completion (effectiveness) is not taken into account [25]. Hornbæk [26] also encourages the measurement of all the three aspects of usability with both subjective and objective methods. However, Hornbæk [26] does not limit the subjective measures to subjective satisfaction. Instead, many objective measures, such as task time, can also be evaluated subjectively: the objective task time with System A may be faster than with System B, but the perceived time may be well faster with System B. The implications such results warrant are dependent on the situation, but overall, the results can give important information on the usability of the system.
3. Measuring Productivity
It is widely agreed that IT investments in general have an impact on the output of the company and thus, on productivity (e.g., [27]). Productivity is a multidimensional term and its meaning can vary depending on the context in which it is used [28]. Productivity can be defined simply as output divided by the input that is used to generate the output: input consists of materials, labour, capital, energy, and so forth, and output consists of products, services or information (e.g., [28–30]). This section provides an overview on the existing tools for measuring productivity of a company. In addition, the limitations of measuring productivity improvement resulting from the use of mobile business services are discussed.
3.1. Traditional Productivity Measurement
Traditionally, productivity measures have been categorised as either total productivity measures or partial productivity measures [29]. In practise, the measurement of total productivity (i.e., comparing all outputs with all inputs) is difficult as different types of outputs and inputs cannot be summed up and compared directly. Therefore, partial productivity measures are commonly used. Partial productivity ratio can be calculated by dividing the total output by one input or some portion of inputs [29]. Labour productivity is a most common example of this kind of measure and it is usually used when measuring the impacts of IT investments. Labour productivity is the ratio calculated by dividing the total output (e.g., number of products) by labour input (e.g., man-hour) [31].
Moreover, indirect productivity measurement can be used when the data needed for partial productivity measures is not available. The idea behind indirect productivity measures is that certain symptoms or phenomena are related to problems in productivity. They include, for example, high defect rates, machine defects, unused capacity, high material scrap, unnecessary transports, poor atmosphere, and long waiting times (e.g., [30, 32]). Indirect productivity measures focus on these factors which are closely related to productivity and which can be used to explain the reasons behind the changes in productivity [31].
According to Misterek etal. [29], the problem with productivity measures is that they do not take into account changes in the quality of the inputs or outputs. Especially in service business, quality and productivity cannot be dealt separately [33]. Moreover, measuring the effects of mobile business services on productivity with these traditional measures may be difficult because of business process changes and indirect influences (e.g., improved customer satisfaction) these services cause [34]. For example, Brynjolfsson and Hitt [35] noted that information technology investment may create intangible assets which in turn may have a positive impact on the productivity of the company but are not included in traditional evaluations. Including intangible aspects could provide a more accurate and realistic view of returns as a result of the investment [36].
3.2. Subjective Productivity Measurement
The use of the traditional productivity measurement methods presented above requires collecting quantitative data from operations. Sometimes the inputs and output are not easily quantifiable and it may be difficult or even impossible to collect the data that is needed for productivity measurement. For example, in service work, inputs and outputs are usually intangible and the quality may vary a lot. In these cases, subjective productivity measurement is one possible method to collect the needed information about the level of or problems in productivity. Subjective productivity measures are based on personnel’s subjective assessments and data is usually collected using surveys or interviews (see, e.g., [31, 37]). As traditional objective productivity measures, also subjective measures can be direct or indirect. Direct productivity measures can be, for example, partial productivity ratios (objective) or survey questions regarding employees’ perceptions of their own productivity (subjective). Indirect productivity measures can be, for example, unused capacity, unnecessary transports, and long waiting times (objective) or survey questions concerning employees’ perceptions of waiting times needed (subjective) [31].
As subjective measurement is also a common way of measuring usability, it could provide a basis for combining usability and productivity perspectives. In addition, it could be a good way of including intangible aspects (such as quality) when measuring the productivity impacts of mobile business services.
4. Exploring the Success Factors of Mobile Business Services
The main aim of this paper was to study the factors that have an effect on the success of mobile business services and to construct a preliminary measurement tool for measuring the level of success. Given the variability of tools already available for measuring usability and productivity (the ones reviewed in previous sections and much more), why would anyone want to develop yet another set of tools? For us, there were two main reasons: first, the current tools do not address the specific context we are interested in, namely, mobile services for business use, and second, the current tools only address either usability—or productivity-related issues—and we strongly feel that both of them need to be addressed in order to measure success. In order to understand the characteristics and success factors of mobile services that are used in a mobile work context, we conducted two case studies.
In the first case study, we examined a mobile business service used for taxi booking and dispatching. Taxi drivers use the service constantly in their daily work while driving. The service is used, for example, for accepting incoming requests, searching customers’ addresses and other locations on the map, and messaging to the dispatching centre and other taxis. The service is used on a fixed in-car device. In the second case study, we studied a mobile business service that is used at construction sites for weekly safety measurements. Safety delegate observes various safety issues at the construction site and uses the mobile service to document the observations. This service is used on a mobile phone.
4.1. Methods and Participants
Our goal is to gain understanding of the phenomena, that is, measuring the success factors of mobile business services, and we are using qualitative research methods to gain that understanding. Therefore, our study is based on an interpretive research paradigm whereby we aim at understanding the context of the system better (see [7]) (cf. the positivist paradigm where the assumption is that the measurable properties of the phenomenon are not dependent of the single observer). As data collection methods, we used focus groups, interviews, and observations. The data collection took place during the spring 2006.
For the first case study, we conducted two focus groups with a total of nine taxi drivers and two pair interviews with dispatchers to analyze the success factors in their mobile work context while using the taxi order booking and dispatching system. To evaluate the service from the perspective of new users, we observed the training of almost 50 taxi driver candidates during three three-hour training sessions. During these sessions, the candidates used the mobile service to complete typical usage scenarios in a classroom setting. To evaluate the flow of interaction and other usability issues of the taxi drivers’ system, two usability specialists spent one hour using a demonstration system to complete typical taxi driver’s tasks (such as accepting a customer order, locating an address from the map, and changing the settings of the interface). To gain a wide business perspective on the themes related to the success of the mobile business service, we interviewed the taxi company’s managing director and the chairman of the board.
For the second case study, we had one group discussion at a construction site, where we discussed the use of the service with the representatives of the construction company and the service provider. In addition, we interviewed one user at another construction site and observed three users at two different construction sites while using the mobile service to perform safety measurement. To better understand the challenges and opportunities that the new service brings along, we also observed the traditional safety measurement process where the safety measurement was done by filling in a paper form. After the observations, we asked the users to fill in the SUS questionnaire in order to get information about the usefulness of the SUS questionnaire and its limitations in this context.
4.2. Data Analysis
The focus group interviews in the first case study and all the other interviews were recorded with a digital voice recorder and transcribed for further analysis. The transcribed interviews were analysed qualitatively in order to find important themes related to mobile work from them (content analysis). Each interview was analysed at least by two researchers. When observing the users, the data was collected by the researchers (by taking notes). The observations were analysed so that the observed process was deeply understood: the roles and tasks of the actors, how they performed their work, which actors were involved in the process, and so forth. SUS questionnaires were analysed following its guidelines. We also compared the results of the questionnaires and the observations in line. After the data collection phase, the researchers had a group discussion where the goal was to elicit a common understanding on the important themes.
5. Results
In this section, we will present the dimensions related to the success of mobile business services we found in the case studies. In Section 5.2, we will present the initial version of the questionnaire for measuring the success of mobile business services.
5.1. Success Factors Revealed in the Case Studies
In both of the case studies, we found that the mobile business service needs to fulfill similar usability goals than other systems. These services need to be effective, efficient, pleasant, and easy to use to make it easier for the mobile workers to accept the new service in their daily work. From the companies’ point of view, improved productivity was seen as an important success factor. This is also in line with previous studies (see, e.g., [3, 4, 11]). In both of the cases, the mobile business service has helped the users in performing their work more productively. Moreover, when the users of the services are mobile, safety is the most important aspect of success. Taxi drivers are driving and construction workers walking around the construction sites while using the services. In both of these cases, users need to observe the environment and minimise their interaction with the service to avoid accidents.
In addition to the general usability goals, productivity and safety, we found that learnability is important for the success of the service. Earlier, Nielsen [14] has emphasised the role of learnability in usability evaluation. In addition, Zhang and Adipat [38] as well as Turel [39] have discussed the importance of learnability in mobile usability. In business use, severe errors should not happen even when the users are learning to use the service and learning should not take too much time. The learning phase may be expensive for both the company deploying the service (work tasks cannot be done as efficiently as possible) as well as for the service provider. For the service provider, the costs are due, for example, to numerous calls to the helpline and extended need for on-site support.
Before using the safety measurement service, the users have to install it into their mobile devices. This phase was found to play a critical role in the success of the service. If the users did not succeed with installation, they could easily switch back to using the traditional pen-and-paper method to perform their work task. For the construction company, this meant that the benefits of the service were not achieved. The installation phase was not as critical in the taxi case, because the taxi drivers did not install the services themselves. However, it should be noted that installation is a one-time activity and increasingly automated for the users. After successful installation, the users are not likely affected by the possible difficulties in the installation process too much. The most important thing is that the users can get the service installed.
In the taxi case, one factor having a large effect on the users’ opinions of the service was the level of support, whether the users felt that they receive enough support and that their problems and ideas are heard and acted upon. Support had also an important role in the construction case where the users of the mobile safety measurement service needed additional help while on the move. For example, during one observation, the user did not know how to answer an incoming call without distracting the safety measure and at the end, he needed to start the measurement again. This resulted in there being two measurements in the system and the user support needed to combine the two measurements together.
Moreover, we found that mobile business services need to take into account issues related to the mobile context. In previous studies, understanding the context is also seen essential for the success of mobile service design [8, 40]. In addition, Kemppilä and Lönnqvist [31] stated that there is a need to go deeper than just tracking the physical attributes to be able to understand the effects of the context better. In analysing the context of work, we need to understand (1) structure or cues that invites users to enter information spaces or tasks (entry points), (2) the mental and physical spaces users are constructing while accomplishing tasks (action landscapes), and (3) artefacts and environmental structures that help users manage the complexity of their tasks (coordinating mechanism) [31].
Exploring the context of work, the environmental context (including surrounding environment and physical elements but also organizational factors) sets requirements for the design of mobile business services. Sometimes, changes in the context are challenging to manage, for example, sunshine or coldness when using mobile services. Organizations provide the work spaces and technology, such as network connections and devices for their employees. In the taxi case study, taxi drivers needed to cope with the changing environmental context and we found that connections were lost from time to time when changing location, cold weather slowed down the device, and sunshine made it difficult to read the screen. In the construction case, lost connection also distracted the main task, that is, the safety measurement process and small keys on the mobile phone were difficult to use, especially in the winter-time with gloves. In addition, new systems need to be compatible with the old ones.
5.2. Development of a Measurement Tool
In order to develop a proper measurement tool for mobile business services, we decided to develop an initial version of questionnaire. A questionnaire as a research method is relatively efficient and cheap. The advantage of a usability questionnaire, compared to laboratory evaluations, expert reviews, and checklists usages, is that a questionnaire is reasonable easy and quick to be carried out and also the real end users of the product are involved in the process. The measures made with a questionnaire can be used, for example, to make comparison of user satisfaction between different versions of the same service or with competing services (benchmarking).
To begin the development of the questionnaire, we took an existing questionnaire, SUS, which we tested in our second case study with construction workers. This questionnaire is simple and quick to fill in, but unfortunately its usefulness for the current context was found to be limited as it does not take into account the mobility of the service or the business context. For example, the claim “I think that I would like to use this system frequently” is confusing as the users must use the mobile service to perform their work tasks. We also noticed that some of the statements in SUS were ambiguous as “I found the various functions in this system were well integrated”.
From the case studies, several success factors were identified: effectiveness, efficiency, user satisfaction, ease of use, safety, learnability, ease of installation, support, factors related to mobile context, and productivity. Effectiveness, efficiency, and user satisfaction are also emphasised in the ISO standard’s definition of usability [13]; efficiency, learnability, and satisfaction are emphasised by Nielsen [14], and ease of use is, for example, part of TAM [19]. In addition, installation and support (technical manuals and online help) are part of QUIS [17].
Some of the above-mentioned issues are addressed by the existing questionnaires (SUMI, QUIS, PSSUQ, SUS, USE questionnaire, TAM, and MPUQ), but they do not specifically focus on the issues that are relevant for the mobile work context. Thus, we revised the otherwise-relevant questions from the existing tools to better measure the success factors that are important for mobile business services. We also added new dimensions of success, namely, safety, mobile context, and productivity.
The existing usability measures have been mainly developed for desktop computers or consumer markets and used in usability tests. In contrast, the usability measures for mobile business services must take into account the different contexts of use and the workload the use of the service poses, among other things. Mobile business services can be used with different devices, such as mobile phones or fixed in-car systems, and the usability measures need to take into account the appropriateness of the device to the users’ task. In addition, the existing tools do not take into account the fact that the service is typically only part of the story; the service may fail to support the work task because of problems in some other parts of the network that is needed to support the use of the service. The mobile service rarely acts as an independent tool in the working environment but information needs to be transferred somewhere. Thus, it is important that the mobile service and the device enable fast and easy connection with other devices and users. To address these issues, the dimension of mobile context includes questions related to the actual physical context, that is, mobile device and environment, as well as the overall system context (see, e.g., [6]).
In the context of consumer services, subjective satisfaction and ease of use are extremely important factors as they have a large effect on the acceptability and adoption of the service [20]. In addition, the users of consumer services often have the freedom of choice—if they do not like the current service, they can choose another one. In business context, the end users (workers) often do not have a choice of not using the system and unfortunately, they often do not even have a say when the company is deciding on which service deploy. Despite of that (or rather, because of that), we feel that subjective satisfaction is important to be taken into account also in the business context, as it is likely to have an effect on other measures, such as the overall work satisfaction and productivity. In order to evaluate the productivity impacts, we added both direct and indirect subjective productivity measures under the theme “mobile work productivity.” In addition, especially the questions under the themes “efficiency” and “effectiveness” are closely related to productivity.
The statements from the existing questionnaires were selected based on the themes and their relevance to business use. The selected statements were modified so that “mobile service” and “work tasks” were emphasised. It should be noted that these themes and questions need to be further adapted based on the context in which they are applied. For example, if the service is intended for taxi drivers and used inside of the vehicles, there is no need to ask if the service does not do that well in dust or while walking. In addition, some themes are more relevant in the deployment phase than in the continued use. For example, ease of installation and learnability are crucial in the early stages of usage, whereas the effects on efficiency and productivity may be observed over longer periods of use.
To summarise, we identified a total of ten different dimensions from our case studies and existing literature that are likely to have a significant effect on the success of mobile services. To meet the demands of measuring the success factors of mobile business services, we found that the existing questionnaires are not sufficient alone. Thus, the following factors were added to the repertoire: safety, mobile context, and impact on mobile work productivity. The list of dimensions and some examples of the revised and added statements for the questionnaire on users’ perceived usability and productivity impacts of the mobile service in work tasks are listed in Table 1.
The dimensions listed in Table 1 are the ones that we recognised from the case studies to be important determinants of the success of mobile business services. However, the dimensions and example items should not be treated as a ready-to-use questionnaire: the dimensions need to be prioritised for each case to focus on the most important aspects of success and more items should possibly be added to the most important dimensions to capture all the aspects of them. In its current state, this questionnaire does not propose a way to collect the ratings from the users. Although we expect Likert-scales to be a useful way to collect ratings, more research is needed to determine the most appropriate way.
6. Discussion
Previous research on usability measurement tools has focused mostly on desktop environments. Usability is a critical issue in any service, but when poor usability may affect the security of the mobile users (and their customers) or the productivity of their work, the issue becomes even more important. In addition, when usability may affect not only the business operations of the company who produced the service but also the companies deploying it, the business impact of usability may be noticeably large. Thus, when considering the success of mobile business services, we claim that usability, context of work, and productivity must be taken into account. Thus far, measuring the success of mobile business services has received little attention. There are several independent measurement tools for usability designers and business managers. To the best of our knowledge, this is the first attempt to combine the dimensions of usability and productivity into a single measurement tool for mobile business services.
In this paper, we identified ten success factors of mobile business services through two case studies by combining the approaches of usability and productivity measurement. These success factors include the ease of installation, learnability, ease of use, efficiency, effectiveness, user satisfaction, factors related to mobile context, safety, support, and mobile work productivity. Most of these factors are included in the current tools for usability measurement. Thus, the existing usability questionnaires were used as the basis for the current tool although they needed to be revised for the current purposes. In addition, themes needed to be added to capture all the important factors in relation to the success of mobile business services, such as the issues related to the mobile device, environment, and the overall system, as well as safety, support, and mobile work productivity.
We expect the results of the questionnaire to be useful, for example, in analysing and benchmarking the level in which the users of the service perceive the usability and productivity of the service to be. Moreover, by reusing the questionnaire, results may be used for detecting changes, for example, in satisfaction and productivity over time. The results not only help the service developers in developing more successful services, but they can also be useful for the company using the service. For example, information about the level of perceived productivity and problems related to productivity in mobile work can be used as diagnostic tools to improve the productivity over time.
In this paper, we are proposing that a questionnaire could be used to measure the success of mobile business services. We acknowledge the limitations of the questionnaire approach (it measures the users’ perception of things rather than the objective truth, users may have difficulties interpreting the questions)—but we claim that the benefits of this approach overweight the limitations: questionnaire is a fast data-collection method, it can address several different themes, and it can provide quantifiable information about the themes in question. However, we do not claim that questionnaire (or any other single data collection method) should be used alone. Objective performance data (task time, customers/day, or revenue) should naturally be used as indicators of success as well. However, relying only on sales or objective efficiency may result in the less successful parts of the service going unnoticed—and potential for further development not being realised.
In our two-year MOMENTO project, the next step is to test the suggested themes and items of the questionnaire in several case studies. In addition to making sure that the dimensions are sufficient and applicable to different domains where mobile business services are used, we will address the issues related to the success of the individual items, the optimal number of items per dimension, and the scoring of the questionnaires. In addition, the validity and reliability of the questionnaire will be evaluated (see, e.g., [41]).
In this paper, we have mainly addressed the success factors from a single worker’s viewpoint. Other themes that emerged in the case studies were related to measuring success factors at the organizational level and in a more objective manner. These issues will be explored and identified in more detail in our future work.
Acknowledgments
This study was conducted as part of the MOMENTO project (ensuring the success of mobile services for business usage by multidisciplinary measurement tools). MOMENTO is a research and industrial cooperation project that started in the beginning of 2006. The project is financed by Tekes (Finnish Funding Agency for Technology and Innovation) and five companies. The authors would like to express our acknowledgments for the representatives of the case companies and their customers who participated in this study. They would also like to express our gratitude to Professor. Kaisa Väänänen-Vainio-Mattila, Dr. Antti Lönnqvist, and Dr. Eija Kaasinen for their continuing support and guidance.