Multicriteria-Based Crowd Selection Using Ant Colony Optimization

Wang, Guan; Ali, Farhad; Yang, Jonghoon; Nazir, Shah; Yang, Ting; Khan, Abdullah; Imtiaz, Muhammad

doi:https://doi.org/10.1155/2021/6622231

Complexity

On this page

Abstract Introduction Literature Review Results and Discussion Conclusion Data Availability Conflicts of Interest References Copyright Related Articles

Special Issue

Complexity and Robustness Trade-Off for Traditional and Deep Models

View this Special Issue

Research Article | Open Access

Volume 2021 | Article ID 6622231 | https://doi.org/10.1155/2021/6622231

Multicriteria-Based Crowd Selection Using Ant Colony Optimization

Guan Wang,^1,2Farhad Ali,³Jonghoon Yang,¹Shah Nazir,³Ting Yang,²Abdullah Khan,³and Muhammad Imtiaz³

Academic Editor: Dr Shahzad Sarfraz

Received13 Dec 2020

Revised28 Dec 2020

Accepted13 Jan 2021

Published23 Jan 2021

Abstract

Internet-enabled technologies have provided a way for people to communicate and collaborate with each other. The collaboration and communication made crowdsourcing an efficient and effective activity. Crowdsourcing is a modern paradigm that employs cheap labors (crowd) for accomplishing different types of tasks. The task is usually posted online as an open call, and members of the crowd self-select a task to be carried out. Crowdsourcing involves initiators or crowdsourcers (an entity usually a person or an organization who initiate the crowdsourcing process and seek out the ability of crowd for a task), the crowd (online participant who is a having a particular background, qualification, and experience for accomplishing task in crowdsourcing activity), crowdsourcing task (the activity in which the crowd contribute), the process (how the activity is carried out), and the crowdsourcing platform (software or market place) where requesters offer various tasks and crowd workers complete these tasks. As the crowdsourcing is carried out in the online environment, it gives rise to certain challenges. The major problem is the selection of crowd that is becoming a challenging issue with the growth in crowdsourcing popularity. Crowd selection has been significantly investigated in crowdsourcing processes. Nonetheless, it has observed that the selection is based only on a single feature of the crowd worker which was not sufficient for appropriate crowd selection. For addressing the problem of crowd selection, a novel “ant colony optimization-based crowd selection method” (ACO-CS) is presented in this paper that selects a crowd worker based on multicriteria features. By utilizing the proposed model, the efficiency and effectiveness of crowdsourcing activity will be increased.

1. Introduction

Nowadays, individuals voluntarily offer their own time, talent, and money to engage in activities that include helping the poor and making the planet a better place [1]. The Internet has made it easier for people to be connected and to be a part of a collaborative function, and this collaboration of people over the Internet is conceptualized into a term known as “crowdsourcing” [2]. Crowdsourcing is modern paradigms that employ cheap labors (crowd) for accomplishing different types of tasks. The task is usually posted online as an open call and members of the crowd self-select a task [3]. Crowdsourcing is an online participative activity in which organizations make use of the heterogeneous group of people having knowledge and skills to complete the task with the announcement of an open call [4, 5]. Social networking improvements have made it possible for organizations to pool collective knowledge from people around the world, i.e., “crowds’ wisdom,” for finding best solutions to various problems [6]. The wide use of social networking services acts as a massive pool of workers. These workers vary in demography and in their population. The information present in their profiles is used for inferring abilities and preferences. Crowdsourcer registers specific crowds by utilizing built-in functionality such as in Facebook using private messages and in Twitter using “@”. There is, therefore, an evolving trend of “managed crowdsourcing” where employees are actively selected [7, 8]. The process of crowdsourcing involves initiators or crowdsourcers (an entity usually a person or an organization who initiate the crowdsourcing process and seek out the ability of crowd for a task), the crowd (online participant who is a having a particular background, qualification, and experience for accomplishing the task in crowdsourcing activity), crowdsourcing task (the activity in which the crowd contributes), the process (how the activity is carried out), and the crowdsourcing platform (software or market place where requesters offer various tasks and crowd workers complete these tasks)[9–12]. Crowdsourcing is widely used in various domains such as tagging images [13], schema matching [14], and entity resolution [15].

The crowdsourcing task is delivered to the group of people (crowd) who complete the tasks. The allocation of the task is an important characteristic in the crowdsourcing context, and it requires suitable techniques. If the allocation of tasks is carried out correctly, it provides best outcomes [11]. In crowdsourcing the task may be outsourced to a dispersed crowd (workers) who might be inexperienced on these tasks [16]. Crowd selection is becoming a challenging issue with the growth in crowdsourcing popularity. There may be an untrustworthy person, who sometimes makes errors in solving various types of tasks. Crowdsourcing will be effective if an appropriate crowd is selected [17–19]. Crowd workers differ in several dimensions, so it is mandatory that we delegate tasks based on different features [20]. A participant may be identified by basic information available in workers’ profiles, such as gender, nationality, education level, his majors, personality test score [21], his skills, and his willingness for performing tasks [22].

Diversity, largeness, and profiles difference of crowd workers across many dimensions (e.g., skills, motives, and socioeconomic backgrounds) are the foundation for the success of a given crowdsourced tasks. In addition to the involvement of dishonest workers the differences among workers may also contribute to varying quality of responses received [23–25].

Identification of high-quality workers is a significant, complex, and realistic problem as in the crowdsourcing activity various tasks are accomplished by global crowd whose size and nature is unknown. Crowd attitudes, behaviors, and skills must be identified prior to assigning tasks. High-quality work can be accomplished by workers possessing unique features such as the level of education, major, and age [21]. Crowd selection is a complex problem in which the skills and knowledge of huge crowd workers are matched with the requirements of a job [26]. Crowd selection or formation is an optimization problem that involves all types of approaches to build a crowd group to whom various tasks will be offered [27]. Various features of crowd workers were identified from existing studies with the aim to use it for crowd selection in crowdsourcing. To the best of our knowledge, the proposed “ACO-CS” is a novel approach as not a single prior study exists that addresses the problem of crowd selection in crowdsourcing using the ant colony approach. By utilizing this approach, the effectiveness and efficiency of crowdsourcing activity would greatly be increased as it selects appropriate crowd workers according to multicriteria features for accomplishing various crowdsourced tasks.

2. Literature Review

Crowd selection is an important step in the activity of crowdsourcing process as the selection may make the activity effective one or may affect the activity. Various techniques were utilized for Crowd Selection in literatures which are discussed in subsequent sections.

2.1. Crowd Selection Based on Trustworthiness

Trust is the key element in choosing employees for a task [19, 27–29]. In different practices such as in the discovery of truth and in selection of workers, trust plays a significant role [30]. Employer can employ a trustworthy crowd worker [28]. SSC (strong social component) and C-AWSA (context-aware worker selection approach) are used to classify trustworthy workers, which is an accurate and useful algorithm for choosing trustworthy employees. It utilizes trust quality for optimization purposes. The forward search algorithm is used to measure trust of a worker [19]. Crowd trust efficiently chooses workers who are trustworthy according to an approval rate. The model effectively distinguishes dishonest workers and trustworthy workers [31].

2.2. Crowd Selection on the Basis of Expertise Filtering

Expertise [11, 26–28, 32–37] level of the crowd is estimated by expertise filtering. According to the expertise level, a right crowd for a task is selected [33, 34]. If they have expertise in the appropriate field, workers will be assigned tasks [27]. As an individual has task expertise, he will attempt to complete it with full attention [26]. Crowd of diverse expertise level is selected for tasks [35].

2.3. Crowd Selection on the Basis of Individual Profiles

Workers interested in participating on various tasks are required to build profiles that consist of various worker information, such as age, gender, skill, interest, and accomplished task history that will be stored on the platform in the worker profile database [38–40]. There are three forms of profiles. A declarative profile is created by workers themselves, the derived profile is determined from the system’s user interaction, and hybrid profile that contains declarative as well as derived information [41].Profiling evaluates the ability of individuals to work [32]. Profiles contain three types of information: first, the voluntary information they provide about themselves; second, the information and criteria that the platforms collect about their job performance; third, the assessments of their customers. Unlike crowd discourses, these platforms allow employees to modify their individual profiles to varying degrees of independence. This is not a concession to the online workers of autonomy and self-determination, but it is because of the competitive structure of the global labor market. Platforms assist consumers in evaluating workers by providing individually tailored profiles (in the absence of traditional recruitment documents and job interviews) [42]. Relevant workers for tasks are filtered using their profiles information. [34, 43]. Based on the level of task quality delivered, these profiles are updated [38].

2.4. Crowd Selection on the Basis of Task-Related Qualification

Several qualification criteria for workers may be defined by requesters. Such criteria often include test forms that are necessary to assess their qualifications [34]. Skill tests are also carried out to confirm worker qualification [44]. The workers are selected on the basis of relevant qualification they possess. For evaluating the ability of workers, qualification tests are conducted and a worker must pass the required examination to work on a project. It determines the capacity of workers in the activity. The qualification tests ensure that each employee has job related knowledge. Workers are selected on the basis of their score in qualification tests [45, 46].

2.5. Crowd Selection on the Basis of Experience

Workers are selected according to their experience in a task [47]. By utilizing the experience strategy, experienced workers are selected [48]. The selection of the experienced worker can make significant differences in the results of a task, i.e., it can produce high-quality results [35].

2.6. Crowd Selection on the Basis of Skills

Skills is a major personal attribute considered for appropriate participant selection [49]. Organization selects skilled work force for various tasks [42]. As the quality depends highly on workers’ skills [43], an Initial screening of the crowd workers is carried out [50]. These screening are also referred as skill assessment which evaluates the crowd according to possessing skills and they are helpful in matching skilled labor to a task [26, 51]. Activity-based positions on the platform are allocated to workers with the essential training and skills. Crowd workers are automatically assigned to tasks if they possess the required skills [27].

2.7. Need for the Proposed Study

Existing studies focus only on single or few features of crowd for addressing the problem of crowd selection that is not sufficient for selection of appropriate crowd to carry out crowdsourcing activity. Our proposed model “ACO-CS” will increase the efficiency and effectiveness of crowdsourcing activity as it selects crowd based on multicriteria features; therefore, appropriate multifeatured crowd will be selected to accomplish various tasks in crowdsourcing activity.

3. Methodology

The selection of quality crowd workers is a challenging issue. The workers are identified by their unique features [21]. These features are identified in the literature analysis and the selection of crowd will highly defend on these identified features. The feature set consists of large redundant data which will affect the appropriate selection of crowd workers. To remove redundancy and complexity, these are filtered out.

3.1. Feature Selection

A feature is an individual assessable asset of the process being investigated [52]. Feature selection (FS) is used in machine learning processes particularly in solving complex feature problems. Feeding a wide range of features into a model of recognition not only raises the strain of computation but also creates the issue widely known as the dimensionality curse. With feature selection, a large and complex dataset is reduced, as appropriate features are sorted out. Feature selection is broad and extends through many areas, including categorization of text, data mining, and identification of patterns and processing of signals [53]. Using the feature selection approach, a feature subset from the original set is selected and the accuracy of the original set is preserved. The efficacy and scalability will be increased by eliminating unnecessary and redundant features [54]. In dealing with larger feature datasets, the selection of features is obligatory. FS is a requirement in real-world problems due to the proliferation of noisy, meaningless, or deceptive characteristics.

Despite the availability of data with hundreds of variables leading to high-dimensional data, many feature selection strategies were used by researchers for selecting best features. The various feature selection techniques provide us with a way to reduce computing time, boost prediction efficiency, and better understanding of machine learning process or pattern recognition. Crowd consists of diverse and multidisciplinary people and this crowd posses unique features. Crowd features are collected from existing research studies. These are positive and negative features. The identification of these features was necessary for distinguishing appropriate and inappropriate crowd workers. These features are then filtered out to remove the ambiguity and complexity. A set of crowd containing multifeatures (Table1) was obtained as a result of combining the features that were captured during literature analysis. According to multicriteria features, our proposed method will select or reject various types of crowd. The various features of the crowd are represented in Table 2 (negative features), Table 3 (positive Features), and Table 1 overall (negative and positive) features.

3.2. Background

Ant colony optimization is a swarm intelligence technique which was introduced by M. Dorigo and his colleagues in 1990 [97]. They were inspired by the foraging behavior of certain species of ant. For leaving marks on encouraging direction that should be followed by other ant’s colony (members), these ants deposit various pheromones (chemical) on the ground. To solve optimization problems [98], ant colony optimization approach is utilized. Two variables are used by ants in solving problems such as heuristic knowledge and the value of pheromones. Quality outcomes can be generated as a result of the mutual communication among artificial ants. This is obtained by pheromone trail values through indirect contact (sensed the pheromone) of various ants. Ants do not change themselves but adjust the way other ants represent and view the problem adaptively [99]. ACO was mostly concerned with addressing the problems of ordering. One of ACO’s recent trends is to address various existing problems in the industrial sector [98].

3.3. Proposed Method

In crowdsourcing operations, the proposed ACO model can be extended to the problem of crowd selection. There are different steps involved in implementing the ACO algorithm in the selection of crowds. Figure 1 shows the overall operation of the proposed system “ant colony optimization-based crowd selection method (ACO-CS).” The proposed method undergoes the process of crowd selection. The selection of crowd starts with the generation of ants that will traverse on various paths (edges) and will select crowds on the basis of the pheromone value present on different edges, If the ant traversal satisfy a stopping criteria, the ants stops (traversal terminates) and the best subset of crowds is generated that will latterly be used for assigning different tasks. If the traversal does not correspond to stopping criteria, then the pheromone value is updated and once again the process is initiated.

3.3.1. Ant Colony Optimization-Based Crowd Selection Method

A set of 10 crowd workers are present (Table 1). The crowd selection technique is to minimize crowd subset which will be less than the original set of crowd; a higher accuracy in the depiction of the original crowd set will be retained. The partial selection may be in any order between the solutions. Simultaneously, the potential crowd to be chosen is not generally affected by the prior crowd attached to the node. It is not, however, necessary that the solutions to the selection of crowds should be of equal scale. The following steps are considered in mapping the crowd selection problem to the ACO algorithm:(i)The graphical representation(ii)Pheromone and heuristic desirability(iii)Updating the pheromone value(iv)Outcomes formation

(1) The Graphical Representation. The problem of crowd selection can also be defined in terms of ant colony optimization problem. ACO generally represents the problem in a graphical form as represented in Figure 2. The nodes signify various types’ crowds, and the edges reflect the corresponding crowd choice. The nodes are linked with each other to permit the selection of any crowd. An optimal subset of the crowd is selected when ant traverse over the graph, i.e., visit various nodes. The ant traversal must satisfy stopping criteria (select optimal multifeatures appropriate crowds). In Figure 2, initially, the ants A, B, C, D, E, F, G, H, I, and J from their nest are allowed to start traversing to different nodes such as C1 or C2 and subsequently to C3, C4, C5, C6, C7, C8, C9, and C10. These ants in the traversal process leave pheromones (represented in Figure 3) which is a chemical substance on different edges. The ants move according to the probability of the pheromones level on various edges, i.e., if the levels of pheromone are high, the ant will select only high pheromones values edges (bold line) and select only those specific nodes (Figure 4).The ant A from the nest will select node C1 and then using transition rule, the ant selects a crowd C4. Next, it chooses C5, C8, and C9 . On reaching C9, the ant traversal satisfies stopping criteria and its stops its traversal and provides a partial solution of the original crowd set “C” that consists of crowd workers C1, C4, C5, C8, and C9. A high accuracy is achieved as an outcome of the crowd workers subset. The crowd subset is then used as a nominee for different tasks.

(2) Pheromone and Heuristic Desirability. Initially teams of crowd are evaluated to identify best crowd workers. A simple multistart local search method decides the initial selection of crowd alternatives. In general, heuristic function is used in combination with the pheromone value in the ACO algorithm to make a right transition. Quality crowd workers are selected by calculating the pheromone and heuristic value. If a crowd worker is to be selected, then this value is assigned with a greater value “1.” On the other hand, if a crowd is not selected, then the pheromone value is kept smaller “0.” An ant in C1 determines whether C3 is to be selected or not, and the decision is taken in accordance with highest pheromones probability on edges and the probability can be calculated using the following formula:

The probability of each ant selecting a node is determined using equation (1) where represents the probability, represents edges, and represents heuristic desirability. If an edge is to be selected, then value is kept higher otherwise lower. The traversal and selection of a node (i.e., crowd) depend on the pheromone value. The ant will traverse on edge having a greater pheromone value.

(3) Updating the Pheromone Value. If the stopping condition is not satisfied by ants traversal, then the pheromone is modified, a new collection of ants is generated, and once again the process iterates. The pheromone is modified according to the following formula on each edge:where value may be considered 0 and 1, and it is the coefficient pheromone trail decay. If the stopping criterion is not satisfied (i.e., best crowd subset is not produced), then the ants modify pheromone. More pheromones are laid on the best solution nodes by the best ants, and as a result, optimal solutions are revealed.

(4) Outcomes Formation. The entire process of ACO-CS initiates the formation of randomly positioned numbers of ants. Then, these ants are positioned on a graph and the numbers of ants are equally set to the number of crowds (i.e., both are 10).The process of path building from a specific crowd starts with every ant. They cross the nodes from those starting positions in a probabilistic fashion until the stopping condition is fulfilled (optimal multifeatures appropriate crowds are selected). For an ideal subset, the resulting crowd subset is obtained and investigated. When best crowd subset has been found, then the experiment ends and it is noted (Figure 1). The pheromone update takes place when the conditions are not met, new ant’s colony is created, and the process repeats again.

4. Results and Discussion

The process of crowdsourcing involves Internet crowds [5, 90]. Crowds are recruited from Internet-enabled societies. Crowd workers possess some attributes, such as qualification, age, gender, language, worker place, skills, past service, and experience. Based on these characteristics, workers are selected [6, 11, 39, 100]. An employer should carefully pick workers to produce quality results [56]. The success of the organization depends on the allocation of tasks to members of the crowd, which requires adequate control systems, such as screening workers [11, 41]. When choosing the crowd, care must be taken as it is noted that the quality increases with a varied choice of crowds [50]. In our process of crowd selection, a crowd set (10 number of crowd “C1, C2, C3, C4, C5, C6, C7, C8, C9, and C10”) possessing multiple features (positive and negative features) are positioned in a graphical structure. The crowds are represented as different nodes connected with each other with the help of edges; an equal number of ants (10 numbers) are generated for the purpose to traverse on various edges and to select various crowd nodes, i.e., partial solution of the crowd set. The partial solutions (crowd subset C) will be evaluated based on the pheromone probability on paths, if it meets the stopping criteria (i.e., it selects the best crowd subset C), then the ants will stop its traversal and will produce the best crowd subset having multiple features. If the ants do not satisfy the stopping criteria, the pheromones are updated, and for the second iteration, the process is initiated. The selection and rejection of nodes (crowd) depend on the probability of pheromone on each edge; if the value is higher, the nodes will be selected and best crowd subset will be produced in ant’s traversal and if the value of pheromone is less, the edge and in turn, the crowd node is rejected. Table 4 represents the ants and their selected path. The probability of edges is calculated to find the best traversing path and in turn selects only appropriate crowd that will be assigned with different tasks in various activities. In our crowd selection method, the crowd subset C1, C4, C5, C8, and C9 by ant A, traversal is selected as the evaluation of the probability of pheromone values on edges linking these nodes (crowds) were higher than the edges linking other nodes (Crowd).

5. Conclusion

In the proposed study, we proposed ant colony optimization-based crowd selection for the problem of crowd selection in crowdsourcing. The key contribution of our research is to select the crowd based on the multicriteria features as the selection of crowd in previous approaches was based on a single or few features that do not guarantee the appropriate selection of crowd and thus affects the crowdsourcing activity. With our presented approach (ACO-CS) best crowds are selected according to multicriteria features that will, in turn, make crowdsourcing activity efficient and effective. The model “ACO-CS” in this paper is presented theoretically with less number of crowds. In future, we will implement it practically as this model is effective (i.e., selects crowd on the basis of multi-criteria features) than existing techniques used for crowd selection; therefore, it will play an important role in the crowd selection phase in crowdsourcing activity.

Data Availability

No data were used to support the findings of this study..

Conflicts of Interest

The authors declare that they have no conflicts of interest.

References

N. Mazlan, S. S. Syed Ahmad, and M. Kamalrudin, “Volunteer selection based on crowdsourcing approach,” Journal of Ambient Intelligence and Humanized Computing, vol. 9, no. 3, pp. 743–753, 2018.
View at: Publisher Site | Google Scholar
J. Lee and D. Seo, “Crowdsourcing not all sourced by the crowd: an observation on the behavior of Wikipedia participants,” Technovation, vol. 55-56, pp. 14–21, 2016.
View at: Publisher Site | Google Scholar
J. Howe, “The rise of crowdsourcing,” Wired Magazine, vol. 14, pp. 1–4, 2006.
View at: Google Scholar
A. L. Zanatta, I. Steinmacher, L. S. Machado, C. R. B. De Souza, and R. Prikladnicki, “Barriers faced by newcomers to software-crowdsourcing projects,” IEEE Software, vol. 34, no. 2, pp. 37–43, 2017.
View at: Publisher Site | Google Scholar
A. Ellero, P. Ferretti, and G. Furlanetto, “Realtime crowdsourcing with payment of idle workers in the retainer model,” Procedia Economics and Finance, vol. 32, pp. 20–26, 2015.
View at: Publisher Site | Google Scholar
I. Dissanayake, J. Zhang, and B. Gu, “Virtual team performance in crowdsourcing contest: a social network perspective,” in Proceedings of the 2015 48th Hawaii International Conference on System Sciences, pp. 4894–4897, Kauai, HI, USA, January 2015.
View at: Publisher Site | Google Scholar
A. Bozzon, M. Brambilla, S. Ceri, M. Silvestri, and G. Vesci, “Choosing the right crowd: expert finding in social networks,” in Proceedings of the 16th International Conference on Extending Database Technology, pp. 637–648, Genoa, Italy, March 2013.
View at: Publisher Site | Google Scholar
C. C. Cao, Y. Tong, L. Chen, and H. Jagadish, “Wisemarket: a new paradigm for managing wisdom of online social users,” in Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 455–463, Chicago, IL, USA, August 2013.
View at: Publisher Site | Google Scholar
E. Estellés-Arolas and F. González-Ladrón-De-Guevara, “Towards an integrated crowdsourcing definition,” Journal of Information Science, vol. 38, no. 2, pp. 189–200, 2012.
View at: Publisher Site | Google Scholar
R. Ali, M. Hosseini, K. T. Phalp, and J. Taylor, “The four pillars of crowdsourcing: a reference model,” in Proceedings of the 2014 IEEE Eighth International Conference on Research Challenges in Information Science (RCIS), Marrakech, Morocco, May 2014.
View at: Publisher Site | Google Scholar
L. Machado, R. Prikladnicki, F. Meneguzzi, C. R. De Souza, and E. Carmel, “Task allocation for crowdsourcing using AI planning,” in Proceedings of the 3rd International Workshop on CrowdSourcing in Software Engineering, pp. 36–40, Austin, TX, USA, May 2016.
View at: Publisher Site | Google Scholar
M. J. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin, “CrowdDB: answering queries with crowdsourcing,” in Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data, pp. 61–72, Athens, Greece, June 2011.
View at: Google Scholar
X. S. Yang, R. Cheng, L. Mo, B. Kao, and D. W. Cheung, “On incentive-based tagging,” in Proceedings of the 2013 IEEE 29th International Conference on Data Engineering (ICDE), pp. 685–696, Brisbane, Australia, April 2013.
View at: Publisher Site | Google Scholar
N. Q. V. Hung, N. T. Tam, Z. Miklós, and K. Aberer, “On leveraging crowdsourcing techniques for schema matching networks,” in Proceedings of the International Conference on Database Systems for Advanced Applications, pp. 139–154, Wuhan, China, April 2013.
View at: Google Scholar
J. Wang, T. Kraska, M. J. Franklin, and J. Feng, “Crowder: crowdsourcing entity resolution,” 2012, https://arxiv.org/abs/1208.1927.
View at: Google Scholar
H. Li, B. Yu, and D. Zhou, “Error rate analysis of labeling by crowdsourcing,” in Proceedings of the ICML Workshop: Machine Learning Meets Crowdsourcing, Atalanta, GA, USA, June 2013.
View at: Google Scholar
M. Garcia Martinez and B. Walton, “The wisdom of crowds: the potential of online communities as a tool for data analysis,” Technovation, vol. 34, no. 4, pp. 203–214, 2014.
View at: Publisher Site | Google Scholar
A. Dubey, G. Virdi, M. S. Kuriakose, and V. Arora, “Towards adopting alternative workforce for software engineering,” in Proceedings of the 2016 IEEE 11th International Conference on Global Software Engineering (ICGSE), pp. 16–23, Irvine, CA, USA, August 2016.
View at: Publisher Site | Google Scholar
Y. Zhao, G. Liu, K. Zheng, A. Liu, Z. Li, and X. Zhou, “A context-aware approach for trustworthy worker selection in social crowd,” World Wide Web, vol. 20, pp. 1211–1235, 2017.
View at: Google Scholar
A. Smirnov, A. Ponomarev, and N. Shilov, “Hybrid crowd-based decision support in business processes: the approach and reference model,” Procedia Technology, vol. 16, pp. 376–384, 2014.
View at: Publisher Site | Google Scholar
H. Li, B. Zhao, and A. Fuxman, “The wisdom of minority: discovering and targeting the right group of workers for crowdsourcing,” in Proceedings of the 23rd International Conference on World Wide Web, pp. 165–176, Seoul, Korea, April 2014.
View at: Publisher Site | Google Scholar
P. Micholia, M. Karaliopoulos, I. Koutsopoulos, L. M. Aiello, G. D. F. Morales, and D. Quercia, “Incentivizing social media users for mobile crowdsourcing,” International Journal of Human-Computer Studies, vol. 102, pp. 4–13, 2017.
View at: Publisher Site | Google Scholar
J. S. Downs, M. B. Holbrook, S. Sheng, and L. F. Cranor, “Are your participants gaming the system? screening mechanical turk workers,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2399–2402, Atlanta, GA, USA, April 2010.
View at: Publisher Site | Google Scholar
A. Ghosh, S. Kale, and P. McAfee, “Who moderates the moderators? crowdsourcing abuse detection in user-generated content,” in Proceedings of the 12th ACM Conference on Electronic Commerce, pp. 167–176, San Jose, CA, USA, June 2011.
View at: Google Scholar
V. C. Raykar and S. Yu, “Eliminating spammers and ranking annotators for crowdsourced labeling tasks,” The Journal of Machine Learning Research, vol. 13, pp. 491–518, 2012.
View at: Google Scholar
J. T. Bush and R. M. Balven, “Catering to the crowd: an HRM perspective on crowd worker engagement,” Human Resource Management Review, vol. 31, no. 1, Article ID 100670, 2018.
View at: Publisher Site | Google Scholar
A. Sarı, A. Tosun, and G. I. Alptekin, “A systematic literature review on crowdsourcing in software engineering,” Journal of Systems and Software, vol. 153, pp. 200–219, 2019.
View at: Google Scholar
H. Amintoosi, S. S. Kanhere, and M. Allahbakhsh, “Trust-based privacy-aware participant selection in social participatory sensing,” Journal of Information Security and Applications, vol. 20, pp. 11–25, 2015.
View at: Publisher Site | Google Scholar
O. Folorunso and O. A. Mustapha, “A fuzzy expert system to trust-based access control in crowdsourcing environments,” Applied Computing and Informatics, vol. 11, no. 2, pp. 116–129, 2015.
View at: Publisher Site | Google Scholar
W. Feng, Z. Yan, L. T. Yang, and Q. Zheng, “Anonymous authentication on trust in blockchain-based mobile crowdsourcing,” IEEE Internet of Things Journal, 2020.
View at: Publisher Site | Google Scholar
B. Ye, Y. Wang, and L. Liu, “Crowd trust: a context-aware trust model for worker selection in crowdsourcing environments,” in Proceedings of the 2015 IEEE International Conference on Web Services, pp. 121–128, New York, NY, USA, July 2015.
View at: Publisher Site | Google Scholar
K. L. Jeffcoat, T. J. Eveleigh, and B. Tanju, “A conceptual framework for increasing innovation through improved selection of specialized professionals,” Engineering Management Journal, vol. 31, no. 1, pp. 22–34, 2019.
View at: Publisher Site | Google Scholar
Z. Zhao, D. Yan, W. Ng, and S. Gao, “A transfer learning based framework of crowd-selection on twitter,” in Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Chicago, IL, USA, August 2013.
View at: Publisher Site | Google Scholar
N. Luz, N. Silva, and P. Novais, “A survey of task-oriented crowdsourcing,” Artificial Intelligence Review, vol. 44, pp. 187–213, 2015.
View at: Google Scholar
Q. Cui, J. Wang, G. Yang, M. Xie, Q. Wang, and M. Li, “Who should be selected to perform a task in crowdsourced testing?” in Proceeddings of the 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC), pp. 75–84, Turin, Italy, July 2017.
View at: Publisher Site | Google Scholar
K. Mao, L. Capra, M. Harman, and Y. Jia, “A survey of the use of crowdsourcing in software engineering,” Journal of Systems and Software, vol. 126, pp. 57–84, 2017.
View at: Publisher Site | Google Scholar
A. Moayedikia, W. Yeoh, K.-L. Ong, and Y. L. Boo, “Improving accuracy and lowering cost in crowdsourcing through an unsupervised expertise estimation approach,” Decision Support Systems, vol. 122, Article ID 113065, 2019.
View at: Publisher Site | Google Scholar
B. Satzger, H. Psaier, D. Schall, and S. Dustdar, “Auction-based crowdsourcing supporting skill management,” Information Systems, vol. 38, no. 4, pp. 547–560, 2013.
View at: Publisher Site | Google Scholar
X. Peng, J. Gu, T. H. Tan et al., “CrowdService: serving the individuals through mobile crowdsourcing and service composition,” in Proceedings of the 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 214–219, Singapore, Singapore, September 2016.
View at: Google Scholar
K. Abhinav, G. K. Bhatia, A. Dubey, S. Jain, and N. Bhardwaj, “TasRec: a framework for task recommendation in crowdsourcing,” in Proceedings of the 15th International Conference on Global Software Engineering, pp. 86–95, Seoul, South Korea, May 2020.
View at: Google Scholar
T. Awwad, N. Bennani, K. Ziegler, V. Sonigo, L. Brunie, and H. Kosch, “Efficient worker selection through history-based learning in crowdsourcing,” in Proceedigs of the 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC), pp. 923–928, Turin, Italy, July 2017.
View at: Google Scholar
H. J. Pongratz, “Of crowds and talents: discursive constructions of global online labour,” New Technology, Work and Employment, vol. 33, no. 1, pp. 58–73, 2018.
View at: Publisher Site | Google Scholar
D. Sánchez-Charles, J. Nin, M. Solé, and V. Muntés-Mulero, “Worker ranking determination in crowdsourcing platforms using aggregation functions,” in Proceedings of the 2014 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), pp. 1801–1808, Beijing, China, July 2014.
View at: Google Scholar
P. Adamska, M. Juźwin, and A. Wierzbicki, “Picking peaches or squeezing lemons: selecting crowdsourcing workers for reducing cost of redundancy,” in Proceedings of the International Conference on Computational Science, pp. 510–523, Amsterdam, The Netherlands, June 2020.
View at: Google Scholar
E. R. Q. Weidema, C. López, S. Nayebaziz, F. Spanghero, and A. V. D. Hoek, “Toward microtask crowdsourcing software design work,” in Proceedings of the 2016 IEEE/ACM 3rd International Workshop on CrowdSourcing in Software Engineering (CSI-SE), pp. 41–44, Austin, TX, USA, May 2016.
View at: Google Scholar
F. R. Assis Neto and C. A. S. Santos, “Understanding crowdsourcing projects: a systematic review of tendencies, workflow, and quality management,” Information Processing & Management, vol. 54, no. 4, pp. 490–506, 2018.
View at: Publisher Site | Google Scholar
M. Alsayyari and S. Alyahya, “Supporting coordination in crowdsourced software testing services,” in Proceedings of the 2018 IEEE Symposium on Service-Oriented System Engineering (SOSE), pp. 69–75, Bamberg, Germany, March 2018.
View at: Google Scholar
S. Alyahya, “Crowdsourced software testing: a systematic literature review,” Information and Software Technology, vol. 127, Article ID 106363, 2020.
View at: Publisher Site | Google Scholar
X. Zhang, B. Gong, Y. Cao, Y. Ding, and J. Su, “Investigating participants’ attributes for participant estimation in knowledge-intensive crowdsourcing: a fuzzy DEMATEL based approach,” Electronic Commerce Research, vol. 145, pp. 1–32, 2020.
View at: Publisher Site | Google Scholar
A. Dwarakanath, U. Chintala, N. C. Shrikanth et al., “CrowdBuild: a methodology for enterprise software development using crowdsourcing,” in Proceedings of the Second International Workshop on CrowdSourcing in Software Engineering, Florence, Italy, May 2015.
View at: Publisher Site | Google Scholar
M. Christoforaki and P. G. Ipeirotis, “A system for scalable and reliable technical-skill testing in online labor markets,” Computer Networks, vol. 90, pp. 110–120, 2015.
View at: Publisher Site | Google Scholar
G. Chandrashekar and F. Sahin, “A survey on feature selection methods,” Computers & Electrical Engineering, vol. 40, no. 1, pp. 16–28, 2014.
View at: Publisher Site | Google Scholar
S. Kashef and H. Nezamabadi-pour, “A new feature selection algorithm based on binary ant colony optimization,” in Proceedings of the 5th Conference on Information and Knowledge Technology, pp. 50–54, Shiraz, Iran, May 2013.
View at: Publisher Site | Google Scholar
H. Liu and L. Yu, “Toward integrating feature selection algorithms for classification and clustering,” IEEE Transactions on Knowledge and Data Engineering, vol. 17, pp. 491–502, 2005.
View at: Google Scholar
Z. Peng, X. Gui, J. An, R. Gui, and Y. Ji, “TDSRC: a task-distributing system of crowdsourcing based on social relation cognition,” Mobile Information Systems, vol. 2019, Article ID 7413460, 2019.
View at: Publisher Site | Google Scholar
L. Tran-Thanh, S. Stein, A. Rogers, and N. R. Jennings, “Efficient crowdsourcing of unknown experts using bounded multi-armed bandits,” Artificial Intelligence, vol. 214, pp. 89–111, 2014.
View at: Publisher Site | Google Scholar
M. Hosseini, A. Shahri, K. Phalp, J. Taylor, and R. Ali, “Crowdsourcing: a taxonomy and systematic mapping study,” Computer Science Review, vol. 17, pp. 43–69, 2015.
View at: Publisher Site | Google Scholar
A. Moayedikia, W. Yeoh, K.-L. Ong, and Y.-L. Boo, “Framework and literature analysis for crowdsourcing’s answer aggregation,” Journal of Computer Information Systems, vol. 60, no. 1, pp. 49–60, 2020.
View at: Publisher Site | Google Scholar
K.-J. Stol, B. Caglayan, and B. Fitzgerald, “Competition-based crowdsourcing software development: a multi-method study from a customer perspective,” IEEE Transactions on Software Engineering, vol. 45, no. 3, pp. 237–260, 2019.
View at: Publisher Site | Google Scholar
T. D. LaToza and A. V. D. Hoek, “A vision of crowd development,” in Proceedings of the 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, pp. 563–566, Florence, Italy, May 2015.
View at: Publisher Site | Google Scholar
C. Qiu, A. Squicciarini, D. R. Khare, B. Carminati, and J. Caverlee, “CrowdEval: a cost-efficient strategy to evaluate crowdsourced worker’s reliability,” in Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, Stockholm, Sweden, July 2018.
View at: Google Scholar
P. Shi, M. Zhao, W. Wang et al., “Best of both worlds: mitigating imbalance of crowd worker strategic choices without a budget,” Knowledge-Based Systems, vol. 163, pp. 1020–1031, 2019.
View at: Publisher Site | Google Scholar
B. Jiang, J. Hamari, and A. Maedche, “Cooperation or competition - when do people contribute more? A field experiment on gamification of crowdsourcing,” International Journal of Human-Computer Studies, vol. 127, pp. 7–24, 2019.
View at: Publisher Site | Google Scholar
A. Ghezzi, D. Gabelloni, A. Martini, and A. Natalicchio, “Crowdsourcing: a review and suggestions for future research,” International Journal of Management Reviews, vol. 20, no. 2, pp. 343–363, 2018.
View at: Publisher Site | Google Scholar
F. Moodley, J. V. Belle, and N. Hasteer, “Crowdsourced software development: exploring the motivational and inhibiting factors of the South African crowd,” in Proceedings of the 2017 7th International Conference on Cloud Computing, Data Science & Engineering-Confluence, pp. 656–661, Noida, India, January 2017.
View at: Publisher Site | Google Scholar
L. Zhai, H. Wang, and X. Li, “Optimal task partition with delay requirement in mobile crowdsourcing,” Wireless Communications and Mobile Computing, vol. 2019, Article ID 5216495, 2019.
View at: Publisher Site | Google Scholar
E. Schenk, C. Guittard, and J. Pénin, “Open or proprietary? choosing the right crowdsourcing platform for innovation,” Technological Forecasting and Social Change, vol. 144, pp. 303–310, 2019.
View at: Publisher Site | Google Scholar
G. D. Saxton, O. Oh, and R. Kishore, “Rules of crowdsourcing: models, issues, and systems of control,” Information Systems Management, vol. 30, no. 1, pp. 2–20, 2013.
View at: Publisher Site | Google Scholar
Y. Amsterdamer, T. Milo, A. Somech, and B. Youngmann, “December: a declarative tool for crowd member selection,” Proceedings of the VLDB Endowment, vol. 9, no. 13, pp. 1485–1488, 2016.
View at: Publisher Site | Google Scholar
Y. Amsterdamer, T. Milo, A. Somech, and B. Youngmann, “Declarative user selection with soft constraints,” in Proceedings of the 28th ACM International Conference on Information and Knowledge Management, Beijing, China, November 2019.
View at: Google Scholar
R. L. Saremi, Y. Ye, G. Ruhe, and D. Messinger, “Leveraging crowdsourcing for team elasticity: an empirical evaluation at TopCoder,” in Proceedings of the 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), pp. 103–112, Buenos Aires, AR, USA, May 2017.
View at: Google Scholar
F. Saab, I. H. Elhajj, A. Kayssi, and A. Chehab, “Modelling cognitive bias in crowdsourcing systems,” Cognitive Systems Research, vol. 58, pp. 1–18, 2019.
View at: Publisher Site | Google Scholar
Y. Wang, Z. Cai, G. Yin, Y. Gao, X. Tong, and G. Wu, “An incentive mechanism with privacy protection in mobile crowdsourcing systems,” Computer Networks, vol. 102, pp. 157–171, 2016.
View at: Publisher Site | Google Scholar
L. G. Pee, E. Koh, and M. Goh, “Trait motivations of crowdsourcing and task choice: a distal-proximal perspective,” International Journal of Information Management, vol. 40, pp. 28–41, 2018.
View at: Publisher Site | Google Scholar
C. Qiu, A. C. Squicciarini, B. Carminati, J. Caverlee, and D. R. Khare, “CrowdSelect: increasing accuracy of crowdsourcing tasks through behavior prediction and user selection,” in Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, Indianapolis, IN, USA, October 2016.
View at: Google Scholar
S. Sharma, N. Hasteer, and J. P. Van-Belle, “An exploratory study on perception of Indian crowd towards crowdsourcing software development,” in Proceedings of the 2017 International Conference on Computing, Communication and Automation (ICCCA), pp. 901–905, Noida, India, May 2017.
View at: Google Scholar
M. Modaresnezhad, L. Iyer, P. Palvia, and V. Taras, “Information Technology (IT) enabled crowdsourcing: a conceptual framework,” Information Processing & Management, vol. 57, no. 2, Article ID 102135, 2020.
View at: Publisher Site | Google Scholar
M. Garcia Martinez, “Inspiring crowdsourcing communities to create novel solutions: competition design and the mediating role of trust,” Technological Forecasting and Social Change, vol. 117, pp. 296–304, 2017.
View at: Publisher Site | Google Scholar
T. Mack and C. Landau, “Winners, losers, and deniers: self-selection in crowd innovation contests and the roles of motivation, creativity, and skills,” Journal of Engineering and Technology Management, vol. 37, pp. 52–64, 2015.
View at: Publisher Site | Google Scholar
I. Dissanayake, N. Mehta, P. Palvia, V. Taras, and K. Amoako-Gyampah, “Competition matters! self-efficacy, effort, and performance in crowdsourcing teams,” Information & Management, vol. 56, no. 8, Article ID 103158, 2019.
View at: Publisher Site | Google Scholar
S. Standing and C. Standing, “The ethical use of crowdsourcing,” Business Ethics: A European Review, vol. 27, no. 1, pp. 72–80, 2018.
View at: Publisher Site | Google Scholar
H. Zheng, D. Li, and W. Hou, “Task design, motivation, and participation in crowdsourcing contests,” International Journal of Electronic Commerce, vol. 15, no. 4, pp. 57–88, 2011.
View at: Publisher Site | Google Scholar
G. Montelisciani, D. Gabelloni, G. Tazzini, and G. Fantoni, “Skills and wills: the keys to identify the right team in collaborative innovation platforms,” Technology Analysis & Strategic Management, vol. 26, no. 6, pp. 687–702, 2014.
View at: Publisher Site | Google Scholar
R. Saremi, “A hybrid simulation model for crowdsourced software development,” in Proceedings of the 2018 IEEE/ACM 5th International Workshop on Crowd Sourcing in Software Engineering (CSI-SE), pp. 28-29, Gothenburg, Sweden, June 2018.
View at: Google Scholar
B. Satzger, R. Zabolotnyi, S. Dustdar et al., “Chapter 8-toward collaborative software engineering leveraging the crowd,” in Economics-Driven Software Architecture, I. Mistrik, R. Bahsoon, R. Kazman, and Y. Zhang, Eds., pp. 159–182, Morgan Kaufmann, Boston, MA, USA, 2014.
View at: Google Scholar
T. D. LaToza, W. B. Towne, A. V. D. Hoek, and J. D. Herbsleb, “Crowd development,” in Proceedings of the 2013 6th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE), pp. 85–88, San Francisco, CA, USA, May 2013.
View at: Google Scholar
E. Segev, “Crowdsourcing contests,” European Journal of Operational Research, vol. 281, no. 2, pp. 241–255, 2020.
View at: Publisher Site | Google Scholar
J. Troll, I. Blohm, and J. M. Leimeister, “Why incorporating a platform-intermediary can increase crowdsourcees’ engagement,” Business & Information Systems Engineering, vol. 61, pp. 433–450, 01 2019.
View at: Google Scholar
O. Tokarchuk, R. Cuel, and M. Zamarian, “Analyzing crowd labor and designing incentives for humans in the loop,” IEEE Internet Computing, vol. 16, no. 5, pp. 45–51, 2012.
View at: Publisher Site | Google Scholar
A. L. Zanatta, L. Machado, and I. Steinmacher, “Competence, collaboration, and time management: barriers and recommendations for crowdworkers,” in Proceedings of the 2018 IEEE/ACM 5th International Workshop on Crowd Sourcing in Software Engineering (CSI-SE), pp. 9–16, Gothenburg, Sweden, May 2018.
View at: Google Scholar
X. Wang, H. J. Khasraghi, and H. Schneider, “Towards an understanding of participants’ sustained participation in crowdsourcing contests,” Information Systems Management, vol. 37, no. 7, pp. 1–14, 2019.
View at: Publisher Site | Google Scholar
M. Alhamed and T. Storer, “Estimating software task effort in crowds,” in Proceedings of the 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 281–285, Cleveland, OH, USA, September, 2019.
View at: Google Scholar
X. Zhang, Z. Chen, C. Fang, and Z. Liu, “Guiding the crowds for android testing,” in Proceedings of the 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C), pp. 752-753, Austin, TX, USA, May 2016.
View at: Google Scholar
S. Franken, S. Kolvenbach, W. Prinz, I. Alvertis, and S. Koussouris, “CloudTeams: bridging the gap between developers and customers during software development processes,” Procedia Computer Science, vol. 68, pp. 188–195, 2015.
View at: Publisher Site | Google Scholar
S. Hosio, J. Goncalves, N. van Berkel, S. Klakegg, S. I. Konomi, and V. Kostakos, “Facilitating collocated crowdsourcing on situated displays,” Human-Computer Interaction, vol. 33, no. 5-6, pp. 335–371, 2018.
View at: Publisher Site | Google Scholar
E. Mourelatos and M. Tzagarakis, “An investigation of factors affecting the visits of online crowdsourcing and labor platforms,” NETNOMICS: Economic Research and Electronic Networking, vol. 19, pp. 95–130, 2018.
View at: Google Scholar
M. Dorigo and C. Blum, “Ant colony optimization theory: a survey,” Theoretical Computer Science, vol. 344, no. 2-3, pp. 243–278, 2005.
View at: Publisher Site | Google Scholar
M. Dorigo, M. Birattari, and T. Stutzle, “Ant colony optimization,” IEEE Computational Intelligence Magazine, vol. 1, no. 4, pp. 28–39, 2006.
View at: Publisher Site | Google Scholar
H. Liu and H. Motoda, Feature Extraction, Construction and Selection: A Data Mining Perspective, vol. 453, Springer Science & Business Media, Berlin, Germany, 1998.
N. Leicht, I. Blohm, and J. M. Leimeister, “Leveraging the power of the crowd for software testing,” IEEE Software, vol. 34, no. 2, pp. 62–69, 2017.
View at: Publisher Site | Google Scholar

Copyright

Copyright © 2021 Guan Wang et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

561

Downloads

937

Citations