Applied Mathematics for Engineering Problems in Biomechanics and RoboticsView this Special Issue
Research Article | Open Access
Mass Collaboration-Driven Method for Recommending Product Ideas Based on Dempster-Shafer Theory of Evidence
In the mass collaboration mode, there exist a large number of product ideas with low value density and thousands of participants who are differed on their professional backgrounds, knowledge structures, and value orientations. It is impossible for each participant to give a comprehensive evaluation of each idea as that in traditional methods for the reasons as mentioned above. In order to solve this problem, a mass collaboration-driven method for recommending product ideas is proposed based on Dempster-Shafer theory of evidence (DST). Firstly, the method for computing basic probability assignment (BPA) function, which can effectively reflect the facticity of experts’ evaluations, is introduced by discounting belief degrees with weights to extract the evaluation information of product ideas. Then, Dempster’s combination rule is used to combine the derived BPA functions for two times: the first one is to combine the discounted BPA functions on all criteria with respect to a specified expert and the other is to combine the combined BPA functions for all experts with respect to a specified alternative. Finally, the steps of mass collaboration-driven method for recommending product ideas based on the DST are proposed. An illustrative example is provided to demonstrate the applicability of the proposed method.
Product idea refers to the production of novel and useful idea to complex, novel, and ill-defined problems [1–3]. Successful innovation requires high-quality ideas and how to effectively generate novel and useful product ideas continues to be a critical issue for both design scholars and practitioners.
At present, research on product ideas is mainly divided into two processes: one is idea generation, whereby ideas emerge in an associative manner, and the other is idea evaluation, whereby generated ideas are evaluated and screened. From the perspective of idea generation. Chou presented an ideation method for generating new product ideas ; Kazmi and Kytola attempted to explore assessment of a company’s new product ideas generation potential linked to their industrial teams’ diversified capabilities as well as their work potential ; Banović et al. used projective and creative research techniques to involve consumers in the process of modification and generation of new product ideas ; Shroyer et al. presented a qualitative case study of a professional design team’s use of ideas generation with analyses at five emergent timescales ; Giller et al. suggested that either unbounded or prohibitive task instructions should be used when crowdsourcing innovative ideas ; Kwon et al. proposed a methodology to be served as an essential supporting tool for generating creative ideas that could spark innovation ; Mirtalaie et al. presented a systemic framework for product designers in the ideation phase of new product development. From the perspective of product ideas evaluation; Steele et al. developed a measure of ideas evaluation self-efficacy [10, 11]; Özaygen and Balagué proposed a methodology to reduce crowd innovation voting bias and to help managers to better select the ideas ; Hao et al. explored the neural correlates underlying the effects of ideas evaluation on idea generation in creative thinking ; Mayseless et al. proposed an explanatory model of ideas centered upon the key role of the left temporoparietal regions in evaluating and inhibiting ideas ; Hoornaert et al. established a new model that can improve the reliability of ideas ranking by means of automated information retrieval methods, linear methods, and nonlinear machine-learning algorithms .
With the development of information technology, more and more subjects can participate in the innovation , and many enterprises have carried out product innovation in mass collaboration mode and achieved initial results [17, 18], as mass collaboration mode has reduced BMW’s R & D cycle from 2 years to half a year and made Linux Kernel’s software system update continuously in weeks; HUAWEI focuses on “user perception” and has established a close cooperative relationship with more than 30 telecom operators, industry alliances and institutions, vertical industry leaders, and so on and has released a new generation of Easy Macro solutions and a new generation of 3D MIMO solutions. Although the mass collaboration mode has helped many enterprises achieve some results, product innovation based on this mode is still an important issue facing the enterprise at present stage. On the one hand, there are a lot of ideas in the generation stage. Thousands of participants can publish their opinions on product innovation with various Internet platforms, thus generating a large number of ideas with low value density. On the other hand, there are many participants in the idea evaluation stage. The participants not only include the internal members of the enterprises but also include the vast number of current and potential users.
Some scholars have done some researches on idea generation and idea evaluation in product innovation, which are of great significance to enrich the research achievements in this field. However, the large number of ideas and participates in the mass collaboration mode leads to the existing achievements which are not suitable for solving the problem of product idea evaluation. The reason is that the participants may come from the relevant subjects such as the internal members and social public, and the knowledge, preference and ability of each participant may be different from each other. It is impossible for each participant to give a comprehensive evaluation of each idea as that in the traditional methods.
In order to solve the above problems, this paper puts forward a mass collaboration-driven method for recommending product ideas based on Dempster-Shafer theory of evidence (DST), on the basis of taking full consideration of the characteristics of the large number of ideas and the large number of participants. Existing methods in this field require all experts to give complete evaluation information on all criteria for each idea, but this is difficult to achieve in the mass collaboration mode. In contrast, considering the incompleteness of expert knowledge, the proposed method allows each expert to evaluate only one or more criteria for an idea in his/her field of expertise. Moreover, it allows each expert to be assigned different weights in each criterion, so as to obtain more reasonable and effective evaluation information. However, the proposed method can only sort the ideas from a static point of view and cannot achieve real-time sorting for the added ideas and evaluation information.
The rest of this paper is organized as follows. In Section 2, we introduce the framework of evaluating product ideas. In Section 3, we propose the method for recommending product ideas based on the DST and analyze the specific steps of the proposed method. In Section 4, we use an example to illustrate the combination process of the proposed method. Section 5 concludes the paper.
2. Framework of Evaluating Product Ideas
2.1. Generation of Ideas
Ever since the publication of Osborn’s (1953) influential book Applied Imagination, many organizations consider group brainstorming as a particularly effective technique for generating large numbers of creative ideas . To stimulate creativity, a group idea-generation technique was developed and popularized under the name of brainstorming . This technique has been extended to electronic brainstorming, in which group members simultaneously produce as many ideas as possible on computers in a short period of time . In view of the advantages of electronic brainstorming that electronic brainstorming groups perform better than face-to-face groups, in both the number of high-quality ideas and the average originality of ideas, we decide to use this method to generate the needed ideas.
In order to get effective product ideas, enterprises should publish innovative requirements on the Internet platform. For example, on the premise of guaranteeing the quality of the product, compared with the extravagant outer packaging, the enterprise can require a simple and clear design in the appearance and packaging of the product. It cannot only reduce the cost of packaging but also attract consumers’ attentions. In other words, the enterprise can require new product ideas to be differentiated in terms of product functionality, durability, style design, delivery, installation, customer training, and consulting services, so that new products are distinguished from the same products on the market.
After the enterprise announces the innovation request, the public can be the participants in the mass collaboration mode and all of them can express their opinions and discuss freely on the various interactive platforms. In order to get as much product ideas as possible, enterprises adopt electronic brainstorming method and make full use of all kinds of online platforms to encourage participants to participate in the process of product ideas generation around innovative requirements. The goal of electronic brainstorming is to generate a list of ideas, applying four rules: (a) focus on quantity, (b) withhold criticism, (c) welcome unusual ideas, and (d) combine and improve one’s own ideas. Thus, a large number of product ideas have been generated.
2.2. Classification of Participants
Nowadays, many companies rely on online innovation communities to collaborate together with users and integrate external user knowledge within firm boundaries for new product development purposes . All people except internal members of the enterprise can be regarded as users, who either have bought a product (current users) or are likely to buy a product (potential users). Numerous studies have shown that positive interaction between users and enterprises is conducive to the transformation of new technologies and the development of new products. In addition, employees, as insiders, have a deeper understanding of the real situation of the enterprise and the goal of product innovation and can participate in the whole process of product innovation. To sum up, according to the differences in value orientation and knowledge structure, this paper divides the participants in the mass collaborative mode into three categories: internal members of the enterprise, current users, and potential users.
Internal members of the enterprise: As the originator and manager of the product ideas, internal members of the enterprise participate in the formation of the ideas in the whole process, which can be divided into managers, technical developers, production managers, and marketers. Managers mainly evaluate an idea from the perspective of economic benefits; technical developers mainly evaluate an idea from the perspective of enterprises’ ability to transform existing technologies into product characteristics; production managers mainly evaluate an idea from the perspective of the production parameters (such as environment, medium temperature, pressure, etc.) and the technological design method; marketers mainly evaluate an idea from the perspective of marketing requirements.
Current users: The current users refer to the users of a technology, product, or service, who have used one product. They are the actual experiencers of the product, have a more comprehensive understanding of the advantages and disadvantages of the product, can provide the corresponding information for the problems existing in the product, and can also evaluate the existing ideas.
Potential users: The potential users are those users who have not been used but have the desire to buy some kind of product or service, have the ability to buy, have the right to make a decision, and have a demand for the function provided by the product. They can evaluate an idea from the perspective of whether the product can satisfy their needs.
2.3. Evaluation Criteria for Product Ideas
Early research on idea used the number of creative ideas as the only criterion to measure the quality of idea. With the deepening of idea evaluation, the quantitative evaluation method in the past is no longer applicable. At present, the general view of the research on product idea evaluation criteria is multicriteria comprehensive decision-making. This view holds that idea evaluation depends on multiple criteria and comprehensive decision-making .
Plucker et al. proposed the idea that the evaluation of product idea should include two dimensions: novelty and practicality . Chan et al. suggested that the customer lifetime value (CLV) calculated can be regarded as the final evaluation criterion used to support idea screening decisions . Rietzschel et al. proposed that a good idea should be new and practical and put forward two criteria: novelty and feasibility . Diedrich et al. believed that the primary criterion for judging an idea is novelty, followed by usefulness . Schwarz and Bodendorf suggested that a good idea should be feasible, novel, profitable, useful, and highly mature . Hart et al. pointed out that technical feasibility is the most commonly used evaluation criterion in idea evaluation . MacCrimmon and Wagner believed that the criteria of idea decision-making need to take into account five aspects: novelty, nonobvious, relevance, practicality, and thoroughness . Carbonell-Foulquié et al. believed that strategic conformity, technical feasibility, customer acceptability, economic performance, and marketing opportunity are 5 major criteria for evaluating an idea . Dean et al. pointed out that the essence of an idea is novelty, based on which four first level criteria of evaluation, novelty, feasibility, relevance and particularity, are established, and two level criteria are established under each criterion .
The above evaluation criteria are based on many angles and discussed the idea itself, but this is not to be ignored that the fundamental purpose of ideas research and development is to better maintain the enterprise’s own development. Therefore, the economic and technological benefits of ideas should also be emphasized. Hietikko proposed a model with two 2-dimensional tables, one for customer and the other for producer . The value for customer is divided into two factors: usefulness and economic efficiency, and the table of producer is divided into marketability and productivity. Eling et al. put forward that when evaluating an idea, we should consider the speed of evaluation and whether it can bring benefits to enterprises . Lauto and Valentin proposed that the decision time and the technological achievements contained in the idea must be taken into consideration . Ferioli et al. further refined the practicability of the idea evaluation criteria into two aspects: technology executable and economic benefits . Gutierrez put forward six principles of idea evaluation, which are core competitiveness, market position, module economic value, R & D capital, risk awareness, and organizational goal contribution .
On the basis of referring to the relevant literature, we select 4 criteria as the evaluation criteria of product ideas, as shown in Table 1.
2.4. Framework of Idea Evaluation
Each product idea can be regarded as an alternative. Suppose that a set of alternative ideas is expressed as , a set of three types of experts is expressed as , and a set of criteria is expressed as . Suppose the number of individuals in the experts Ei is Ii and is the sth individual in Ei, then the set of individuals in Ei can be expressed as . According to the above, E1= Internal members of the enterprise, E2=Current users, E3= Potential users, c1= Novelty, c2= Acceptability, c3= Usefulness, and c4= Feasibility. After the generation of the ideas, for each idea, every expert can give an evaluation information on the performance of a certain criterion according to his own knowledge and experience. For examples, internal members of enterprise can evaluate the above 4 criteria according to the principle of whether an idea can bring benefits to enterprises. Current users and potential users can evaluate the above 4 criteria according to the principle of whether an idea meets their needs. Suppose that the evaluation information of alternative ak evaluated by experts Ei with respect to criterion cj is expressed as . Thus, the framework of idea evaluation can be shown in Figure 1.
3. The Proposed Method
In this paper, based on the comprehensive consideration of weight, the DST and the Pignistic probability are used for ideas selection and ranking. The DST is an extension of classical probability theory by generalization of the Bayesian theory of subjective probability. Being a mathematical framework for representation of uncertainty, the DST combines the degrees of belief derived from independent items of evidences. It mainly deals with four concepts: frame of discernment, basic probability assignment (BPA), the belief or mass function, and the plausibility . Its advantages are mainly manifested in the following three aspects: first, it can handle more uncertainty in real world; second, no prior distribution is needed before the combination of evidence from individual information sources; third, it allows one to specify a degree of ignorance in some situations instead of being forced to be assigned for probabilities . The relevant definitions are shown as follows [40–43].
3.1. Information Extraction Method
In the mass collaboration mode, considering the differences in knowledge structure, cognitive ability, and value orientation, every expert can only evaluate one or several criteria of an idea, which determines that there may be uncertain information about “incomplete known” or “completely unknown” in the evaluation information. Compared with the fact that each expert can evaluate all the criteria in traditional methods, the BPA function, as an expression of uncertain information, allows each expert to evaluate the criteria that he has the ability to evaluate, thereby maximally ensuring the authenticity of the evaluation information [44–46]. In view of this, we use the BPA function to extract the evaluation information.
Definition 1. Suppose the grade of an idea is expressed as , each grade is exclusive, and the finite set of mutually exclusive and exhaustive propositions is the frame of discernment.
Definition 2. Supposing that the set of every subset of is , is any subsets of Θ, and m(θ) represents the degree of belief, if the mapping function could fulfillthen is called the BPA function of Θ. If m (θ) > 0, θ is called a focal element.
In this paper, let the frame of discernment is , θ1=Excellent, and θ2=Non-excellent, which represents the answer of “Is the alternative ak excellent or not on criterion cj for experts Ei?”. In order to obtain the evaluation information of three types of experts within the same framework of discernment Θ, we assume that the choice of with respect to criterion cj on alternative ak is . For example, if thinks that the alternative ak is excellent on criterion cj, then ; if could not give any evaluation information on the performance of ak on criterion cj, then . Suppose is the complementary set of , and we use and to define the discriminant function of the performance of ak on criterion cj, so we can obtain the degree of belief of alternative ak evaluated by experts Ei with respect to criterion cj, expressed as . The calculation methods of and are shown aswhere , , and .
When and , reflects the degree of local ignorance; when , reflects the degree of global ignorance. In particular, when =1, , which means that the experts Ei do not give any evaluation information on the performance of ak on criterion cj.
3.2. Weight Determination Method for Three Types of Experts
Because of the differences in knowledge structure and value orientation, the weights of the three types of experts on a criterion are different . Weight is the subjectively relative importance degree of an expert to another with respect to a criterion and usually can be determined by some computing methods such as Analytic Hierarchy Process (AHP) and Analytic Network Process (ANP). In this paper, we suggest using AHP method to determine the weights of three experts on a criterion. Three types of experts are compared two by two, thus the judgment matrix is constructed . The relative importance of each expert is obtained through normalization and a series of calculation steps. The weight of experts Ei on criterion cj can be expressed as wij, , .
In addition, in order to ensure that the evaluation information is more scientific and reasonable, the weight of each criterion should also be considered. We also suggest using AHP method to determine the weights of each criterion, which can be expressed as wj, , .
3.3. Information Combination Method
In this paper, Dempster’s combination rule is used to combine the derived BPA functions for two times. The first time is to combine BPA functions on all criteria with respect to the specified experts Ei, so we can obtain the overall evaluation value for alternative ak on all criteria given by experts Ei (called individual combination). The second time is to combine the combined BPAs for all experts with respect to a specified alternative ak, so we can obtain the overall evaluation value for alternative ak for all experts (called group combination).
Due to the weight of each criterion and the weight of various experts on each criterion being different, in order to ensure the authenticity of evidence sources, we adopt the Shafer’s discounting method to discount the obtained BPA function , and the discounting method is shown in
After making the Shafer’s discounting, is integrated with the weight of each criterion and the weight of each type of expert, and then Dempster’s combination rule can be used to make companion. Dempster’s combination rule can be expressed as with , where θ, θA,, and θB are subsets of Θ, is the orthogonal sum operator, and K is a normalization constant, called the conflict coefficient of two BPAs
Through the first combination, we can get the comprehensive evaluation value of experts Ei on all criteria of alternative ak, that is, . Then Dempster’s combination rule is used again to combine the combined BPA functions . Obviously, the combination process can be represented as , so we can obtain the overall evaluation value for alternative ak for all experts, expressed as .
3.4. Method Steps
The steps of mass collaboration-driven method for recommending product ideas based on the DST are as follows.
Step 1. Identify alternative ideas. In the mass collaboration mode, the product ideas are identified by various participants based on product innovation requirements. A set of ideas can be expressed as , and the main purpose of this paper is to rank them.
Step 2. Set decision parameters. Suppose a set of criteria is expressed as , a set of three types of experts is expressed as , a set of grades is expressed as , a set of utilities for grades is expressed as , a set of weights of experts’ type is expressed as , and a set of weights of criteria is expressed as .
Step 3. Select the first idea. Let k=1 and take a1 as the initial evaluation idea to be evaluated.
Step 4. Expert evaluation. In the mass collaboration mode, three kinds of experts, internal members of the enterprises, current users, and potential users, can evaluate the various attributes of existing ideas. The evaluation value of alternative ak evaluated by experts Ei with respect to criterion cj can expressed as
Step 5. Make individual combination. Taking the specific experts Ei as the combination target, based on wij and wj, Shafer’s discounting is used to discount and get the discounted BPA functions . Then Dempster’s combination rule is used to combine and we get the comprehensive evaluation value of experts Ei on all criteria of alternative ak, expressed as .
Step 6. Make group combination. Taking all experts as the combination target, Dempster’s combination rule is used to combine , and we get the overall evaluation value for alternative ak for all experts, expressed as .
Step 7. Determine probability distribution. The BPA functions obtained by Step 6 are converted to the form of probability distribution by the Pignistic probability. So we can obtainwhere is the concept of belief, is the concept of plausibility, and .
Step 8. Change the alternative idea. Let k=k+1; if k ≤ K, it means taking as the current evaluation idea, and going back to Step 4; if k>K, it indicates that the combination of all the ideas has been completed and going to Step 9.
Step 9. Rank alternatives. By combining grade with probability, we can get the probability distribution of the overall value of ak, which can be expressed as . According to the rank utility and the expected utility theory, the total utility of ak is calculated as The greater the total utility Uk is, the better ak is. Then we can get the rank of all the ideas.
4. Illustrative Example
HUAWEI Technologies is a private high-tech firm based in Shenzhen, China, focusing on the provision of next generation telecommunications networks. Huawei is committed to providing Internet-protocol (IP)-based fixed & mobile communication (FMC) solutions to ensure that end users are able to experience consistent communication services at anytime, anywhere. HUAWEI’s vision is “to enrich life through communication,” and its mission is to “to focus on our customers’ market challenges and needs by providing excellent communications network solutions and services in order to consistently create maximum value for customers.” HUAWEI focuses on “user perception” and has established a close cooperative relationship with more than 30 telecom operators, industry alliances and institutions, vertical industry leaders, and so on. In order to give full play to the wisdom of group users, HUAWEI set up HUAWEI interactive community on the official website to encourage the majority of participants to participate in the platform exchange. It aims at providing a platform for direct communication with users, collecting users’ opinions and needs, allowing users to define the next product of HUAWEI, discussing and communicating in circles in the form of a community, providing a rich data document, and allowing users to release their opinions on the platform. As a result, HUAWEI can receive a lot of product ideas, but how to select effective product ideas among a large number of product ideas with low value density is an urgent problem to be solved. This paper takes HUAWEI innovation as the background, and assumes that there are several product ideas, which can be evaluated by three types of experts: internal members of enterprise, current users, and potential users, so as to illustrate the decision making process of the proposed method.
In August, 2018, HUAWEI interactive community Managers released a post entitled “What new APP features do you want? Say it, and win M3 panel computer!”. All participants can respond to this post, either by presenting new ideas, or by evaluating existing ones. This post is valid for one month. In the meantime, the enterprise received 45 ideas and needed to rank these ideas for easy selection. In this paper, we take 3 ideas as the representative to introduce the ranking process of ideas. So, the set of ideas can be expressed as , a1= Change the theme color of APP according to the color of cell phone shell, a2= Home service center makes a manageable-sorted look, and a3= Add automatic save draft function.
This paper proposes the following assumptions: (a) there are 300, 500 and 200 individuals in three types of experts, namely, E1= Internal members of the enterprise, E2=Current users, E3= Potential users; (b) there are four criteria c1= Novelty, c2= Acceptability, and c3= Usefulness, c4= Feasibility; (c) the weights of the four criteria are assumed to be 0.17, 0.28, 0.26, 0.29; (d) on criterion c1, the weights of three types of experts are assumed to be 0.2, 0.4, and 0.4; on criterion c2, the weights of three types of experts are assumed to be 0.3, 0.4, and 0.3; On criterion c3, the weights of three types of experts are assumed to be 0.2, 0.5, and 0.3; On criterion c4, the weights of three types of experts are 0.4, 0.3, and 0.3; (e) the evaluation set of grades can be expressed as , θ1=Excellent, and θ2=Non-excellent, which represents that each expert is asked by the question “Is the alternative ak excellent or not on criterion cj?”, and the utility of each grade is set as . In this paper, the evaluation information of the 1000 experts on alternative ak is obtained through simulation, as shown in Table 2.
The numbers in Table 2 indicate the number of individuals in Ei who think alternative ak is excellent, nonexcellent, or uncertain on criterion cj. Combined with Table 2, the evaluation information can be transformed into BPA functions by (2), which are shown in Table 3.
The evaluation value of alternative ak evaluated by experts Ei with respect to criterion cj is shown in Table 3, expressed as . The weights of each criterion and the weights of experts on each criterion are substituted into (3), and the Shafer’s discounting is used to discount , so the BPA functions discounted by weight can be obtained. The discounted BPA functions are shown in Table 4.
According to Dempster’s combination rule, we use (4) to combine the discounted BPA functions . For the first time of combination, we can get the comprehensive evaluation value of experts Ei on all criteria of alternative ak, expressed as . Then, using Dempster’s combination rule to combine again, we can get the overall evaluation value for alternative ak for all experts, expressed as . By substituting in (5), we can get the probability distribution of comprehensive performance at all grades of alternative ak (see the 2-3 columns of Table 5). By substituting grade utility and probability distribution into (6), we can calculate the total utility Uk of alternative ak (see the column of Table 5,). According to the total utility Uk, we can know that the ranking is a3> a1>a2.
In the mass collaboration mode, the large number of ideas and participates leads to the existing methods which are not suitable for solving the problem of product idea evaluation. In this paper, we propose a mass collaboration-driven method for recommending product ideas based on DST. Firstly, the BPA function computing method is introduced by discounting belief degrees with weights to extract the evaluation information of product ideas, which can effectively reflect the facticity of experts’ evaluation. Secondly, Dempster’s combination rule is used to combine the derived BPA functions for two times: the first one is to combine the discounted BPA functions on all criteria with respect to a specified expert and the other is to combine the combined BPA functions for all experts with respect to a specified alternative. Finally, the steps of mass collaboration-driven method for recommending product ideas based on the DST are proposed. An example is proposed to illustrate the recommendation mechanism by the proposed method. The main contribution of this paper is that the DST is innovatively introduced to solve the problem of recommending product ideas in the mass collaboration mode, in which each participant can expediently give real assessments with his/her experiences and their assessments can be reasonably combined with Dempster’s rule. It is worth noting that this paper only proposes the mass collaboration-driven method for recommending product ideas from the static combination perspective but does not consider the dynamic interaction. As a result, studying the dynamic interaction recommendation problem for the product ideas in the mass collaboration mode will be the next research direction.
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
This research was supported by the National Natural Science Foundation of China (NSFC) under Grant Nos. 71874167 and 71462022, the Fundamental Research Funds for the Central Universities under Grant No. 201762026, and the Special Funds of Taishan Scholars Project of Shandong Province under Grant No. tsqn20171205.
- T. M. Amabile, “The social psychology of creativity: A componential conceptualization,” Journal of Personality and Social Psychology, vol. 45, no. 2, pp. 357–376, 1983.
- M. D. Mumford and S. B. Gustafson, “Creativity Syndrome: Integration, Application, and Innovation,” Psychological Bulletin, vol. 103, no. 1, pp. 27–43, 1988.
- M. A. Runco and G. J. Jaeger, “The Standard Definition of Creativity,” Creativity Research Journal, vol. 24, no. 1, pp. 92–96, 2012.
- J.-R. Chou, “An ideation method for generating new product ideas using TRIZ, concept mapping, and fuzzy linguistic evaluation techniques,” Advanced Engineering Informatics, vol. 28, no. 4, pp. 441–454, 2014.
- S. A. Kazmi, M. Naarananoja, and J. K. Wartsila, “Identifying Dissimilarities among Global Teams while Pursuing New Product Idea Generation Practices,” Procedia - Social and Behavioral Sciences, vol. 229, pp. 376–386, 2016.
- M. Banović, A. Krystallis, L. Guerrero, and M. J. Reinders, “Consumers as co-creators of new product ideas: An application of projective and creative research techniques,” Food Research International, vol. 87, pp. 211–223, 2016.
- K. Shroyer, T. Lovins, J. Turns, M. E. Cardella, and C. J. Atman, “Timescales and ideaspace: An examination of idea generation in design practice,” Design Studies, vol. 57, pp. 9–36, 2018.
- T. Gillier, C. Chaffois, M. Belkhouja, Y. Roth, and B. L. Bayus, “The effects of task instructions in crowdsourcing innovative ideas,” Technological Forecasting & Social Change, vol. 134, pp. 35–44, 2018.
- H. Kwon, Y. Park, and Y. Geum, “Toward data-driven idea generation: Application of Wikipedia to morphological analysis,” Technological Forecasting & Social Change, vol. 132, pp. 56–80, 2018.
- M. A. Mirtalaie, O. K. Hussain, E. Chang, and F. K. Hussain, “A decision support framework for identifying novel ideas in new product development from cross-domain analysis,” Information Systems, vol. 69, pp. 59–80, 2017.
- L. M. Steele, G. Johnson, and K. E. Medeiros, “Looking beyond the generation of creative ideas: Confidence in evaluating ideas predicts creative outcomes,” Personality and Individual Differences, vol. 125, pp. 21–29, 2018.
- A. Özaygen and C. Balagué, “Idea evaluation in innovation contest platforms: A network perspective,” Decision Support Systems, vol. 112, pp. 15–22, 2018.
- N. Hao, Y. Ku, M. Liu et al., “Reflection enhances creativity: Beneficial effects of idea evaluation on idea generation,” Brain and Cognition, vol. 103, pp. 30–37, 2016.
- N. Mayseless, J. Aharon-Peretz, and S. Shamay-Tsoory, “Unleashing creativity: The role of left temporoparietal regions in evaluating and inhibiting the generation of creative ideas,” Neuropsychologia, vol. 64, pp. 157–168, 2014.
- S. Hoornaert, M. Ballings, E. C. Malthouse, and D. Van den Poel, “Identifying New Product Ideas: Waiting for the Wisdom of the Crowd or Screening Ideas in Real Time,” Journal of Product Innovation Management, vol. 34, no. 5, pp. 580–597, 2017.
- S. Zeng, J. Gonzalez, and C. Lobato, “The effect of organizational learning and Web 2.0 on innovation,” Management Decision, vol. 53, no. 9, pp. 2060–2072, 2015.
- X.-F. Xu, J. Hao, Y.-R. Deng, and Y. Wang, “Design optimization of resource combination for collaborative logistics network under uncertainty,” Applied Soft Computing, vol. 56, pp. 684–691, 2017.
- X. Xu, W. Zhang, N. Li, and H. Xu, “A bi-level programming model of resource matching for collaborative logistics network in supply uncertainty environment,” Journal of The Franklin Institute, vol. 352, no. 9, pp. 3873–3884, 2015.
- E. F. Rietzschel, B. A. Nijstad, and W. Stroebe, “Productivity is not enough: A comparison of interactive and nominal brainstorming groups on idea generation and selection,” Journal of Experimental Social Psychology, vol. 42, no. 2, pp. 244–251, 2006.
- N. Michinov, E. Jamet, N. Métayer, and B. Le Hénaff, “The eyes of creativity: Impact of social comparison and individual creativity on performance and attention to others' ideas during electronic brainstorming,” Computers in Human Behavior, vol. 42, pp. 57–67, 2015.
- D. M. DeRosa, C. L. Smith, and D. A. Hantula, “The medium matters: Mining the long-promised merit of group interaction in creative idea generation tasks in a meta-analysis of the electronic group brainstorming literature,” Computers in Human Behavior, vol. 23, no. 3, pp. 1549–1581, 2007.
- J. Füller, G. Jawecki, and H. Mühlbacher, “Innovation creation by online basketball communities,” Journal of Business Research, vol. 60, no. 1, pp. 60–71, 2006.
- P.-A. Verhaegen, D. Vandevenne, J. Peeters, and J. R. Duflou, “Refinements to the variety metric for idea evaluation,” Design Studies, vol. 34, no. 2, pp. 243–263, 2013.
- J. A. Plucker, R. A. Beghetto, and G. T. Dow, “Why Isn't Creativity More Important to Educational Psychologists Potentials, Pitfalls, and Future Directions in Creativity Research,” Journal of Educational Psychology, vol. 39, no. 2, pp. 83–96, 2004.
- S. L. Chan, W. H. Ip, and C. K. Kwong, “Closing the loop between design and market for new product idea screening decisions,” Expert Systems with Applications, vol. 38, no. 6, pp. 7729–7737, 2011.
- E. F. Rietzschel, B. A. Nijstad, and W. Stroebe, “The selection of creative ideas after individual idea generation: Choosing between creativity and impact,” British Journal of Psychology, vol. 101, no. 1, pp. 47–68, 2010.
- J. Diedrich, M. Benedek, E. Jauk, and A. C. Neubauer, “Are creative ideas novel and useful?” Psychology of Aesthetics, Creativity, and the Arts, vol. 9, no. 1, pp. 35–40, 2015.
- S. Schwarz and F. Bodendorf, “Attributive Idea Evaluation,” International Journal of Knowledge-Based Organizations, vol. 2, no. 1, pp. 77–91, 2012.
- S. Hart, E. J. Hultink, N. Tzokas, and H. R. Commandeur, “Industrial companies' evaluation criteria in new product development gates,” Journal of Product Innovation Management, vol. 20, no. 1, pp. 22–36, 2003.
- K. R. Maccrimmon and C. Wagner, “Stimulating ideas through creativity software,” Management Science, vol. 40, no. 11, pp. 1514–1532, 1994.
- P. Carbonell-Foulquié, J. L. Munuera-Alemán, and A. I. Rddríguez-Escudero, “Criteria employed for go/no-go decisions when developing successful highly innovative products,” Industrial Marketing Management, vol. 33, no. 4, pp. 307–316, 2004.
- D. Dean, J. Hender, T. Rodgers, and E. Santanen, “Identifying Quality, Novel, and Creative Ideas: Constructs and Scales for Idea Evaluation,” Journal of the Association for Information Systems, vol. 7, no. 10, pp. 646–699, 2006.
- E. Hietikko, “A Two Dimensional Matrix Presentation for Idea Evaluation,” International Journal of Engineering Research and Applications, vol. 3, pp. 1673–1676, 2013.
- K. Eling, F. Langerak, and A. Griffin, “The Performance Effects of Combining Rationality and Intuition in Making Early New Product Idea Evaluation Decisions,” Creativity and Innovation Management, vol. 24, no. 3, pp. 464–477, 2015.
- G. Lauto and F. Valentin, “How preference markets assist new product idea screening,” Industrial Management & Data Systems, vol. 116, no. 3, pp. 603–619, 2016.
- M. Ferioli, E. Dekoninck, S. Culley, B. Roussel, and J. Renaud, “Understanding the rapid evaluation of innovative ideas in the early stages of design,” International Journal of Product Development, vol. 12, no. 1, pp. 67–83, 2010.
- E. Gutierrez, I. Kihlander, and J. Eriksson, “What's a good idea? Understanding evaluation and selection of new product ideas,” in Proceedings of the International Conference on Engineering Design Stanford, pp. 121–132, USA, 2009.
- G. S. Mitra Thakur, R. Bhattacharyya, and S. Sarkar Mondal, “Stock portfolio selection using Dempster–Shafer evidence theory,” Journal of King Saud University - Computer and Information Sciences, vol. 30, no. 2, pp. 223–235, 2018.
- C. Zhang, Y. Hu, F. T. Chan, R. Sadiq, and Y. Deng, “A new method to determine basic probability assignment using core samples,” Knowledge-Based Systems, vol. 69, pp. 140–149, 2014.
- S. Zeng and Y. Xiao, “A method based on topsis and distance measures for hesitant fuzzy multiple attribute decision making,” Technological and Economic Development of Economy, vol. 24, no. 3, pp. 969–983, 2018.
- S. Zeng, Z. Mu, and T. Baležentis, “A novel aggregation method for Pythagorean fuzzy multiple attribute group decision making,” International Journal of Intelligent Systems, vol. 33, no. 3, pp. 573–585, 2018.
- S. Zeng, “Pythagorean fuzzy multiattribute group decision making with probabilistic information and OWA approach,” International Journal of Intelligent Systems, vol. 32, no. 11, pp. 1136–1150, 2017.
- Y. Du, Y. Wang, and M. Qin, “New evidential reasoning rule with both weight and reliability for evidence combination,” Computers & Industrial Engineering, vol. 124, pp. 493–508, 2018.
- X. Deng, Q. Liu, Y. Deng, and S. Mahadevan, “An improved method to construct basic probability assignment based on the confusion matrix for classification problem,” Information Sciences, vol. 340-341, pp. 250–261, 2016.
- Y.-W. Du and W.-M. Xu, “Multiattribute group decision making based on interval-valued intuitionistic fuzzy sets and analytically evidential reasoning methodology,” Journal of Intelligent & Fuzzy Systems: Applications in Engineering and Technology, vol. 33, no. 5, pp. 2953–2960, 2017.
- Y.-W. Du and Y.-M. Wang, “Evidence combination rule with contrary support in the evidential reasoning approach,” Expert Systems with Applications, vol. 88, pp. 193–204, 2017.
- Y. Du, N. Yang, and J. Ning, “IFS/ER-based large-scale multiattribute group decision- making method by considering expert knowledge structure,” Knowledge-Based Systems, 2018.
- A. Gnanavelbabu and P. Arunagiri, “Ranking of MUDA using AHP and Fuzzy AHP algorithm,” Materials Today: Proceedings, vol. 5, no. 5, pp. 13406–13412, 2018.
Copyright © 2018 Yuan-Wei Du et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.