Abstract
Under the smart engineering system (SES), there is a huge demand for evaluating the efficacy of a large-scale networked intelligent perception system (IPS). Considering the large-scale, distributed, and networked system characteristics and perception task demands, this paper proposes a conceptual system for IPS efficacy evaluation and, on this basis, designs the architecture of the efficacy evaluation system. A networked IPS model is constructed based on domain ontology, an index system is quickly established for efficacy evaluation, the evaluation methods are assembled automatically, and adaptive real-time organization strategies are generated for networked perception based on efficacy estimate. After exploring these key technologies, a prototype system is created for the service-oriented integrated efficacy evaluation platform and used to verify and integrate research results. The research provides support for the efficacy evaluation theories and methods of large-scale networked IPS.
1. Introduction
In recent years, many advanced information technologies have emerged and evolved quickly, including the Internet of things (IoT), artificial intelligence (AI), cloud computing, and big data. In the meantime, many complex issues have surfaced in urban public governance, public safety, industrial manufacturing, agricultural production, and ocean, such as energy, environmental protection, traffic and tourism, emergency management, production-service process, disaster prevention, and mitigation [1–6]. To mitigate or solve various problems, advanced information technologies are combined with physical entities and other resource elements through overall planning and optimal allocation. The optimization of advanced information technology is a very cutting-edge research problem. Many researchers have done research in this field, such as the related theories, methods, and technologies of three-way decision-making, and have made a lot of achievements in this problem. The synergy between them gives birth to smart engineering systems (SESs) of different modes, which improve the level of smart services. Typical SESs include smart cities, intelligent transportation, smart factories, intelligent agriculture, and wisdom oceans [7–13]. A three-way decision has also played a great role in some fields [14–16].
The current research on SES construction mostly focuses on the top-level design and decision application of the system, as well as the key technologies like information processing, storage, transmission, and perception [17–19]. However, there are not many in-depth studies on the efficacy of the intelligent perception system (IPS), which is based on performance and function. Few scholars have considered if the perception ability of IPS could satisfy the demand of perception tasks. There are still many indescribable difficulties in the mathematical description of intelligent sensing systems. We cannot use a very accurate formula to describe the problem, which leads to the accuracy of the intelligent sensing system.
During SES construction, the large-scale complex networked IPS with various sensors and multiple functions faces severe challenges, such as diverse perception tasks and objects, high requirement on perception ability, and complicated perception environment. The IPS must measure, differentiate between, and track many objects accurately in a wide range. The huge amount of data perceived by the system carries obvious features of the big data, namely, complex data relationships, various types, fast storage/reading, and low application value density. IPS is characterized by a large amount of data and too complex information. It must be processed twice before it can be used as computable data to participate in efficiency evaluation.
How to evaluate the efficacy of the large-scale networked IPS under the SES becomes an unavoidable problem with theoretical, methodological, and technical challenges, because this is hard to format the question, and the theoretical, methodological, and technical challenges also have some need for research. The evaluation of perception efficacy is the guide for the construction of the large-scale networked IPS and the theoretical basis of system capacity measurement, because it exists throughout the lifecycle of such systems from planning and demonstration, design, research, and development (R&D) to testing and use, associates closely with specific conditions and tasks, and takes root on system functions and performance [20, 21].
Therefore, it is a fundamental and practical issue to explore the theories and techniques for the large-scale networked IPS efficacy evaluation under the SES. To construct high-quality large-scale networked IPS, this paper systematically investigates every aspect of efficacy evaluation environment, ranging from architecture, conceptual system, model base, index system, and method base to integrated platform construction.
2. Research Roadmap
Our research is based on relevant theories, technologies, and methods of system science and complex systems engineering, computer science, information science, IoT, cloud computing, mobile Internet, big data, service-oriented technology, and evaluation theories and methods. Driven by the demand for efficacy evaluation of the large-scale networked IPS, this paper designs the architecture of efficacy evaluation environment based on the conceptual system of efficacy evaluation and makes breakthroughs on such key techniques as the modeling, index system, and evaluation method of the IPS, as well as adaptive generation of perception strategy based on efficacy estimate. In this way, an integrated prototype system is established for efficacy evaluation to verify and integrate the research results. The technical roadmap of the research is shown in Figure 1.
3. Methodology
3.1. Conceptual System of the Large-Scale Networked IPS Efficacy Evaluation
3.1.1. Conceptual System of Efficacy Evaluation
There is not yet a unified definition for the connotation or denotation of efficacy. But it is generally agreed that the core of efficacy lies in three elements, namely, condition, time, and task, as well as their associations. The term efficacy was originally proposed for evaluating the demand for weapon systems. Later, the concept gradually extends from military application to the social economy system.
From the perspective of methodology, efficacy evaluation was initially implemented through probability statistics and military operation research and went through such stages as the coupling between qualitative and quantitative evaluations and simulation-based system evaluation. Currently, efficacy evaluation is service-oriented and big data-based, involving human-machine integration, which lays the theoretical and methodical bases for the large-scale networked IPS efficacy evaluation under the SES.
Considering the IPS performance, functions, and use efficacy of the large-scale networked IPS, the efficacy of the system is conceptualized as the comprehensive measurement of the ability to complete a specific task under given conditions, time, and environment, contraposing the task demand of the large-scale networked IPS and according to the capability bases like technical performance, functions, and application scenarios. The system efficacy can be calculated by where is the measure of system performance, i.e., the capacity of the system under 100% utilization of system availability and ability; is the measure of system availability, i.e., the degree of the system completing its expected functions during the application; and is the measure of system applicability, i.e., the demonstration of system capability in executing the perception task. , , and are independent of each other. These measures can be characterized by mathematical methods such as the index method and probability method. The index method and the probability method can be easy to describe the problem and get results as soon as possible, but the index method is not good at complex problems in the cost.
Based on different phases of the lifecycle of the networked IPS including the demonstration, design, R&D, testing, and use, the efficacy evaluation essentially needs to adopt unified, standard, and universal measures to evaluate each subefficacy and combine the series of subefficacy evaluations into a composite efficacy by a certain logical rule or from the bottom up.
For a given perceptual task, the composite efficacy can be calculated from several interconnected and mutually constrained efficacy elements, for example, analytic hierarchy process (AHP), fuzzy comprehensive evaluation method, gray whitening weight function method, and TOPSIS method, including the networked perception model, characterization index, and assessment and environment methods. Figure 2 shows the relationship between the efficacy elements, and the task is very complex, and each element can take charge of different tasks. Hence, that can be explained according to different questions.
The connotation representation system for the networked IPS efficacy is tailorable. It can be dynamically adjusted according to the specific phase and objective. Efficacy indices are generally used in the actual application phase of the networked IPS. The actual efficacy of the system needs to be tested and evaluated under complex scenarios [22]. Considering the principles and strategies for hierarchical decomposition of complex systems, i.e., wholeness, hierarchy, and completeness, the networked IPS efficacy is decomposed into a perception resource layer, a technical performance layer, an application function layer, a capability layer, a task layer, and a perception task requirement layer (Figure 3).
The understanding and demand of system efficacy evaluation vary with the task boundaries, environmental conditions, and the focuses and perspectives of subjects. Therefore, the criteria, method, and index system must be developed properly for system efficacy evaluation.
3.1.2. Composite Efficacy Index Model for Networked IPS
Drawing on the theory of index method, the composite efficacy of the networked IPS can be modeled by where is the composite efficacy index, which is a dimensionless relative value; are subefficacy indices of system performance, availability, and applicability, respectively; , , and are the weights of subefficacy indices (), which depend on the specific purpose and change with task conditions; is the task conditions of the whole lifecycle, i.e., planning and demonstration, design, R&D, testing, and use; are the actual system performance, availability, and applicability, respectively; and are the ideal system performance, availability, and applicability, respectively.
3.2. Service-Oriented Architecture and Flow of Networked IPS Efficacy Evaluation
System efficacy evaluation involves multiple interconnected elements: effectiveness evaluation scenario, model system, index system, methodology system data system, and display control. Among them, the sensor model and the networked IPS model should be constructed based on domain ontology. The effectiveness evaluation scenarios cover task, objectives, and environment. In addition, the networked IPS model supports the system data service; the method base provides on-demand restriction of evaluation methods. Figure 4 shows the set of elements in the networked IPS efficacy evaluation. Obviously, the efficacy evaluation environment is compatible with the hardware-in-the-loop state of sensors, i.e., in support of semiphysical simulation or full physical simulation.
Based on the theories and methods of complex systems, efficacy evaluation theories and techniques, web service technical framework, and ontologies and semantic network theories and methods, this paper designs a service-oriented integrated efficacy evaluation system, which consists of four layers: a resource layer, a service layer, an application support layer, and an application layer (Figure 5).
The resource layer provides the platform with basic supports like data, models, methods, and instances. This layer mainly drives all the resources in the system and visualizes them for object management. On the resource layer, the various bases could be distributed or concentrated. But all of them are centered on services, providing support for the implementation of web services.
The service layer, which is built on the resource layer, integrates the basic technical means of efficacy evaluation, offers the management tools and interface components for resources (e.g., data, models, algorithms, and knowledge), and encapsulates and calls businesses.
The application support layer encompasses the basic functional components and modules required for constructing the efficacy evaluation system and provides the application layer with consistent interfaces, thereby simplifying service development. The component technology of this layer helps to manage the various business components on the superior layer and dynamically reconfigure any business components.
According to the requirements of the evaluation task, the application layer utilizes application support tools and adopts the task-based management strategy. This layer is mainly responsible for decomposing a task into the functional combination of several business components and calls the dedicated resources and services of the system to generate actual efficacy evaluation modules.
The evaluation task is executed in the following process: First, the system provides the user with a service interface, calls different services or service portfolios as per user demand for evaluation, and optimizes the called service portfolio to generate the job to be executed. Once generated, the job is decomposed, organized, scheduled, and controlled by our architecture, and the final result is fed back to the user. Under the architecture, the job scheduling module receives all the generated jobs and decomposes each job into multiple parallel subjobs via the parallel job controller. Then, the intelligent scheduling module assigns one or more virtual resources on the platform to each subjob and controls the execution of each subjob. The entire flow is illustrated in Figure 6.
3.3. Sensor and Networked IPS Modeling Based on Domain Ontology
Domain ontology, an important type of ontology, can provide unique concepts and their relations, the basic principles, and the activities in specific fields. The technical realization of ontologies relies on advanced ontology languages [23]. Currently, several ontology languages are available, such as XML-based ontology exchange language (XOL) and the web ontology language (OWL) derived from OIL and DAML+OIL (XML: extensible markup language; DAML: DARPA agent markup language; DARPA: US Defense Advanced Research Projects Agency; OIL: ontology inference layer).
The field ontology of a single sensor should be built based on clear domain concepts and their relationships. One of the key technical paths for building a macro language library is to establish a standard description of the sensor model by domain ontology.
The single sensor model can be formally depicted as where Id, Name, Category, Function, Inputs, Outputs, Task, and Qos are identifier, name, sensor type, functional description, inputs, outputs, task, and quality of service, respectively, and is the system scale. Multiple can be combined by the actual logic of time and space and formed into a stereo regional ocean perception system.
Figure 7 shows the domain ontology of the networked IPS, which describes the concepts and their logical relationships covered by the system. The domain ontology of the networked IPS lays the basis for sharing the knowledge of accurate model depiction and facilitates the unified control of the relevant data.
With the networked IPS modeling as the goal, the knowledge in relevant fields is acquired, described, and represented by the features of domain ontology of sensors: conceptualized, clear, formalized, and shared. The modeling involves multiple construction techniques for the model chain, namely, standard description of sensor model, model verification, model combination, model query, model matching, model management, and model generation.
The networked IPS model can be formally depicted as where Id, Name, Number, Function, Coverage, Accuracy, Task, and Qos are identifier, name, number of sensors, functional description, coverage, data accuracy, task, and quality of service, respectively.
Networked control, as the key and focus of the networked IPS modeling, is established based on the following technologies such as sensor networking, information distribution, information fusion, state/situation awareness, resource scheduling, and network control. The elements of networked control include networking objects, structure, objectives, means, and strategies. The state/situation awareness modeling of the networked IPS mainly requires the description framework and formal modeling method of state/situation awareness knowledge and the knowledge characterization and reasoning technologies for state/situation awareness elements.
The various problems in networked perception, namely, information processing mechanism and flow, data recording and analysis, and system index evaluation, can be solved by accessing multisensor signals and data, functional reorganization, time-space integration, networked control, and networked data processing. As shown in Figure 8, networked control modeling is aimed at establishing the following models: functional reorganization model, multisensor control strategy model, perceptual task model, control model, interface module, the first mock exam model, information fusion processing model, comprehensive situation model, data recording and analysis model, evaluation model, etc.
The model query mainly uses semantic technologies, such as model content analysis/interpretation, knowledge extraction/expression/learning, and reasoning mechanism, to match the information in model resources with the user’s demand for model information, with the aim of better satisfying the query demand.
During the modeling of sensors and the networked IPS based on domain ontology, multiple roles are generally involved including perception business modeling personnel, model development personnel, model management personnel, flow planning personnel, and users.
3.4. Rapid Construction of Indices, Automatic Assembly of Methods, and Parameter Acquisition
The EIS for networked IPS efficacy is the basis for depicting and characterizing the attributes of networked perception. Following the principles of building a scientific, complete, and measurable EIS, the efficacy evaluation is divided into performance evaluation, function evaluation, and efficacy evaluation (associated with specific tasks), in view of the different roles of efficacy evaluation in different stages of the networked IPS construction. The EIS of networked IPS efficacy covers index value measurement, index weighting, sensitivity analysis, index evaluation, parameter acquisition, etc.
Currently, efficacy is generally evaluated qualitatively, quantitatively, or through both qualitative and quantitative approaches. The mainstream qualitative evaluation method is expert consultation, while the popular quantitative evaluation methods are system effectiveness analysis (SEA) [24], test statistics, and information entropy method. The combination of qualitative and quantitative approaches has been adopted more frequently than qualitative evaluation or quantitative evaluation. The combined approaches include the analytic hierarchy process (AHP), fuzzy analysis, multiattribute decision-making, gray evaluation, and cloud centroid method.
Each evaluation method applies to a specific range of issues. Since a single method has limited applicability, it is important to automatically assemble methods to improve evaluation efficiency and quality. Which method to choose depends on the evaluation task, given conditions, and distribution of efficacy parameters. This is the only way to obtain evaluation results that are satisfactory, scientific, and practical.
For the established EIS of networked IPS efficacy, the data could be collected in real time or not in real time during semiphysical simulation or full physical simulation. The data storage system would save multisource system data into the database and offer data storage, sorting, calling, query, and screening. The preprocessing of evaluation data is designed around the EIS. By removing abnormalities, redundancies, and noises and filling the missing terms, the quality of the collected system data reaches the requirement of the evaluation system. During data processing, the original data are fused, and the precise values of the data are obtained through time-space unification, correlation analysis, and cluster analysis. The processed data are ready to be called by the solving module of the evaluation model.
3.5. Real-Time Generation of Networked IPS Organization Strategy Based on Efficacy Estimate
According to the efficacy estimates of the network IPS obtained from different perception tasks, ] perception efficacy and organization strategy are optimized iteratively, such that the most feasible networked perception strategy is introduced to the network intelligent perception strategy plan base.
During networked perception, the application effects of networked perception strategy are recorded in real time, perception efficacy and organization strategy are optimized iteratively, and the autonomous learning and automatic adaptation technologies are supplemented continuously. In this way, the networked perception strategy is improved, and more perception knowledge and empirical schemes are saved in the strategy plan base, providing an abundance of strategies for the networked IPS in different scenarios. Then, it is possible to generate perception strategies for different perception tasks (Figure 9).
4. Design Service-Oriented Efficacy Evaluation Platform
The system design is guided by perception demand, top-level design, comprehensive integration, domain knowledge, and real-world situation. According to the proposed novel architecture for efficacy evaluation, the findings of the networked IPS model base, index system, and method base are combined to build a service-oriented prototype platform for integrated system efficacy evaluation. The prototype platform is adopted for the technical realization, verification, and optimization of the findings above to solve the technical problems in real R&D, thereby enhancing the perception ability of the large-scale networked IPS in different scenarios.
Referring to the overall structure of mature platforms, the components of the service-oriented prototype platform for integrated system efficacy evaluation are designed from the perspective of system engineering. The functional modules of the platform include business analysis tools, service development environment, management and maintenance tools, process design tools, service registry, and process execution engine (Figure 10).
Figure 11 shows the regional marine fishery environment IPS under wisdom ocean. The system is aimed at collecting observations of marine objects and environment by using multiple instruments such as radar, photoelectric sensor, and automatic identification system (AIS). The main technical indices of the system include the multigranular stereo perception ability of marine environment, objects, and activities within 40 km, which adopt information from different sensors, and different information is transformed into stranded format data, which can be used to calculate the ability to receive and process the information from multisource sensors: 5 radars, 6 photoelectric sensors, and 5 AISs.
Based on our results on index design, method assembly, and efficacy prediction, this paper chooses information perception capacity, information transmission capacity, information storage and processing capacity, decision-making support capacity, information security capacity, and remote operation and maintenance capacity as primary evaluation indices and breaks them down into secondary indices (Figure 12). On this basis, a prototype platform of service-oriented integrated efficacy evaluation is developed to verify our research findings (Figure 13).
The research results can apply to the overall demonstration, design, joint debugging, and efficacy evaluation of the large-scale networked IPSs under the SES, such as smart city IPS and wisdom ocean IPS.
5. Conclusions
This paper mainly explores the efficacy evaluation of the large-scale networked IPS under the SES. The efficacy evaluation research starts from and is supported by the conceptual system, which mainly consists of the connotation or denotation, meaning, flow, criteria, and environment of efficacy evaluation. The evaluation architecture is studied to guide the top-level design of efficacy evaluation, so as to satisfy the demand for efficacy evaluation of the large-scale networked IPS under the SES. The model base, as the foundation and core of efficacy evaluation environment, depicts and illustrates the attributes, features, and laws of sensors and the networked IPS. Our model base mainly includes the single-sensor model and the networked IPS model, both of which are developed based on domain ontology, as well as model organization and management technology. In addition, the keys of efficacy evaluation environment, i.e., EIS and evaluation methods of efficacy, are investigated in detail. On this basis, the various elements of the efficiency evaluation system are synthesized into a service-oriented integrated physical efficacy evaluation platform, which encompasses such components as task scenes, model base, index base, method base, and display control. The physical platform provides an important tool to verify and realize our efficacy evaluation strategy, enhance the capacity of the networked IPS, and support the construction of such a system. The future research will further verify our system through semi- and full physical simulations and provide a computing environment in support of the efficacy evaluation of the large-scale networked IPS under the SES.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.