Abstract
With the rapid developments of wireless communication and increasing number of connected vehicles, Vehicular Ad Hoc Networks (VANETs) enable cyberinteractions in the physical transportation system. Future networks require real-time control capability to support delay-sensitive application such as connected autonomous vehicles. In recent years, fog computing becomes an emerging technology to deal with the insufficiency in traditional cloud computing. In this paper, a fog-based distributed network control design is proposed toward connected and automated vehicle application. The proposed architecture combines VANETs with the new fog paradigm to enhance the connectivity and collaboration among distributed vehicles. A case study of connected cruise control (CCC) is introduced to demonstrate the efficiency of the proposed architecture and control design. Finally, we discuss some future research directions and open issues to be addressed.
1. Introduction
Nowadays, networking technology has become the key element in various application areas with the increasing number of connected devices. Networked control systems (NCSs), which consist of spatially distributed sensors, actuators, and controllers in the close-loop communication network, are the powerful enablers to meet the demands for both information dissemination and data analysis [1, 2]. This characteristic of distributed architecture design makes NCSs easier to implement, maintain, and share information resources, meeting the requirements of future connected system [3]. With the development of wireless communication technology, the transmission rate and reliability of information exchange through wireless link gradually meet the needs of future NCSs. In particular, the use of wireless communication eliminates unnecessary wiring among system components and can be upgraded easily [4]. Therefore, in wireless networked control systems (WNCS), such as wireless sensor networks and wireless multihop networks, more flexible system architecture design, more easily resource utilization, and increased safety can be achieved [5].
However, the usage of wireless networks in the data transmission introduces message delay and packet loss which cannot meet the emerging requirements for real-time control applications [6]. For example, higher requirements are emerged such as sufficiently reliability and low latency. Previous cloud-based architecture has a high response delay and is not suitable for delay-sensitive applications. Delays may cause the performance degradation and even more serious safety issues. In addition, for real-time control applications, the requirements of transmission reliability and stability introduce further research challenges. The complex network structures, large network scales, flexible communication topologies, and variable communication environments can also destabilize a WNCS and degrade system performance. In addition, traditional single-controller-based closed-loop control architecture is difficult to meet the needs of real-time control applications [7–10]. In order to improve overall system performance and efficiency, multicontroller design is adopted with the advantages in modularity, scalability, and robustness. With the growing number of connected vehicles require more communication, computing, and storage resources in the network.
Considering the current situation of WNCS, traditional cloud-based system architectures have pronounced limitations with extracommunication cost and waste on networking resources. Therefore, it is necessary to introduce edge computing and use cooperative features of wireless networks to optimize system performance and stability [11, 12]. In this case, all underlying distributed controllers are able to share information within a certain neighboring area and cooperate to make decisions alleviating computation burden and design complexity [13]. Fog computing expands the cloud computing paradigm, which is seen as a promising technique to spread the computing resources from the cloud computing servers to the edge in order to balance the load [14]. The integration of cloud, fog, and edge nodes at the lowest perception layer takes the advantages of low latency and flexible network topology [15]. Among all the service types, the WNCS based on fog computing provides a new solution for the development of real-time control applications. Fog computing makes it available for the central data processing and storage on-line at the cloud and distributed caching and computing resources at the edge [16].
In recent years, with the increasing number of vehicles on the road, intelligentization and networking of vehicles have been widely concerned by automobile manufacturers and research institutions [17, 18]. With the new round of technological revolution from the Internet to the Internet of Things (IoT) [19], the construction of Intelligent Transportation Systems (ITSs) has become a tendency to deal with the serious problems of traffic safety and road congestion [20–22]. In order to achieve safe, energy-saving, and efficient control, the ability of flexibly information sharing, environmental awareness, intelligent decision-making, coordinated control, and execution is needed among vehicles [23]. Applying WNCS technologies to the intelligent control of vehicles can strengthen the information exchange, make full use of system resources, and promote the realization of ITS. Since fog platform can provide computing, storing, and networking services, it is considered as a promising solution in the connected vehicles scenario [24–26].
In the safety aspect, the collision avoidance (CA) system plays a vital role in preventing the driver-caused accidents [27, 28]. Especially in the cruise control problem, applying wireless communication among vehicles, rather than completely relying on sensors equipped on each vehicle, has the potential to provide better system performance [29, 30]. In order to deal with above challenges, more flexible design for the vehicular platoon becomes an emerging trend. Connected cruise control (CCC), as a potential solution, has been proposed to hold a smooth traffic flow with flexible connectivity structure and communication topology [31, 32].
Recently, advanced driver assistance systems (ADAS) have attracted widespread attention in order to improve the flow rate and safety of the traffic flow [17, 33]. In the cruise control problem, the CA technology plays a vital role in preventing traffic accidents, and the use of wireless communication technology to replace sensors directly equipped between traditional vehicles can further provide better system performance [28, 30]. Therefore, the design of more flexible vehicle platoon system based on wireless communication technology has become as an emerging trend [27, 29]. The CCC system, guarantee a stable traffic flow through flexible communication topologies and connection structures, has become a potential solution for future development [31, 32]. In this paper, we investigate the CCC system as a case study by implementing the proposed architecture. The main contributions of this paper are listed as follows. (i)We propose a novel fog-based distributed network architecture toward real-time control applications. The critical system variables involved within the communication and control system that may influence system performances are analyzed(ii)We formulate the CCC problem as a case study of the proposed fog-based distributed network architecture, and the control scheme is analyzed thoroughly with the cloud, fog, and connected vehicles. Then, a linear quadratic optimization problem is formulated to regulate vehicle's longitudinal motion when the time delay is considered(iii)Based on the fog control nodes, the whole vehicle system can successfully split into multiple distributed vehicle platoons that each has an automated vehicle at the tail. Then, a two-step control scheme is proposed to solve the optimal CCC problem. (i) At the offline step, a backward recursion approach is used to derive the optimal control gain. (ii) At the online step, the optimal control strategy can be real-time calculated based on the current system states and local cached information(iv)In a distributed control scheme, the optimal CCC strategy can be derived as a linear function of current platoon state information and the last control strategy
The remainder of this article is organized as follows. We propose a novel fog-based distributed network architecture toward real-time application in the following section. Next, a practical case study is provided to validate the high efficiency of the proposed architecture. Afterward, some discussions are presented in future research directions and open issues. Finally, the conclusion is made.
2. Proposed Architecture
With the recent revolution in wireless networking applications, such as IoT, Tactile Internet [34], and Vehicular Ad Hoc Networks (VANETs) [35], fog computing becomes a promising enabler to carry out a massive amount of communication, computation, storage, and networking services between edge devices and traditional center cloud servers [36]. More specifically, fog computing is more suitable for real-time control applications with the advantages of low latency, location awareness, wider geographical distribution, wider range of mobility, suitable for more nodes, and support for network heterogeneity [37].
2.1. Overall Architecture Design
In this paper, we propose a fog-based distributed network architecture toward real-time control applications. Figure 1 depicts the overall system model consisting of three logical layers, the cloud, distributed fog, and edge smart devices. The cloud can serve multiple fog nodes, and a fog node can serve several dozens of edge nodes. The edge nodes can be sensors, embedded smart devices, mobile terminals, and vehicles. Each fog node can not only act as an independent service provider but also work together with other fog nodes to achieve collaborative control and services to edge nodes. The fog layer can be further classified into two categories, determined by their capability. Normal fog nodes are governed by the more powerful nodes. In the proposed architecture, the fog nodes have the ability of communicating, processing, and caching to complete two kinds of functions. On the one hand, the fog nodes realize the control of the edge local network, such as resource allocation and transmission path planning [38]. On the other hand, the fog nodes act as remote controllers, generate control strategies in real time according to the state information of the controlled platform and network characteristics to implement close-loop feedback control. Therefore, the fog nodes can both govern the wireless network as well as the real-time control of the controlled platform, which can effectively avoid entirely relying on the cloud. This scheme relieves the problem of high latency, and the usage of distributed cooperative control alleviates the transmission instability caused by wireless channels in traditional single-controller systems. The above fog-based system architectures can be applied to delay-sensitive systems in smart grid, vehicular network, wireless sensor and actuator network, and smart medical systems.
Figure 2(a) shows the typical WNCS model with the controlled plant, sensors, actuators, controllers, and relay nodes. Compared with the architecture in Figure 1, the plant along with sensors and actuators is seen as the edge node; the routers and controllers are seen as the fog nodes. The plant states are sampled at periodic intervals by the sensors capable of wireless communication. In order to realize the closed-loop control, each packet consisting of the plant states is transmitted to the controller over the wireless link; the controller computes the control signal based on the sensor measurements, then sent it to the actuator to ensure desirable dynamic and steady-state response.
(a)
(b)
2.2. Critical System Variables
In the proposed system, there are several critical variables involved with the combination of control system and wireless communication as shown in Figure 2(b). The practical physical systems are continuous-time systems where typical WNCS considers a discrete-time plant model with digital controllers. In the WNCS, periodic sampling is widely used, and shorter sampling period increases the network traffic causing higher message delay and message loss probability.
In particular, the network-induced delays mainly consist of the communication delays among sensors, controllers, and actuators, varying with communication channel quality. With different applications using various communication networks, the delays may possess diverse characteristics such as random or deterministic, constant, or time-varying. The uncertainty of delays significantly degrades the system performance. In addition, inevitably channel fading or shadowing in the dynamic wireless communication increases the packet loss probability, and retransmission is unfitted in real-time applications. In this case, seamless channel and bandwidth managements are required to optimize the network performance. In WNCS, the communication channel is often seen as an uncertain linear system with additive Gaussian noise. In order to design control strategies with partial state observation, estimation of the unknown variables based on measurements observed over time can be used.
In the proposed architecture, novel design is needed to optimize the system performance of control and communication. Since the raw sensor data in local networks are converged into the fog nodes with unavoidable invalid data, the fog nodes are responsible of filtering and temporarily storing the data. Fog nodes are also capable of the management of the local network with resource allocation, transmission path planning, and congestion control. In addition, the dynamically changes of nodes result in the time-varying network topology. In this case, it is difficult to maintain the system stability with single centralized controller [8]. In the modern control system, multiple distributed controller collaboration can enhance the control performance and provide better system stability.
3. Case Study: Connected Cruise Control
Today’s automakers, Internet and leading technology companies are striving to develop innovative technologies to provide a fully integrated and highly intelligent vehicle experience. Electric and self-driving cars, the advanced state of artificial intelligence (AI) technology, and connectivity through wireless communication are revolutionizing the future of mobility [39, 40]. Connected vehicles are the emerging tendency in automobile manufacturing industries providing advanced quality of service (QoS) and attractive performance enhancements. The VANET has been widely perceived as a promising concept for the realization of ITS with both safety and efficiency promotion. Vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications are implemented in VANETs to support road safety, navigation, entertainments, and other application services.
In this section, we further investigate the system architecture toward real-time autonomous driving applications. In particular, to fulfill low latency requirements, traditional cloud-based network architecture may be insufficient with the growing number of vehicles. Based on the fog-based WNCS architecture shown in Figure 1, Figure 3 presents its application in the vehicular network scenario. In this architecture, vehicles are seen as edge devices equipped with sensors and On Board Units (OBUs). The perception of real-time traffic conditions is fulfilled by roadside sensors and vehicle’s onboard sensors. Vehicles are capable of wireless communication broadcasting their kinematic data and transportation information in the local network. Each vehicle plays the role of packet sender, receiver, or even router within their reach via wireless medium. Rather than moving at random, vehicles tend to move in an organized fashion with the slowly varying network topology. RSUs and RSU controllers (RSUCs) are seen as fog nodes to provide more powerful packet transmission and computing functions as well as storage capabilities in the network in a timely, highly efficient, and coordinated way. RSUs are managed by the RSUC which has more resources for computing, storage, and communication through Internet to the cloud. The traditional core cloud facilitates the large-scale data processing and storage on-line and provides global view with big data and AI technology.
3.1. Problem Formulation and Control Design
As one of the widely used applications of Automatic Driving Assistance Systems, cruise control systems can effectively improve road traffic efficiency, reduce traffic accidents, improve vehicle fuel economy, and promote the realization of the ITS [41]. The CCC system, as an integration of the sensor-based cruise control system and wireless communication, can make full use of system resources to effectively avoid collision along with smooth drive [31]. Figure 4 depicts the considered scenario where vehicles generally move at a relatively constant speed, such as following a paved highway, and can be controlled as a whole platoon. The vehicular platoon consists of two kinds of vehicles: the human-driven vehicles and the connected and automated vehicles (CAVs). The CAVs are able to communicate with each other within a designated area through V2V communication and communicate with RSUs through V2I communication [20, 42]. The system is a location-based distributed control system, and the connectivity structure is basically stable or changing slowly. The local wireless network includes a RSU and vehicles governed by the RSU, and the transmission path through V2V and V2I is planned dynamically in RSUs to collaborate the resources and balance the load. The close-loop control of the CAV is realized with the collaboration of the cloud, fog, and vehicles. The cloud can provide big data processing and storage capability which is connected through the Internet. The fog, namely, the RSUs and RSUCs, brings powerful communicating and computing capabilities to the connected vehicles via V2I communication. The RSUs are responsible for local information processing and filtering, thereby effectively relieving the workload on the cloud, making it more suitable for real-time control applications. The large amount of moving vehicles is capable of sensing the real-time traffic information. In particular, at each sampling interval, the state information of all vehicles in the platoon is sent to the RSU. Then, the vehicle platoon model is formulated, and the optimal control strategy for the CAV is calculated by the RSU. The cooperative control is achieved by the information sharing among the RSUs of the control strategies. In addition, the transmission path through V2V and V2I is planned dynamically in RSUs to allocate the resources and balance the load. Finally, the control signal is downloaded to the CAV to regulate the longitudinal motion based on its real-time states, and the close-loop control is realized.
In this subsection, the optimal control design is investigated for the CCC system with communication delays in a heterogeneous platoon including CCC and human-driven vehicles. First, the mathematical model of the vehicle dynamics for the connected vehicles is formulated. Here, we investigate the longitudinal control of CAVs where the velocity and distance between two consecutive vehicles are considered. The CAV’s dynamics is formulated as where , namely, the acceleration, is the control signal of -th CAV, is the time delay introduced by the V2V communication, and and are the velocity and headway, respectively.
The behavior of the human-driven vehicle can be formulated as [31, 43]. where denotes the range policy, and are the system parameters determined by the human driver, and and are the velocity and headway of -th human-driven vehicle.
Considering vehicles in the whole system as shown in Figure 4, we assume there are CAVs, and then, the whole system can be split into vehicular platoons with single CAV in each platoon. The -th vehicular platoon has vehicles that . We define the state vector as , where and denote the state errors to the desired states and Then, base on (1) and (3), the mathematical model for the whole system can be formulated as where the control signal is an -dimensional vector corresponding to CAVs, and the coefficient matrices can be expressed as and the block matrices are where , and , .
Then, the corresponding discrete-time system model can be formulated as where and is the sampling period, and is the network-induced delay. Here, the short-delay case is considered that is smaller than the sampling period.
The objective is to design an optimal control strategy to reduce energy consumption and improve traffic capacity. In the uniform traffic flow, each vehicle is tracking the desired velocity and headway. Thus, in order to minimize the deviations of CAV’s headway, velocity, and acceleration, the optimization problem of the control system is typically designed as where is the finite time horizon and and are the symmetric positive definite weight matrices. Solving the optimization problem, we can obtain the control signals for the CAVs to regulate their longitudinal motion.
In the proposed architecture, RSUs and RSUCs can provide powerful communicating and computing capabilities in a coordinated way, enabling connected vehicles in the local wireless network to interact with each other. Here, vehicles tend to move in an organized fashion in order to maximize the throughput of the traffic flow. Therefore, a uniform desired velocity is to be realized for the whole system. Since the RSUs can share the local information with each other, the desired states for the traffic flow can be determined by the RSUCs with global view. Then, with multiple RSUs implemented in the system to provide seamless experience for all vehicles, we can split the whole system into several parts with a single RSU serving one corresponding vehicular platoon. In this case, when each platoon reaches the desired states, the total cost can be minimized, and the traffic capacity is maximized. Thus, the whole system can split into vehicular platoons as . In particular, each vehicle platoon has multiple vehicles with a CAV at the tail, and other vehicles are human-driven. For the given -th vehicular platoon, there are total vehicles that we have .
In this case, for the decentralised control system, the optimization problem in (9) can be equivalent to a noncooperative control game for each vehicle platoon as where for CAV , ; , , and are the , , and coefficient matrices as parts of the original matrix , matrix , and matrix in (7), respectively; the control signal is for the -th CAV; and and are the corresponding symmetric positive definite weight matrices.
In order to solve the optimal control strategy , we define a new state vector , and we have where the coefficient matrices are
Then, by using , we can rewrite the optimization problem in (10) as where
Then, as a part of cost function , the residual cost function is defined as
Based on our previous works [44], the residual cost function can be derived as a quadratic function of platoon states that
Then, the optimal control strategy is derived as where
| ||||||||||||||||||||||||||||||
The optimal controller design can be summarized as in Algorithm 1. The message flows for the control system are shown in Figure 5. In particular, numerous RSUs are seen as multiple remote controllers in the WNCS model. CAVs equipped with sensors and actuators connect to RSUs through wireless link. This distributed and collaborate design promotes the system reliability and stability. Fog nodes deal with real-time responsiveness while the cloud server focuses on delay-tolerate data processing and storage.
The control scheme is designed in the following steps. First, at each sampling interval, the state information of the whole platoon is captured by vehicles and uploaded to the RSUs via V2I communication. The RSUs are responsible for local information processing and filtering. To enhance the capacity and efficiency of the system, the selected traffic flow information is transmitted to the cloud server. Based on the historical and real-time data with global insight, the cloud employs a powerful AI to learn the generic trends of traffic flow in the near future. In the cloud, the desired speed of vehicles can be determined to realize dynamic traffic flow management. The computing results in the cloud are then downloaded to the RSUs. Then, the vehicle platoon model and the optimal control problem are formulated in the RSUs. The off-line part of the control strategy proposed in Algorithm 1, namely, the coefficient , can be calculated by the RSU which is then transmitted to the CAV. Finally, the real-time control signal is computed by the on-board controller based on the coefficient and the real-time measurements of vehicle’s states, namely, the on-line part of Algorithm 1, to regulate vehicle’s longitudinal motion. In the next time period, the updated vehicle states are feeded back to the RSU to realize the close-loop control.
3.2. Simulations and Results
In this subsection, we provide simulations of the connected vehicle system to study the performance of the proposed control scheme. Here, the typical scenario of the three-vehicle platoon is considered [31, 43], which can be easily extended to more vehicle scenarios. In particular, each platoon consists of 2 normal human-driven vehicles and a single CAV at the tail. In addition, in order to validate the performance of the distributed control, 3 distributed platoons with different initial states are considered in the simulation, which can also be easily extended to more complex distributed control system. For the first platoon, the initial states for the CAV are set to be [m] and [m/s]. For the second platoon, the initial states for the CAV are set to be [m] and [m/s]. For the last platoon, the initial states for the CAV are set to be [m] and [m/s]. The global desired velocity for the traffic flow is set as [m/s], and the desired intervehicle distance is set to be [m]. The sampling periods and communication delays are different in each platoon, which are set as ; ; and to verify the effectiveness of the proposed algorithm when the communication topology changes.
Figures 6 and 7 show the headway and velocity trajectories of the CAVs in 3 different vehicle platoons, respectively. It can be seen that the CAVs’ velocity and headway are tracking the desired states in the presence of various time delays, and the remained state errors are close to almost zero. Figure 8 shows the acceleration of the CAVs, all of which go down to near zero within about 5 seconds. The results depict that all 3 CAVs gradually reach the same desired states with the proposed control strategy. Therefore, all of the 3 platoons can simultaneously track the desired velocity no matter what the initial states are. This indicates that the proposed network architecture and control algorithm can regulate the uniform traffic flow with multiple distributed controlled vehicle platoons.
4. Future Research Challenges and Open Issues
Current research has achieved great success in the design of low latency and high efficiency network system, but many of the technologies are not fully resolved in practical applications. In the following, we conclude the most talked key research directions and challenging issues to be addressed.
4.1. Tradeoff between Communication and Control
Based on the analysis of the critical system variables involved with wireless communication, it is obvious that the condition of the communication network directly influences the control performance. In the existing research of WNCS, due to the complexity of the control system itself, researchers often separate the design of communication network and control strategies independently. However, in practical applications, the network characteristic tends to change dynamically, and the corresponding control strategies also need to be changed. If the network resource management and control strategy optimization can be jointly designed, the performance and the system stability can be further improved. In the independent design from the control aspect, control strategies are normally carried out with certain communication requirements, which results in the loss of serviceability in other environments. Even the communication system can satisfy the control requirements, predefined communication variables may result in the waste of wireless resources with low system efficiency. Therefore, it is essential to jointly design more integrated control algorithms that consider both communication and control so that the whole system can be compatible with various delay-sensitive and safe-critical applications. However, in order to realize the integration of control systems with communication, multiple interaction and coordination among system components should be reevaluated. For example, higher information rate brings better control performance, which at the same time leads to traffic congestion and increases computational burden. In the view of resource allocation of communication networks, since the wireless resource consumption observably increases with growing QoS requirements, the joint design approach may use various communication techniques to satisfy different QoS requirements. In the case study, we consider the network-induced delay and packet dropout in the design of the control strategy. Further research is needed in the controller design with other consideration such as transmission path planning, energy consumption, and resource allocation. In addition, the dynamic nature of the vehicles results in the time-varying network topology. In this case, it is difficult to maintain the system stability, and the controller design remains challenging.
4.2. Security and Privacy Problems
Security issues should be carefully considered in the whole system, including radio access, data transmission, and controller design. However, current solutions based on classic secrecy and authentication methods do not meet the needs of real-time control applications. Existing centralized security protection protocols might not be applicable in the distributed network system and may bring significant delay in delay-sensitive applications. Therefore, it is a great challenge to achieve both high security and control performance under limited resources. In the connected vehicle scenario, the sensor-based and V2X-enabled automated vehicles rely heavily on electronic equipments with sensitive personal information needed to be kept secure, such as the position data, social network information, and multimedia records. In this case, some specific procedures need to be taken into consideration to avoid potential cybersecurity attacks [45]. Therefore, information security and privacy problems need further investigation.
4.3. Application Considerations
In the whole network, fog not only acts as middle-ware between the cloud and edge but also provides quick and efficient control services. Depending on the type of service, the fog servers take the responsibility of deciding the kind of tasks to be processed locally and that to be offloaded to the cloud [12]. In particular, fog has relatively weaker storage and computation resources than the cloud, so whether a certain application’s data are stored locally or transmitted to the cloud becomes a problem. In addition, what kind of network structure is suitable for what type of applications, what kind of communication technology will be used between fogs, and how cooperation will be realized between fogs are major research issues that need further investigation. Different use cases come with diverse requirements. How to customize the network to provide corresponding network resources with specific control applications remains challenging.
With the removal of wiring in WNCS, permanently power supply is often removed, so additional limitations on the energy consumption of the system components arise. The tradeoff between control performance and network lifetime should be considered thoroughly. More fog nodes mean more energy consumption, less computing resource result in longer time delay. How to address the tradeoff in energy consumption and computation in order to improve the performance of green communications also needs further research.
5. Conclusion
In this paper, we propose a novel fog-based distributed network architecture toward real-time control applications. Three logical layers, the cloud, distributed fog, and edge smart devices, are considered. We discuss the critical system variables of communication and control systems, including sampling period, message delay, message dropout, and noisy channel. Fog enriches the network resources making real-time control possible. A case study of CCC is introduced to demonstrate the feasibility of the proposed architecture in the field of vehicular networks. The optimal control design for the CAV is proposed to realize uniform traffic flow. Simulations indicate that the performances are improved with the low latency and real-time control capabilities. Finally, we discuss some future research directions and open issues to be addressed.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that there is no conflict of interest regarding the publication of this paper.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (No. 61701010), the Beijing Social Science Fund (19GLC051), the Beijing Nova Program of Science and Technology (Z191100001119094), and the Scientific Research Plan of Beijing Municipal Commission of Education under Grant KM201910005026.