#### Abstract

This paper investigates the secrecy outage performance of a downlink cooperative nonorthogonal multiple access- (NOMA-) assisted hybrid satellite-terrestrial network, where a satellite source communicates with two terrestrial users based on NOMA, in the presence of a terrestrial eavesdropper. We assume that there is no direct link between the satellite source and the far terrestrial user, and the near terrestrial user acts as a relay to forward the information to the far terrestrial user. The near user is assumed to support both half-duplex mode and the full-duplex mode for information relaying. We derive the closed-form exact expressions for the secrecy outage probability (SOP) of the near user, as well as the closed-form approximate expressions for the SOP of the far user based on the Gaussian-Chebyshev quadrature. Theoretical results are validated by conducting simulations. The impacts of various system parameters on the performance comparison between the half-duplex mode and full-duplex mode are investigated. Particularly, the full-duplex mode is shown to outperform the half-duplex mode when the received signal-to-noise ratio (SNR) at the near user is low or the residual self-interference is low.

#### 1. Introduction

Integrated/hybrid satellite-terrestrial networks have been proposed to tackle the limitations of standalone satellite networks and terrestrial networks [1]. The main difference between integrated and hybrid satellite-terrestrial networks is that the terrestrial component of an integrated network is controlled by the satellite resource and network management system and uses the same frequency bands as the satellite component, while the terrestrial component of a hybrid network is independent from the satellite component and does not necessarily use the same satellite frequency [2]. In this paper, we do not put restrictions on the relationship between the terrestrial and satellite components in the satellite-terrestrial networks and thus focus on the hybrid satellite-terrestrial networks.

Hybrid satellite-terrestrial networks may face some major challenges, such as the lack of availability of line-of-sight (LoS) link due to obstacles; shadowing effects or weather factors will degrade the communication quality between the terrestrial users and the satellites [3]. In this regard, cooperative technology is an effective method to overcome these challenges by the cooperation between network nodes [4]. Specifically, cooperative technology can be divided into two types: one employs dedicated relays to forward the information to the users with inferior channel condition, and the other one relies on the users with better channel condition to forward the information to the users with inferior channel condition [4]. Most of the existing studies on hybrid satellite-terrestrial networks applied cooperative technology based on dedicated relays to forward the messages from the satellites to the terrestrial users, and such networks are called hybrid satellite-terrestrial relay networks [5, 6].

Meanwhile, nonorthogonal multiple access (NOMA) is a promising technique for improving the spectral efficiency of satellite-terrestrial networks [7, 8]. NOMA uses nonorthogonal transmission technology to superpose information of multiple users in the same time and frequency domain, while the receivers use successive interference cancellation (SIC) technology to decode their information [7, 9]. A key feature of NOMA technology is that users with better channel conditions can have prior information of other users. By using this prior knowledge, cooperative NOMA (CNOMA) has been utilized to further improve the performance of far users with inferior channel conditions [10–13]. Recently, a large number of studies have applied NOMA on terrestrial users in hybrid satellite-terrestrial relay networks [14, 15]. For example, [14] investigated the outage probability of a NOMA-assisted hybrid satellite-terrestrial relay network and derived the closed-form expression for the outage probability of each NOMA user.

Due to the wide coverage of satellite communications, there are serious security problems for hybrid satellite-terrestrial networks [16–18]. In this respect, physical layer security (PLS) has become a promising technology that complements and significantly improves the security of wireless networks [18] and has been applied in hybrid satellite-terrestrial networks to enhance the security of terrestrial users. It is noted that although applying CNOMA to hybrid satellite-terrestrial networks can improve the performance of far users, almost all existing studies have considered CNOMA with dedicated relays. However, when dedicated relays are not available, cooperation among users needs to be considered. To the best of the authors’ knowledge, there is no work on the secrecy outage performance analysis of CNOMA-assisted hybrid satellite-terrestrial networks with user cooperation. Although the secrecy performance of CNOMA-assisted terrestrial networks has been investigated in existing literature such as [11], the current results cannot be directly applied to the hybrid satellite-terrestrial networks, since the channel environments of the hybrid satellite-terrestrial networks are much more complicated than those of the terrestrial networks [19]. This research gap motivates the work in this paper.

The main contributions of this paper are summarized as follows: (i)We consider a CNOMA-assisted hybrid satellite-terrestrial network with a satellite source, one far terrestrial user, and one near terrestrial user, in the presence of a terrestrial eavesdropper. It is assumed that there is no direct link between the satellite source and the far terrestrial user, and thus, the satellite source sends the information of both users to the near terrestrial user, and then, the near terrestrial user acts as a relay to forward the information to the far terrestrial user under the half-duplex relaying mode or the full-duplex relaying mode. It is also assumed that the eavesdropper wiretaps not only the information broadcast by the satellite source but also the information forwarded by the near terrestrial user(ii)Under the half-duplex relaying mode and the full-duplex relaying mode of the near user, the closed-form exact expressions for the secrecy outage probability (SOP) of the near terrestrial user are derived, and the closed-form approximate expressions for the SOP of the far terrestrial user are derived based on the Gaussian-Chebyshev quadrature(iii)Theoretical results are validated by Monte Carlo simulation results. It is shown that the full-duplex relaying mode outperforms the half-duplex relaying mode when the received signal-to-noise ratio (SNR) at the near terrestrial user is low or the residual full-duplex self-interference is low

The rest of this paper is organized as follows. Section 2 introduces the related work. In Section 3, we present the system model. In Sections 4 and 5, the SOPs under the half-duplex mode and the full-duplex mode are investigated, respectively. In Section 6, Monte Carlo simulations are provided to verify our theoretical results, followed by conclusions in Section 7.

#### 2. Related Work

Hybrid satellite-terrestrial relay networks have attracted a lot of research attention. In [6], the outage performance of a hybrid satellite-terrestrial relay network was investigated, and the outage probabilities under two cooperative relaying strategies were theoretically derived. In [20], a multiuser hybrid satellite-terrestrial relay network with opportunistic scheduling was considered, and the approximate as well as the asymptotic expressions for the outage probability were derived. In [21], an uplink satellite-terrestrial relay network with multiple decode-and-forward terrestrial relays was considered, and the closed-form expressions for the outage probability and the throughput were derived under the partial relay selection scheme. In [22], the performance of cognitive-uplink fixed satellite service and terrestrial fixed service was investigated, and an analytical expression for the network capacity was derived. In [23], a satellite and aerial-integrated network with rate-splitting multiple access (RSMA) was considered, and an iterative penalty function-based algorithm was proposed to solve the problem of maximizing the system sum rate. In [24], a secure beamforming scheme for a RSMA-based cognitive satellite-terrestrial network in the presence of multiple eavesdroppers was proposed to maximize the secrecy-energy efficiency.

NOMA or CNOMA has been applied to hybrid satellite-terrestrial networks as well. Specifically, in [15], a NOMA-assisted hybrid satellite-terrestrial relay network with small-cell users and macrocell users was considered, and the closed-form expressions for the outage probabilities were derived. In [25], a satellite multicast network was considered to share the millimeter wave spectrum with a terrestrial network employing NOMA technology, and the system sum rate was maximized by optimizing the beamforming vectors and the power coefficients. In [16], the effect of hardware impairments on the secrecy performance of a NOMA-assisted hybrid satellite-terrestrial relay network was investigated with colluding or noncolluding eavesdroppers, and the closed-form expressions for the SOP under the partial relay selection scheme were derived. In [26], the performance of a NOMA-assisted hybrid satellite-terrestrial relay network was investigated under the partial relay selection scheme and imperfect SIC, and the closed-form expressions for the outage probability and the ergodic capacity were derived. Besides NOMA and CNOMA, there are other emerging technologies such as reconfigurable intelligent surface (RIS) [27, 28], which have been considered to improve the performance of hybrid satellite-terrestrial networks. For example, in [27], the beamforming was designed to minimize the total transmit power under the user rate constraints for a RIS-aided hybrid satellite-terrestrial relay network.

Physical layer security has been studied in CNOMA networks. In [29], the physical layer security of a two-user CNOMA network, where the strong user acts as a relay for the weak user, in the presence of an eavesdropper, was studied, and the closed-form expressions for SOPs of the two users were derived. The work in [30] extended the work in [29] to a multiantenna scenario. In [31], the secrecy performance of a CNOMA network based on half-duplex and full-duplex amplify-and-forward and decode-and-forward relaying protocols in general - fading channels was investigated. Physical layer security is also a hot research topic in hybrid satellite-terrestrial networks. In [17], the impacts of the relay selection and user scheduling scheme on the secrecy performance for a hybrid satellite-terrestrial relay network with multiple terrestrial relays were investigated. In [18], a downlink hybrid satellite-terrestrial relay network with multiple terrestrial users with the help from multiple cooperative relays, in the presence of multiple eavesdroppers, was considered; the secrecy performances under amplify-and-forward and decode-and-forward protocols were investigated. In [32], the secrecy performance for a NOMA-assisted hybrid satellite-terrestrial network consisting of a satellite source and multiple terrestrial users with the help from a terrestrial base station was investigated.

To our best knowledge, current works on the secrecy performance of CNOMA-assisted hybrid satellite-terrestrial network networks such as [17, 18, 32] are all based on dedicated cooperative relays. However, when dedicated relays are unavailable, the cooperation among users is desirable. Such consideration motivates the work in this paper to investigate the secrecy performance of CNOMA-assisted hybrid satellite-terrestrial networks with user cooperation.

#### 3. System Model

We consider a CNOMA-based hybrid satellite-terrestrial network consisting of a geosynchronous earth orbit (GEO) satellite source () and two terrestrial users (near user and far user ), in the presence of an eavesdropper (), as shown in Figure 1. The direct transmission link between and is assumed to be absent due to shadow fading caused by raining, fog, or other obstacles [18, 20]. We assume that the near user serves as a decoded and forwarding relay to help in transmitting information to the far user , while wiretaps the information broadcast by the satellite source and the information forwarded by the near user (it is assumed that the eavesdropper equips two receivers: one for decoding the satellite signals and the other one for decoding the terrestrial signals, to eavesdrop the information of both the satellite source and the near user [33]). Let , , , and denote the channel coefficients from to , from to , from to , and from to , respectively (the channel state information (CSI) is often assumed to be available for resource scheduling. This can be realized by channel training and estimation [34]. In particular, in the training phase, broadcasts pilot signal; and receive the pilot signal for channel estimation and then feed the CSI back to ). Additionally, all the links in the network are considered to be independent and flat fading. Similar to the existing works [16, 35], perfect successive interference cancellation (SIC) is assumed to be available. It is assumed that delay-sensitive services are carried by , and there is a minimum rate denoted as required to guarantee the quality of service (QoS) for .

The satellite links from to and from to are assumed to follow the shadowed Rician fading model [20, 21, 36], which is the most commonly used channel model in land mobile satellite communication systems [37]. Under the shadowed Rician fading model, the channel coefficient can be written as where is the scaling parameter including various practical effects such as the free space loss (FSL) and the antenna gain and is the small-scale fading. The can be expressed as [37] where is the wavelength, is the distance between the terrestrial user and the center of the satellite beam, km is the height of the GEO satellite, and and denote the receive antenna gain and the satellite beam gain of the terrestrial user, respectively.

According to [38], can be expressed as where is the maximum antenna gain at the boresight and is the off-boresight angle. Denote by the angle between the terrestrial receiver and the satellite beam and by the 3 dB angle. The expression for is given by [37] where denotes the maximal beam gain, , and and represent the first-kind bessel functions of orders 1 and 3, respectively. To attain the best beam gain, we have [39], and thus, and , where .

Under the shadowed Rician fading model, the probability distribution function (PDF) of can be described as [16, 21] where , , , is the average power of the LOS component, is the average power of the multipath component, is the Nakagami parameter, and denotes the confluent hypergeometric function [40].

The terrestrial links from to and from to are assumed to follow the Rayleigh fading model. Thus, the PDF and the cumulative distribution function (CDF) of are given by respectively, where is the average channel gain.

It is assumed that all the nodes except are equipped with a single antenna [16, 17, 41]. It is also assumed that is equipped with one or two antennas. Specifically, works under the half-duplex mode when only one antenna is available and works under the full-duplex mode when two antennas are available, where one antenna is for transmitting and the other one for receiving. Next, we discuss these two modes.

##### 3.1. Half-Duplex Mode

When works under the half-duplex mode, the entire transmission time is divided into two phases. In the first time phase, transmits the superimposed signal (considering hardware impairments is not the focus of this paper. Interested readers may refer to [39, 42]) to with , where is the transmit power of and is the power allocation coefficient for . The received signal at is given by where denotes the additive white Gaussian noise (AWGN) at . It is assumed that adopts SIC to decode the received message [16, 26]. Specifically, is decoded first, and then, is decoded by cancelling the from the received message (in order to successfully perform SIC, a minimum received power difference is required to distinguish between the desired signal and the interference signal [43]. This means that the difference between the two power allocation coefficients and shall be large enough such that SIC can be successfully performed at . It is noted that the performance analysis in this paper does not put restriction on the values of and . Thus, our analysis is still valid when the values of and are set to satisfy the SIC requirement). The signal-to-interference-plus-noise ratio (SINR) at for decoding is where and . Then, after applying SIC, the SNR at to decode its own signal is given by

Meanwhile, since there is a direct link between and , the can also receive the information broadcast by . Thus, the message received by is expressed as where denotes the AWGN at . Here, we consider a worst-case scenario, where has multiuser detection capability and adopts parallel interference cancellation (PIC) to decode the messages of and [16, 44, 45]. Thus, the received SNRs at to decode the messages and in the second time phase are respectively, where and .

In the second time phase, forwards the information to , where can also intercept. The received message at and are respectively, where denotes the AWGN at and is the transmit power of . Thus, the SNRs at and for decoding in the second time phase can be written as respectively, where and Note that and follow exponential distributions with mean values and , respectively, where and .

The end-to-end SNR at for decoding is the minimum SNR in the two time phases, which can be written as [26]

As for to decode , a worst-case scenario is considered, where adopts the maximal ratio combining (MRC) to combine the received in the first and the second time phases. Thus, the end-to-end SNR at for decoding is

##### 3.2. Full-Duplex Mode

When works under the full-duplex mode, during the whole transmission time, can transmit with the power and the power allocation coefficients and , while can forward the information to with transmit power . The message received by can be expressed as where is the residual self-interference signal. We assume that multiple analog and digital self-interference cancellation techniques are adopted to suppress the self-interference. Accordingly, can be modelled as a Gaussian distributed signal which is independent from the transmitted signal due to various affecting factors in the self-interference cancellation procedures [46]. Thus, for the convenience of later analysis, we denote the power of as with to identify the relationship between the self-interference signal power and the transmit signal power [47–49].

It is assumed that decodes first, and the SINR at for decoding is where . After applying SIC to cancel the from the received message, the can proceed to decode , where the SNR for decoding is

Since relays to with transmit power , the message received by is and the SNR for decoding at is given by

Thus, the end-to-end SNR at for decoding is the minimum SNR that can be achieved at and , which is written as

The message received at can be written as

It is assumed that adopts PIC to decode and , and thus, the SINRs for decoding and are given by respectively, where

#### 4. Secrecy Outage Probability under Half-Duplex Mode

The SOP, defined as the probability that the secrecy rate is below a given threshold [41, 50, 51], is investigated. Under the half-duplex mode, the SOP of is expressed as where .

##### 4.1. SOP of the Near User

First, the SOP of is studied. According to the definition in (26), the SOP of can be expressed as where denotes the SINR for the target secrecy rate denoted as for under the half-duplex mode.

In order to obtain a closed-form expression for , we first deduce the CDF of and the PDF of . Specifically, the CDF of can be obtained as

From (5), by supposing , the PDF of can be expressed as

Under integer , the PDF of can be rewritten as [16, 21, 32] where and is the Pochhammer symbol [40]. Then, the CDF of can be calculated as

By using (31), the CDF of in (28) can be rewritten as

By following similar procedures, the PDF of can be obtained as

By substituting (32) and (33) into (27), the can be obtained as

By defining and and with the help of [40], we can simplify (34) as

Through some simple calculations of (35), the closed-form expression for the SOP of can be obtained as

*Remark 1. *When approaches to infinite, the CDF of can be approximated as [16, 26]
where means the infinitesimal of higher order for . Thus, the in the high SNR region is approximated as
It is seen from (38) that as , the approaches to zero. Therefore, the SOP of does not saturate when the SNR increases.

##### 4.2. SOP of the Far User

Then, we derive the SOP of . The SOP of can be expressed as

To simplify (39), the CDF of and the PDF of are first derived. The CDF of is expressed as where can be written as

Note that when , the inequality cannot be satisfied. Thus, we have and . Otherwise, when , the and can be evaluated as respectively. Based on the above analysis, the CDF of is given by

As for the PDF of , it can be obtained as [40] where and

By substituting (43) and (44) into (39), the is obtained as given by (45), where

To the best of authors’ knowledge, it is hard to derive the exact expression for in (45). Thus, we adopt the Gaussian-Chebyshev quadrature to approximate as [52] where is given by (47), is the number of terms, is the Gaussian weight, which is given in Table (25.4) of [52], , and is the zero of Legendre polynomials.

*Remark 2. *In the high SNR region, i.e., and , we have and . Thus, can be approximated as
where and can be obtained as
respectively. It can be shown that, unlike , the SOP of saturates in the high SNR region.

*Remark 3. *Note that when , we have , and thus, the SOP of user is . This means that the power allocation coefficient for shall be smaller than ; otherwise, will be always in secrecy outage.

#### 5. Secrecy Outage Probability under Full-Duplex Mode

This section derives the SOPs of all the users under the full-duplex mode. The SOP for can be expressed as

##### 5.1. SOP of the Near User

First, the SOP of the near user under the full-duplex mode is studied. The SOP of can be rewritten as where is the SNR for the target secrecy rate for under the full-duplex mode.

From (18), (23), (30), and (31), the CDF of and the PDF of can be derived as respectively. By inserting (52) and (53) into (51), we get

*Remark 4. *In the high SNR region, i.e., and , the SOP of can be approximated as

It can be shown that the SOP of user saturates in the high SNR region. This is because the residual self-interference restricts the achievable secrecy rate of under the full-duplex mode.

##### 5.2. SOP of the Far User

Then, we investigate the SOP of , which can be expressed as

Similar to (43), the CDF of can be derived as

The PDF of is derived as

By substituting (57) and (58) into (56), the can be derived as given by (59), where .

Since it is hard to derive an exact closed-form expression for , similar to the half-duplex mode, with the help of Gaussian-Chebyshev quadrature, in (59) can be approximated as [52] where is given by (61) and .

*Remark 5. *When and , the can be approximated as
where
and . It can be shown that the SOP of saturates in the high SNR region, which is caused by the residual self-interference.

*Remark 6. *Note that when , we can have . This means that will be always in secrecy outage when the power allocation coefficient for is not smaller than . Since , we have . Thus, based on the analysis in Remark 3, the threshold for the power allocation coefficient for under the half-duplex mode is larger than that under full-duplex mode. This indicates that more power can be allocated to under the full-duplex mode than that under the half-duplex mode, provided that the SOP of is not .

#### 6. Numerical Results

In this section, we verify our theoretical results via conducting simulations. The system parameters and channel fading parameters are presented in Tables 1 and 2, respectively, where FHS denotes heavy shadowing and AS denotes average shadowing [16, 18, 37].

Besides, we assume dB, dB, bits/s/Hz, dB, and [18, 32, 49].

Figure 2 compares the SOPs with FHS and AS for the near user under different values of . The theoretical results are shown to match well with the simulation results, which verifies the correctness of the theoretical results. As expected, the secrecy outage performance with AS is better than that with FHS. It is seen that compared to the half-duplex mode, the secrecy outage performance under the full-duplex mode is better when is small. When is large, it is seen that the secrecy outage performance under the full-duplex mode saturates and underperforms that under the half-duplex mode. This is consistent with our analysis in Remarks 1 and 4 that the SOP of under the half-duplex mode does not saturates in the high SNR region, while it saturates under the full-duplex mode. This indicates that the near user prefers the full-duplex mode when the SNR is low and prefers the half-duplex mode when the SNR is high.

Figure 3 compares the SOPs with FHS and AS for the far user under different values of . It is seen that the secrecy outage performance with AS is better than that with FHS, which is expected. When is small, it is seen that the SOP under the full-duplex mode is lower than that under the half-duplex mode. It is also seen that when is large, the SOPs under the full-duplex mode and the half-duplex mode both saturate, which is consistent with the analysis in Remarks 2 and 5. Besides, it is seen that compared to the half-duplex mode, the SOP under the full-duplex mode is higher with FHS and is lower with AS. This indicates that for the far user, the half-duplex mode is preferred only when the SNR is high and the channel fading is severe.