Abstract
Federated learning (FL) enables clients to train a machine learning model collaboratively by just aggregating their model parameters, which makes it very useful in empowering the IoTs with intelligence. To prevent privacy information leakage from parameters during aggregation, many FL frameworks use homomorphic encryption to protect client’s parameters. However, a secure federated learning framework should not only protect privacy of the parameters but also guarantee integrity of the aggregated results. In this paper, we propose an efficient homomorphic signcryption framework that can encrypt and sign the parameters in one go. According to the additive homomorphic property of our framework, it allows aggregating the signcryptions of parameters securely. Thus, our framework can both verify the integrity of the aggregated results and protect the privacy of the parameters. Moreover, we employ the blinding technique to resist collusion attacks between internal curious clients and the server and leverage the Chinese Remainder Theorem to improve efficiency. Finally, we simulate our framework in FedML. Extensive experimental results on four benchmark datasets demonstrate that our framework can protect privacy without compromising model performance, and our framework is more efficient than similar frameworks.
1. Introduction
Traditional machine learning (ML) uses huge amounts of data collected from various sources to train models. However, data sharing from different devices or organizations may disclose privacy information about the owners. To solve the dilemma between protecting data privacy and leveraging the AI benefits to these data sensitive domains, federated learning (FL) [1, 2] is proposed to train ML models without sharing data directly. Because FL can protect the privacy while utilizing data, it has great application prospect in many scenarios [3–6], especially in IoTs [7, 8].
FL enables devices to collaboratively build a global ML model by only aggregating their model parameters with their data kept at their local storage. However, it still has the problem of privacy leakage. Existing researches [9–11] revealed that the exposed gradients still retain sensitive information about the training data. To avoid exposing gradients of a client during the training process, many privacy preserving frameworks based on cryptographic techniques have been proposed. Because homomorphic encryption (HE) allows aggregating the gradients through encryption envelope, privacy preserving FL frameworks based on HE have aroused many researchers’ interests [9, 12–14].
However, because HE consists of complex cryptographic operations, these frameworks always bring heavy overhead. On the other hand, all clients in these frameworks share the same key pairs. If the server colludes with any client to get the secret key, the privacy protections will be invalid. Apart from efficiency and collusion attack issues, most privacy preserving FL frameworks ignore verifying the integrity of the aggregated results. However, a malicious or compromised server may forge the aggregated results to seduce clients to expose more sensitive information. Therefore, a privacy preserving FL framework should consider efficiency, privacy, and integrity.
This paper aims to address these problems by designing an efficient homomorphic signcryption based privacy preserving framework for FL in IoTs. We compare our framework with others in Table 1. Our contributions are summarized as follows:(i)We design a homomorphic signcryption mechanism to encrypt and sign the clients’ gradients at the same time. According to its additive homomorphic property, our mechanism allows the server to aggregate the gradients and signatures securely and allows the clients to verify the integrity (correctness) of the aggregated results.(ii)We employ the blinding technique to resist collusion attacks. Even if clients try to collude with the server, they cannot steal privacy information of other clients’ parameters.(iii)We combine the clipping and quantizing technique in Batchcrypt [13] and the Chinese Remainder Theorem to reduce the number of complex cryptographic operations and ciphertexts. Thus, our framework is more efficient in computation and communication than similar frameworks.(iv)We present a comprehensive analysis for the security of our framework, which proves the privacy security of the gradients and the model as well as the integrity of the aggregated results. Finally, we simulate our framework in FedML on four benchmark datasets (and corresponding models): Federated MNIST, CINIC-10, CIFAR-10, and CIFAR-100. The experimental results demonstrate that our framework has high computation and communication efficiency.
The rest of the paper is organized as follows: the preliminaries and related works are briefly introduced in Section 2. The detailed procedures of our framework are presented in Section 3. The security analysis and experimental evaluation are provided in Sections 4 and 5, respectively. Finally, we conclude our paper in Section 6.
2. Preliminaries and Related Work
In this section, we introduce the preliminaries and related works of privacy preserving FL.
2.1. Paillier One-Way Trapdoor Permutation
In this paper, we employ the Paillier one-way trapdoor permutation [15] to realize the homomorphic signcryption mechanism. The details of the Paillier one-way trapdoor permutation are described as follows:(i)Key generation: select two large prime numbers and randomly with and compute the modulus and the least common multiple . Select a random group generator , where the order of divides . The key pairs is .(ii)Encryption: for any message , split into with . The ciphertext of is .(iii)Decryption: the decryption process is as follows:
The function is defined as .
2.2. Batchcrypt
In Batchcrypt [13], the authors proposed a method to clip and quantize the gradients; it consists of two functions:(i): C=compute the clipping threshold according to the number of gradients , the maximum gradient , and the minimum gradient of each layer.(ii): quantize the gradient into bits according to the scaled quantization range in case the sum of gradients from all clients overflows. Here, denote the number of clients.
Batchcrypt [13] allows clipping and quantizing the gradients into fixed bit width integers, which can reduce the number of parameters in encryption and decryption.
2.3. Chinese Remainder Theorem
Suppose are positive pairwise coprime integers. Let . Then there exists one unique integer satisfying the following group of congruences: . Similarly, we can unpack to get . To ease description, we denote the packing function as and the unpacking function as . For any two packed data and , the equation holds. Thus, we know that the CRT satisfies additive homomorphism.
2.4. Privacy Preserving Federated Learning
Many researches have been devoted to privacy preserving FL (PPFL), and they are mainly based on cryptographic methods. Shokri and Shmatikov [16] employed selective parameter update atop differential privacy to protect training record privacy. Geyer et al. [17] applied differential privacy directly to guarantee client level differential privacy. But these differential privacy based approaches reduced model accuracy significantly. Bonawitz et al. [18] leveraged secure aggregation to protect privacy in FL. Nevertheless, it exposes the trained model in the plaintext form. To address this problem, Phong et al. [9] proposed an additively homomorphic encryption based FL framework. However, it brought heavy computation and communication overhead because the encryption scheme involves complex cryptographic operations and large ciphertexts. To improve efficiency, Batchcrypt [13] tried to encode groups of quantized gradients large integers, but their method cannot defeat collusion attacks among internal curious clients and the server.
2.5. Verifiable Federated Learning
In FL, a malicious or compromised server may falsify aggregated results returned to the clients. Several methods have been proposed to solve this problem.
Ma et al. [19] used bilinear aggregate signature [20] for integrity verification. But the verification process involves all clients. For a large number of clients, this process is time-consuming. Xu et al. [21] employed a homomorphic hash function to verify integrity, but it cannot protect the privacy of the jointly trained model. Zheng et al. [22] designed Helen, a framework utilized to secure multiparty computing protocol [23], to verify the correctness of the aggregated results. Nevertheless, the framework does not support training complex models such as neural networks. Zhang et al. (referred to as CRT-Paillier in Section 5) [24] combined bilinear aggregate signature and Paillier encryption [15] to protect privacy and integrity of the aggregated results. But they cannot resist collusion attacks between the server and internal curious clients. Fu et al. proposed VFL [25] by employing Lagrange interpolation to realize secure aggregation and check the integrity of the aggregated results, but the parameter splitting process is costly, making it unsuitable for large deep learning models.
To summarize, among PPFL frameworks, many of them ignored verifying the correctness of the aggregated results. For verifiable frameworks, they either cannot apply to large nonlinear models, or cannot protect the privacy of the model, or cannot resist collusion attacks between the server and the clients. Our work is to address these problems.
3. Homomorphic Signcryption-Based Privacy Preserving Federated Learning Framework for IoT
3.1. Threat Model and Design Goals
In our framework, we assume security threats from three different perspectives:(i)Internal curious clients may try to steal privacy information about other clients by inspecting the ciphertexts transmitted during the training process. Here we should note that, because the ultimate goal of the clients in is to train a good ML model, the curious clients may collude with the server to steal private information about other clients, but they will not collude with the sever to tamper the aggregated results.(ii)The server may try to steal the jointly trained model because of its great economic value. But the model should be the common property of the clients.(iii)The server may return falsified aggregated results to clients driven by some unexpected motivations.
Consequently, to enable a privacy preserving FL framework under the aforementioned threat model, the design goals are summarized as follows:(i)Correctness: when the server and clients operate strictly according to the protocol, the aggregated results should be correct. This ensures that the clients achieve their goal and the final model has good performance.(ii)Data privacy: data privacy means the privacy of a client’s training data and gradients. The server and curious clients cannot gather any private information about the data and gradients from messages they receive.(iii)Model privacy: model privacy refers to the privacy of the jointly trained model. The server or any other party not in the framework cannot steal the model by inspecting the immediate data transmitted during the learning process.(iv)Verifiability: all clients can verify the integrity of the aggregation of parameters so that clients can detect the malicious behavior if the server returns tampered aggregated results.
3.2. Overview
As illustrated in Figure 1, our framework consists of 4 phases: initialization, model training, aggregation, and update. We summarize the 4 phases as follows:(i)Initialization phase: the Public Key Generator (PKG) determines the model parameters and generates Paillier keypairs, Pseudo-Random Generator , random seed , and pairwise coprime large numbers for all clients.(ii)Model training phase: all clients train the ML model locally clip and quantize the gradients into bits long integers, pack them using CRT, mask them with blinding factors, and then signcrypt them.(iii)Aggregation phase: the server aggregates the ciphertexts from all clients and distributes the result to them.(iv)Update phase: each client firstly decrypts the aggregated ciphertexts and then verifies the integrity of the aggregated gradients. If the result is correct, it unpacks them into plain aggregated gradients and updates their local model accordingly. Otherwise, it would terminate the learning process.
3.3. Detailed Construction
3.3.1. Initialization
For simplicity, we assume there are clients, and each client is uniquely indexed by a number . The set of client index is represented by . In our framework, the PKG needs to initialize the model parameters and generate keys and PRG as follows:(1)Initializing model parameters: the PKG specifies the architecture of the model, randomly initializes the model parameters as ( denotes the number of layers of the learning model), and specifies an appropriate learning rate .(2)Specifying packing parameters: the PKG generates positive integers that are pairwise coprime . We define . These integers are used to pack gradients.(3)Specifying blinding parameters: the PKG randomly chooses integer sets such that for ( stands for the number of parameters of the model) and then sends to client .(4)Specifying PRG: the PKG sends each client the same pseudorandom generator and seed . The pseudorandom generator is used to synchronize random numbers among clients for signcrypting the gradients in each training round.(5)Generating signcryption keys: the PKG sends each client a Paillier key pair for encryption and decryption and a secret value to sign the gradients, where , , and .
3.3.2. Model Training Phase
The model training phase of client consists of the following four steps:(1)Local training: in round of training, client trains its local model on dataset and computes the gradients .(2)Packing: client sends the layer-wise maximum value , minimum value , and the size of layer to the server. After receiving those layer-wise clipping parameters from all clients, the aggregator server calculates the clipping threshold , where and compute the maximum and minimum of a set, respectively. After receiving the clipping thresholds, client quantizes its gradients with , client partitions the quantized gradients into groups If is not divisible by , should be padded with 0 s. Then client packs them into with and . After packing, the number of cryptographic operations and ciphertexts is greatly reduced; thus the computation and communication efficiency is improved.(3)Blinding: to resist collusion attacks between curious clients and the aggregation server, client blinds by adding its blinding factors to get .(4)Signcrypting: for each blinded gradient , client computes its signature and uses the to synchronize with other clients a random number , where will be used as seed for the PRG in the next round of training. Finally, clients compute a signcryption of as .
We summarize the steps of model training phase in Algorithm 1.
|
3.3.3. Aggregation Phase
(1)Aggregation: after receiving the ciphertexts from all clients, the server aggregates them according to the layer index . Because Paillier trapdoor permutation is additively homomorphic, the sever computesBecause for and , we have
3.3.4. Update Phase
After receiving the aggregated ciphertexts , each client completes the update phase as follows.(1)Decrypting: client firstly decrypts the ciphertext: Then it computes the aggregated signature as follows:(2)Verifying: for each client , if the equation holds, the aggregation results are correct. Otherwise, it is supposed to be falsified, and the learning process terminates.(3)Unpacking: if the termination condition is not satisfied, client unpacks the plain aggregated gradients , where .(4)Updating: client updates the model parameters and prepares for the next round of aggregation.
We summarize the steps of update phase in Algorithm 2.
|
4. Security Analysis
Theorem 1. The privacy of the clients’ gradients is protected against internal curious clients.
Proof. Suppose an internal curious client intercepts a ciphertext from client by intercepting the messages between client and the server. Then client has . Though it can decrypt them to get and , it cannot infer any information about the gradients because the blinding factor is hidden from it. Therefore, our framework can protect the privacy of the clients’ gradients against internal curious clients.
Theorem 2. The privacy of the clients’ gradients is protected against collusion attack between clients.
Proof. Assume the set of collusion clients is with . For clients , their ciphertexts and contain blinding factors and that are unknown to clients in and the server. For any aggregated ciphertexts include and , the situation is similar. Thus, even when the server colludes with clients, they cannot infer any information about other clients’ gradients.
Theorem 3. In our framework, each client can independently verify the correctness of the aggregated results, and the server cannot deceive the clients with tampered aggregated results.
Proof. If the clients receive the correct aggregated results, obviously the equation holds. Assume a client receives forged aggregated results, and without loss of generality, we assume the sever falsified the aggregated results toIf the malicious server tries to successfully fool the verification mechanism of our framework, it must make sure the following equation holds: ; namely, the server should select and that satisfy . However, it is impossible because is confidential to the server. Therefore, we prove that the server cannot deceive the clients with tampered aggregated results.
5. Performance Evaluation
We evaluate the performance of our framework in this section. Specifically, we first compare its accuracy against that of the original FL framework in four benchmark datasets: Federated MNIST, CINIC-10, CIFAR-10, and CIFAR-100. The models corresponding to the datasets are listed in Table 2. Then we compare its computation and communication cost with that of the other two verifiable PPFL frameworks: VFL [25] and CRT-Paillier [24].
5.1. Experimental Setup
Our evaluation experiments are conducted on a Dell T7920 workstation with 1 Intel Xeon Silver 4210R CPU and 32 GB RAM. The OS is Ubuntu 18.04. We employ the FedML [26] in its standalone simulation computing paradigms to build the baseline framework. In both our and CRT-Paillier frameworks, the open-sourced python-Paillier [27] is adopted as the Paillier HE implementation. In our experiments, according to VFL [25] and CRT-Paillier [24], the gradients in their frameworks are of 32-bit length, while the gradients in our framework are clipped and quantized into 16-bit length with the dACIQ and Quantize function. The Paillier key size in both our framework and CRT-Paillier is set to 2048 bits, just as recommended in NIST [28].
5.2. Discussion of Results
Accuracy: we compared the performance of our framework with that of the original FL framework (Figure 2). For FMNIST, CINIC-10, CIFAR-10, and CIFAR-100, our framework can achieve an accuracy of 81.01%, 70.39%, 85.36%, and 65.60%, respectively, which is very close to that of the original FL, which has an accuracy of 80.99%, 70.49%, 85.81%, and 65.23%, respectively. The results demonstrate that our framework can guarantee the privacy of the learned model without compromising its performance. In fact, our quantized training sometimes has better results. Prior quantization work has observed a similar phenomenon [29], where the stochasticity introduced by quantization can reduce overfitting, similar to the function of dropout layer [30].
(a)
(b)
(c)
(d)
Time cost, in Figure 3, we compared the time cost of our framework with that of VFL and CRT-Paillier. In VFL, the degree of the interpolation polynomials is set to (VFL-4) and (VFL-8), respectively. It is known that the security level and cost of VFL are higher with larger . In CRT-Paillier, because the plaintext in Paillier encryption should be less than the modulus , the maximum group size is 60 to avoid overflow error. While the gradient in our framework quantized into 16 bits, the maximum batch size is 120.
(a)
(b)
(c)
(d)
Figure 3(a) shows that the encryption cost per client of all frameworks increases linearly with the amount of parameters. The encryption costs of our framework are 9.06 s for parameters and 24.17 s for parameters, respectively. In addition, the encryption costs of CRT-Paillier are 11.73 s and 35.55 s, VFL-8 are 33.83 s and 184.15 s, and VFL-4 are 12.19 s and 66.99 s. Though both our framework and CRT-Paillier employ Paillier encryption to protect privacy, our framework needs fewer encryption operations and takes less time because its batch size is larger. For VFL, though the interpolation is very fast, the parameter splitting costs are expensive. Thus, our framework is more efficient than VFL-8 and VFL-4.
Figure 3(b) presents the decryption cost of all frameworks. The decryption costs of our framework are 8.35 s for parameters and 21.10 s for parameters. The encryption costs of CRT-Paillier are 9.91 s and 43.21 s, VFL-8 are 2.22 s and 11.87 s, and VFL-4 are 0.91 s and 4.68 s. Because our framework and CRT-Paillier utilize the Paillier scheme to protect privacy, which involves exponentiations and modular multiplications with large integers, the decryption costs of both our framework and CRT-Paillier are a little higher than VFL. But our framework takes less time than CRT-Paillier because of its larger batch size.
Figure 3(c) displays the total time cost of a client. From the figure we know that the total cost of a client of our framework is less than both VFL and CRT-Paillier, and the gap grows linearly with the number of parameters. We should note that, as giant companies or organizations prefer large deep learning models to boost their performance, our framework is more suitable for large AI models.
For the server, the overhead is caused by secure aggregation. Figure 3(d) shows the aggregation cost of the server. Because of larger batching size, our framework needs fewer sum operations on the ciphertexts. Thus, our framework is more efficient than CRT-Paillier and VFL for the server.
Communication cost: we compared the communication cost of our framework with that of VFL and CRT-Paillier in Figure 4. Since we simulate different FL frameworks in the standalone computing paradigms of FedML, we use the ciphertext size exchanged between the server and clients as the metric for communication cost. In VFL-4, each gradient is randomly split into 4 parameters. Though the authors keep the number of parameters the same as the original model by employing CRT to batch parameters, the size of the batched results grows accordingly. Thus, the ciphertext expansion rate for VFL-4 is 4. Similarly, the ciphertext expansion factor of VFL-8 is 8. In CRT-Paillier framework, every 60 gradients are grouped together and then are encrypted to get a bit-length ciphertext, the ciphertext expansion factor is approximately . Similarly, the ciphertext expansion factor of our framework is approximately . Therefore, our framework is more communication efficient than VFL and CRT-Paillier.
(a)
(b)
6. Conclusion
In this paper, we have designed a preserving FL framework for IoT based on the homomorphic signcryption mechanism we designed. In our framework, each client can aggregate the gradients securely and verify the integrity of the aggregated results. Besides, our framework can also resist the collusion attacks between the server and at most clients. Finally, experiments on four benchmark datasets show that our framework can protect the privacy and integrity of the learned model while guaranteeing its performance, and our framework is more efficient in computation and communication than existing similar frameworks. In future work, we will try to design more flexible privacy preserving framework that allows dynamic joining in and dropping out of clients.
Data Availability
MNIST dataset is available at https://yann.lecun.com/exdb/mnist/.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work was supported by Innovative Research Team in Engineering University of PAP (KYTD201805) and Natural Science Foundation of Shaanxi Province (2020JM-537).