- About this Journal ·
- Abstracting and Indexing ·
- Aims and Scope ·
- Annual Issues ·
- Article Processing Charges ·
- Author Guidelines ·
- Bibliographic Information ·
- Citations to this Journal ·
- Contact Information ·
- Editorial Board ·
- Editorial Workflow ·
- Free eTOC Alerts ·
- Publication Ethics ·
- Recently Accepted Articles ·
- Reviewers Acknowledgment ·
- Submit a Manuscript ·
- Subscription Information ·
- Table of Contents
International Journal of Distributed Sensor Networks
Volume 2013 (2013), Article ID 481580, 10 pages
Physical Activity Recognition Utilizing the Built-In Kinematic Sensors of a Smartphone
The Key Lab for Health Informatics of Chinese Academy of Sciences, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China
Received 7 December 2012; Revised 27 February 2013; Accepted 12 March 2013
Academic Editor: Shijian Li
Copyright © 2013 Yi He and Ye Li. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Physical activity (PA) recognition has recently become important in activity monitoring for the public healthcare. Although body-worn sensors are well suited to collect data on activity patterns for long periods of time, users may forget to wear special microsensors. On the contrary, more and more people take smartphone with them almost anytime. At present, most popular smartphones have three built-in kinematic sensors (triaccelerometer, gyroscope, and magnetic sensor) which could be utilized to recognize PA. This study utilized three built-in kinematic sensors in a smartphone to recognize PA and found out which features derived from the three sensors were significant to different PA. We used a combined algorithm of Fisher’s discriminant ratio criterion and criterion for feature selection. A hierarchical classifiers system including fourteen classifiers was proposed and employed to recognize fifteen activities. The optimal features derived from the built-in kinematic sensors of the smartphone were selected from 140 features. The results indicated that the accelerometer was significant to PA recognition, while gyroscope and orientation sensor were effective to recognize the change of body posture and detect falls, respectively. The total classification accuracy of 95.03% demonstrated the feasibility of utilizing the built-in kinematic sensors of the smartphone to recognize PA.
Physical activity (PA) recognition has recently become important in activity monitoring for the public healthcare. The progressive decline in the PA level due to the adoption of sedentary lifestyles has been associated with the increasing incidence of obesity, diabetes, and cardiovascular diseases [1, 2]. In addition, a “long lie” caused by a fall may result in negative psychological and physical outcomes . The fear of falling can lead to a further decrease of physical activity. Thus, a reliable PA recognition system in daily life would not only encourage people to do more outdoor activities but also be an assessment of activities of daily living for chronic treatment.
Body-worn sensors [4, 5] are well suited to collecting data on activity patterns for long periods of time. In contrast to other approaches, such as laboratory-based systems  or video analysis [7, 8], they can be used under conditions of free living with minimal inconvenience to the user . However, users may forget to wear these special microsensors. At present, smartphone is a fast-growing device which people take with it anytime. Most of state-of-the-art smartphones have the built-in kinematic sensors (such as triaccelerometer, gyroscope, and magnetic sensor), Global Positioning System (GPS), Wi-Fi, Bluetooth, camera, proximity sensor, microphone, and so forth.
The built-in kinematic sensors of smartphones are used to recognize PA, which not only reduces the cost of hardware but also exploits existing communication module and ubiquitous monitor. The previous studies have demonstrated the effectiveness of using a smartphone to detect a fall . Acceleration was measured by two devices, a smartphone and an independent accelerometer, to confirm that fall detection using a smartphone is a feasible and highly attractive technology.
Most of the previous studies [10–12] only used accelerometers to research PA recognition. In our earlier study , we also utilized accelerometer built in a smartphone to detect falls and raised the alarm by Multimedia Messaging Service (MMS). However, acceleration only reflects the change of PA in linear acceleration, but angular velocity and orientation information were not reflected by an accelerometer.
There are several studies using the combination of accelerometer, gyroscope, and magnetic sensor. Zhang et al.  evaluated a continuous activity recognition system, using three small wearable wireless sensors and a smartphone to store, transmit, analyze, and update data. Three sensors, each composed of a triaxial accelerometer and a triaxial gyroscope, are attached to the chest and both thighs. The recognition algorithm was developed using a decision tree based on time series data and spectrum analysis. Bahle et al.  found a particularly interesting result that adding the magnetic disturbance features significantly improves recognition based on the vector norm of accelerometers and gyroscopes. Chiang et al.  measured essential movements of human body through wireless sensor network body motion sensors that involve the accelerometer and gyroscope. Activities are recognized by the fuzzy algorithm. Jeannet et al.  used a monitor containing a 3D accelerometer and a gyroscope fixed on the subject’s tee-shirt, on the chest. Multiple sensors systems could identify physical activity in a more refined way by recognizing subactivities such as jogging or rowing, due to the fact that a bigger amount of information on the body motion can be collected. However, the more the sensors were employed to record physical activity, the higher the interference of the measuring system with the spontaneous behavior of the user . Therefore, the development of simple, small and light-weight systems to monitor physical activity in daily life should be recommended.
The aims of this study were to utilize the built-in kinematic sensors (triaccelerometer, gyroscope, and magnetic sensor) of a smartphone to recognize PA and find out which features derived from the built-in kinematic sensors were significant to recognize PA. This work will make the assessment of activities of daily living and fall detection more ubiquitous and portable.
The rest of the paper is organized as follows: Section 2 provides the method of the PA recognition. The experimental results and discussion are presented in Sections 3 and 4. Finally, Section 5 concludes the paper.
The block diagram of the main data processing scheme was described in Figure 1. The built-in accelerometer, gyroscope, and the orientation sensor of a smartphone collected information which reflected acceleration, angular velocity, and orientation of PA. And then features were extracted from the information, and the optimal features were selected. Subsequently, all activities were classified, and if a fall was detected, the alarm information about the location of the fall would be sent to the preset person by MMS.
2.1. Device and Data Collection
A smartphone (Samsung I9023 Nexus S, mm, 129 g, Android operating system version 2.3.6, Samsung, Korea) was placed inside an adjustable band attached to the chest of a subject, as shown in Figure 2. The activity recognition system requires firm attachment of the smartphone to the subject.
The smartphone has built-in triaxial accelerometer (STM KR3DM) with 19.6 m/s2 maximum range and 0.019 m/s2 resolution, triaxial gyroscope sensor (STM K3G) with 34.9 rad/s maximum range and 0.0012 rad/s resolution, and triaxial magnetic field sensor (Asahi Kasei AK8973) with 2000 μT maximum range and 0.0625 μT resolution. An application software of PA recognition was developed and installed on the smartphone. This application measured the acceleration, angular velocity, and orientation of the smartphone.
The standard sensor coordinate system of the smartphone is defined relative to the screen. As shown in Figure 2, the -axis () is horizontal and points to the right. The -axis () is vertical and points up, and the -axis () points toward the outside of the screen. The , , and of smartphone are according to the mediolateral, vertical, and dorsoventral directions of the subject, respectively.
The coordinate system of the accelerometer is the same as the standard sensor coordinate system. The accelerometer measures acceleration along (), (), and (), respectively. The low pass filter with 0.25 Hz cutoff frequency was employed to separate acceleration force to gravity acceleration (GA) and linear acceleration (LA). So was separated into and , the same as and .
The coordinate system of the gyroscope is the same as the one used for the accelerometer. Rotation is positive in the counterclockwise direction. Gyroscope measures the rate of rotation around (), the rate of rotation around (), and the rate of rotation around () in radian/s.
The orientation sensor is software-based and derives its data from the accelerometer and the geomagnetic field sensor. The orientation sensor monitors the position of the smartphone relative to the earth’s frame of reference and measures azimuth (), pitch (), and roll (). is the angle between magnetic north of the earth and the positive of the smartphone. When is aligned with magnetic north, the value is 0, and when is pointing south, the value is 180. is the angle between the horizontal plane of the earth and the positive . is positive when the positive rotates toward the positive , and the range of values is from 90 degrees to −90 degrees. is the angle between the horizontal plane of the earth and the positive . is positive when the positive rotates toward the positive , and the range of values is from 180 degrees to −180 degrees.
In Android operating system, four different sampling frequencies (fastest, game, normal, and UI) of sensors can be selected. The values of the four frequencies are not constant and depend on the computational workload of the smartphone. The fastest sampling frequency is selected and can reach 50 Hz normally. The acceleration, gyroscope, and orientation signals were sampled at 25 Hz and stored on a Secure Digital (SD) card in the smartphone. Table 1 showed the descriptions of activities performed by a subject. Ten healthy volunteers (6 males, age: years, body mass index (BMI): kg/m2 and 4 females, age: years, BMI: kg/m2) participated in the study.
2.2. Feature Extraction
The sliding window approach was employed to divide the sensor signal into smaller time windows (Figure 3). At a sampling frequency of 25 Hz, each window with 50% overlap represents 1.6 seconds. The features were extracted from the sliding windows signals for activity recognition. (1) Signal Magnitude Vector is the signal magnitude vector of linear acceleration. can be represented as Likewise, and have used the same computational method. (2) Tilt angle (TA) of body trunk TA is defined as the angle between the positive and the gravitational vector, as follows: Twenty signals derived from the acceleration, angular velocity, and orientation are listed as follows: , , , , , , , , , , , , , , , , , , , and TA.
As shown in Table 2, seven kinds of statistical features were defined and described. And all statistical features of each signal were calculated at each window. So 140 features were extracted from the acceleration, angular velocity, and orientation measured by the built-in kinematic sensors of the smartphone. represents the average value of over the window, and the remaining 139 features are defined similarly (Table 3).
2.3. Feature Selection and Classification
A combined algorithm of Fisher’s discriminant ratio (FDR) criterion and criterion was used for feature selection . Firstly, normalize the features to have zero mean and unit variance. Then the features were ranked by the scalar feature selection, which employs FDR criterion and a cross-correlation measurement between pairs of features. FDR is commonly employed to quantify the discriminatory power of individual features between two classes. The FDR is defined as where , are the mean values of features and , are the standard deviation of features in the two classes, respectively.
Then the 10 highest-ranked features were selected. And the exhaustive search method with the criterion was employed to select the best combination of two features. The criterion is done by the following equations: where is the within-class scatter matrix and is the between-class scatter matrix. The respective definitions are where denotes the a priori probability of th class Ci and is the respective covariance matrix of class Ci. Moreover, is the global mean vector.
Furthermore, large values of indicate that classifiers have small within-class variance and large between-class distance.
A hierarchical classifiers system was shown in Figure 4. According to level of intensity of physical activity, Classifier 1 was used to differentiate between static and dynamic activity. Likewise, Classifier 3 and Classifier 7 utilized the level intensity of activity to recognize high level of intensity of physical activities, such as falls, jumping, and running. In addition, Classifier 2 employed tilt angle (TA) of body trunk derived from accelerometers to identify lying from sitting/standing.
Moreover, Classifier 5 was employed to classify jumping and falls. Falls are said to have occurred if at least two consecutive peaks in the signal magnitude vector above a defined threshold are recorded . According to the orientation, falls were divided into the categories of right-side fall (FR), left-side fall (FL), backward fall (FB), or forward fall (FF).
Furthermore, there were no obvious features to classify some certain activities, such as walking, walking downstairs and walking upstairs. In a classifier, all possible combinations splitting the activities were exhaustively formed, and for each combination its value was calculated. The combination with the largest value was chosen to be the scheme of Classifier 6. Likewise, Classifiers 4, 10 used the same method to choose the combination. Fourteen classifiers for two classes were employed to recognize fifteen activities.
3.1. The Features Selected
The scatter graphs of part classifiers were shown in Figure 5. The symbols of pluses and circles indicated points into the two classes of classifiers, respectively. In addition, Figure 5 pointed out the combinations of selected features resulted in well separated classes in the 2-dimensional feature space.
As shown in Table 4, twenty-eight well selected features for fourteen classifiers included nineteen features derived from accelerometer, six features derived from gyroscope and three derived from the orientation sensor.
In these three sensors, accelerometer was dominant to PA recognition. In addition, gyroscope served well in recognizing the change of body posture such as transitions, and the orientation sensor was effective to detect falls. As illustrated in Figure 6, can be used to distinguish between forward fall and backward fall. Likewise, can be used to distinguish between left-side fall and right-side fall.
3.2. The Application of PA Recognition in the Smartphone
Figure 6 illustrated the application of PA recognition implemented in the smartphone when a subject changed the motion from motionless to walking. The green line indicated the SVM value of the accelerometer in real time. Five common activities (jumping, running, walking, lying, and motionless) can be represented by the blue line.
3.3. The Recognition Results for All Activities
We used hold-out method  to estimate the classifiers, which partitions the data into two mutually exclusive subsets: a training set and a test set. A set of experiments was performed with 10 subjects, with each performing a set of 16 different activities as described in Table 1. 600 windows were randomly divided into two parts: a training set (400 samples) and a test set (200 samples).
The confusion matrix in Table 5 was a representative of recognition errors for the test set. From the confusion matrix, we can see that walking may be misclassified as walking upstairs or walking downstairs, because these activities contain similar movements and frequencies at the lower limbs. In addition, falls may be recognized as sit-to-lie, because the posture of the two activities were lying at last. The accuracies of static, transitions, dynamic, and falls were 98.75%, 94.625%, 91.8%, and 97.63%, respectively. A total recognition rate of 95.03% achieved that our proposed PA recognition system based on the smartphone could accomplish high recognition rates for the fifteen activities. The application of PA recognition was displayed in Figure 7. The green line indicated the SVM value of the accelerometer real time, and the blue line presented the PA.
A PA recognition system based on a smartphone was developed. The optimal features derived from the built-in kinematic sensors of the smartphone were selected from 140 features for each classifier.
In the previous studies [22, 23], the recognition rates of walking, walking upstairs, and walking downstairs were low, caused by the similarity of these three activities (Figure 8). In this study, the three classes were well separated. Figure 9 presented the scatter graphs of three activities (walking, walking upstairs, and walking downstairs), employing a combination of Std, Std, and Std. Pluses, circles, and dots indicated points from walking, walking upstairs, and walking downstairs, respectively.
The PA recognition system based on the smartphone provided several advantages, such as existing communication module and ubiquitous monitor. The previous studies have used Short Messaging Service (SMS) or Multimedia Messaging Service (MMS)  to send information of a fall to a guardian. In an earlier study, we warned falls of the elderly to a guardian by MMS including time, map of suspected fall location, and GPS coordinate. When a fall was detected, an automatic MMS would be sent to preset people. According to the convenient communication module of smartphones, more information of PA could be sent in real time. A remote PA recognition system based on a private cloud platform  will be developed in our future work.
The main limitation of our approach is that the smartphone must be worn on the chest. People put phones in the pocket of shirts or pants according to their respective habits, which would cause loose attachment between the body and smartphone resulting in the invalidness of the algorithm eventually. Khan et al.  used linear discriminant analysis to recognize PA based on a single triaxial accelerometer that allowed subjects to carry the sensor freely in any pocket without firm attachment. In future work, we plan to develop a PA recognition system based on a smartphone according to wearing habits of users.
For this study, the optimal features derived from the built-in kinematic sensors of the smartphone were selected to recognize PA. The results of selected features suggested the accelerometer was significant to PA recognition, and gyroscope and orientation sensor were effective to recognize the change of body posture or detect falls, respectively. Despite the limitations of this system which requires firm attachment of the smartphone to a subject, the experimental results demonstrated the feasibility of utilizing the built-in kinematic sensors of the smartphone to recognize PA.
This work was supported in part by the National S&T Major Project of China (no. 2011ZX03005-001), Guangdong Province and Chinese Academy of Sciences Comprehensive Strategic Cooperation Project (no. 2011A090100025), and National Natural Science Foundation of Youth Science Foundation (no. 31000447).
- U. Ekelund, S. Brage, H. Besson, S. Sharp, and N. J. Wareham, “Time spent being sedentary and weight gain in healthy adults: reverse or bidirectional causality?” American Journal of Clinical Nutrition, vol. 88, no. 3, pp. 612–617, 2008.
- S. Pinto, M. Ki, T. Pouliou, and C. Power, “Associations between sedentary behaviour and cardiovascular, haemostatic and diabetic markers in mid-adult life (45 years),” Journal of Epidemiology and Community Health, vol. 65, pp. A7–A8, 2011.
- K. Robie, “Falls in older people: risk factors and strategies for prevention,” The Journal of the American Medical Association, vol. 304, pp. 1958–1959, 2010.
- M. Yuwono, B. D. Moulton, S. W. Su, B. G. Celler, and H. T. Nguyen, “Unsupervised machine-learning method for improving the performance of ambulatory fall-detection systems,” Biomedical Engineering Online, vol. 11, p. 9, 2012.
- B. Caby, S. Kieffer, M. de Saint Hubert, G. Cremer, and B. Macq, “Feature extraction and selection for objective gait analysis and fall risk assessment by accelerometry,” BioMedical Engineering Online, vol. 10, article 1, 2011.
- A. Sixsmith and N. Johnson, “A smart sensor to detect the falls of the elderly,” IEEE Pervasive Computing, vol. 3, no. 2, pp. 42–47, 2004.
- Z. A. Khan and W. Sohn, “Abnormal human activity recognition system based on R-transform and kernel discriminant technique for elderly home care,” IEEE Transactions on Consumer Electronics, vol. 57, pp. 1843–1850, 2011.
- W. Lao, J. Han, and P. H. N. de With, “Automatic video-based human motion analyzer for consumer surveillance system,” IEEE Transactions on Consumer Electronics, vol. 55, no. 2, pp. 591–598, 2009.
- S. J. Preece, J. Y. Goulermas, L. P. J. Kenney, D. Howard, K. Meijer, and R. Crompton, “Activity identification using body-mounted sensors—a review of classification techniques,” Physiological Measurement, vol. 30, no. 4, pp. R1–R33, 2009.
- R. Y. W. Lee and A. J. Carlisle, “Detection of falls using accelerometers and mobile phone technology,” Age and Ageing, vol. 40, pp. 690–696, 2011.
- S. Nishiguchi, M. Yamada, K. Nagai et al., “Reliability and validity of gait analysis by android-based smartphone,” Telemedicine and E-Health, vol. 18, pp. 292–296, 2012.
- G. Blázquez Gil, A. Berlanga, and J. M. Molina, “InContexto: multisensor architecture to obtain people context from smartphones,” International Journal of Distributed Sensor Networks, vol. 2012, Article ID 758789, 15 pages, 2012.
- Y. He, Y. Li, and C. Yin, “Falling-incident detection and alarm by smartphone with Multimedia Messaging Service (MMS),” E-Health Telecommunication Systems and Networks, vol. 1, pp. 1–5, 2012.
- Y. Zhang, S. Markovic, I. Sapir, R. C. Wagenaar, and T. D. Little, “Continuous functional activity monitoring based on wearable tri-axial accelerometer and gyroscope,” in Proceedings of the 5th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth), pp. 370–373, 2011.
- G. Bahle, K. Kunze, and P. Lukowicz, “On the use of magnetic field disturbances as features for activity recognition with on body sensors,” in Proceedings of the 5th European Conference on Smart Sensing and Context (EuroSSC '10), pp. 71–81, 2010.
- S. Y. Chiang, Y. C. Kan, Y. C. Tu, and H. C. Lin, “Activity recognition by fuzzy logic system in wireless sensor network for physical therapy,” Intelligent Decision Technologies, pp. 191–200, 2012.
- P. Y. Jeannet, K. Aminian, C. Bloetzer, B. Najafi, and A. Paraschiv-Ionescu, “Continuous monitoring and quantification of multiple parameters of daily physical activity in ambulatory Duchenne muscular dystrophy patients,” European Journal of Paediatric Neurology, vol. 15, no. 1, pp. 40–47, 2011.
- A. G. Bonomi, “Physical activity recognition using a wearable accelerometer,” in Sensing Emotions, vol. 12 of Philips Research Book Series, pp. 41–51, 2011.
- S. Theodoridis, K. Koutroumbas, A. Pikrakis, and D. Cavouras, Introduction to Pattern Recognition: A Matlab Approach, Academic Press, 2010.
- D. M. Karantonis, M. R. Narayanan, M. Mathie, N. H. Lovell, and B. G. Celler, “Implementation of a real-time human movement classifier using a triaxial accelerometer for ambulatory monitoring,” IEEE Transactions on Information Technology in Biomedicine, vol. 10, no. 1, pp. 156–167, 2006.
- L. P. Devroye and T. J. Wagner, “Distribution-free performance bounds for potential function rules,” IEEE Transactions on Information Theory, vol. 25, no. 5, pp. 601–604, 1979.
- D. Maguire and R. Frisby, “Comparison of feature classification algorithm for activity recognition based on accelerometer and heart rate data,” in Proceedings of the 9th IT & T Conference, pp. 1–11, Dublin, Ireland, 2009.
- R. K. Ibrahim, E. Ambikairajah, B. Celler, N. H. Lovell, and L. Kilmartin, “Gait patterns classification using spectral features,” in Proceedings of the IET Irish Signals and Systems Conference (ISSC '08), pp. 98–102, Galway, Ireland, June 2008.
- C. He, X. Fan, and Y. Li, “Toward ubiquitous healthcare services with a novel efficient cloud platform,” IEEE Transactions on Biomedical Engineering, vol. 60, no. 1, pp. 230–234, 2013.
- A. M. Khan, Y. K. Lee, S. Lee, and T. S. Kim, “Accelerometer's position independent physical activity recognition system for long-term activity monitoring in the elderly,” Medical and Biological Engineering and Computing, vol. 48, no. 12, pp. 1271–1279, 2010.