Abstract
Several emerging issues concerning the development of interactive learning environment were left unsolved although e-learning has been applied for years. With several studies indicate that more interaction between students and systems increases students’ level of interest and allows them to focus on learning support. Due to the way current interactive learning tools are designed, users have to wear or operate actual tools in order to carry out the required learning procedures. The use of tools for long durations of time results in user fatigue. Hence, this study incorporates the Microsoft Kinect as interactive tool for detecting gestures in the e-learning process. This study also uses the interaction method that we had developed on Facebook to interact with the proposed learning system. The experiments in this study are divided into five parts: system performance of the 3D web engine, gesture accuracy, system and gesture usability, system and gesture satisfaction, and learning satisfaction of the learner. Also, the gesture design was accepted by learners when they interacted with the learning system. Our research shows that our concept as well as the features of our system can fully support social learning and enhance interaction between users in learning environments.
1. Introduction
Social networks give structure to complex relations between individuals or organizations. There are many websites building social networks, such as Facebook, Twitter, and YouTube. Facebook is the most popular social network website in current internet world. Virtually everyone has a Facebook account and people use it every day to keep connected with their social circles [1]. Furthermore, social learning has always been the most natural way to learn new things. Through discussion, interaction, and collaboration between students, teachers, or student groups, students can construct related knowledge or experience via process of social interaction. However, traditional e-learning systems usually have less communication functionalities for supporting social learning [2]. They mainly play the role of a teaching assistant, which enables teachers to share learning content and manage classes as well as enables students to receive learning content from the system. Available methods for discussing problems or enhancing other types of interaction in traditional e-learning systems include only e-mail, short messaging, chat rooms, or forums in some advanced systems [3]. The interaction usually appears between teacher and students for questions and answers, seldom between students for problem discussions or idea brainstorming. It cannot provide an environment for fully supporting social learning. To enhance interaction and make the learning experience more diverse, we try to utilize the power of social networks. As an extension of traditional communication networks, social network sites enable users to share and discuss their interest and ideas in the network [4]. It makes users become closer to their friends and have more interaction in their daily lives. In order to utilize the power of social networks, we integrate our e-learning system with social networks to create a new learning environment, learning in social networks. From the perspective of learning, in our system, students and teachers can engage in learning activities as usual as well as share learning-related information on the social network sites through the social functionalities integrated into our system. From the perspective of social networks, user can see others’ learning information as well as enhance interaction and discussion on learning topics, which achieves the goal of social learning. For some advanced functionalities, we first collect users’ social and learning information and then build a recommendation system based on that information. Then we build a tool for information visualization, called the Interactive 3D Social Graph, based on 3D and Kinematic technologies, which makes complex data more readable and clear.
The purpose of this research is to design a tool for visualizing social network data from the famous social networking website Facebook as well as propose a new interaction and navigation technique that uses Kinect to explore and interact with social networks [5]. The tool as well as the new method of interaction should help users interact and explore their social networks and also help learners experience better learning interaction. The purposes of this study are as follows:(i)visualize data obtained from social network websites;(ii)design a suitable and comfortable gesture for exploring and interacting using Kinect;(iii)integrate gestures to enhance learners’ level of interest and interaction in social networks.
In Section 1 of this paper, we briefly introduce the proposed interactive tool as well as its integration with e-learning. In Section 2, we introduce several types of gesture-based interactions. In Section 3, we discuss the methods used to visualize data, design interaction, and conduct exploration. In Section 4, we describe the evaluation as well as discussion of our tool and integration efforts. Section 5 is the conclusion and description of future works.
2. Related Works
2.1. Importance on Information Visualization for Interaction
Information visualization is the study of visual representations of abstract data to reinforce human cognition. The visualization of social network data as a graph has a long tradition in the social networks studies in social science. Freeman [2] gives the example of using shape, color, position, and the size of nodes to be the main features for presenting distinct view on the data. Many social networks visualizations researches applied for researcher analysis tool instead of provide it for end-users.
In recent years, with the growth of social network sites, projects about social networks visualization for the end-users showed up. Heer came out with their tool called Vizter tool [6]. Vizter tool does the visualization of one of the famous social network website, Friendster. They provided exploration tool for the users to play around their social network. The features such as connectivity highlighting, link-age views for viewing network content, X-ray mode, and profile search for exploring member profile data and visualization makes this tool is more explore-based play than analysis. The other project came from Matta and Pfeiffer [7]. They proposed social networks visualization tool for Facebook. In their research they studied the comparison between 3D visualization of social network and 2D visualization. The result showed that the visualizations of social networks can be more effective if 3D is utilized. We present tool for 3D visualization of integrated e-learning system and social network.
2.2. Application of Gestures Interaction
A gesture may be a motion of the body that contains several pieces of information [6]. It is a method of nonverbal communication in which visible bodily actions communicate particular messages, either in place of speech or together and in parallel with spoken words. Gestures include movement of the hands, face, or other parts of the body. Gestures differ from physical nonverbal communication that does not communicate specific messages, such as purely expressive displays, proxemics, or displays of joint attention [8]. Gestures allow individuals to communicate with a variety of feelings and thoughts, from contempt and hostility to approval and affection, often integrated with body language in addition to words when they speak [9]. There are two types of gestures, which are distinguished [10] below:(i)offline gestures: those gestures that are processed after the user interaction with the object. They occur when a certain action has already occurred. An example is the gesture to activate a menu;(ii)online gestures: direct manipulation gestures. They are used to scale or rotate a tangible object.
With the growth of technology nowadays, it is possible for us to perform gestural interaction even without physical contact with the device or object, which is called touch-less gestural interaction. Touch-less gestural interaction enables users to explore and control multimedia information space and/or digital devices through body movements and gestures, without the burden of physical contact. Interaction is said to be touch-less if it can take place without mechanical contact between the human and any part of the artificial system. Touch-less gestural interaction can be multimodal. Touch-less gestural interaction in-the-small has received marginal interest by the HCI community. When using touch-less interaction technologies, effort and time can be saved [11]. From all of those devices, Kinect is the most suitable and more natural for interaction with 3D space, due to the capability of Kinect to detect whole body movement through skeletal-based algorithms. Their own body movements and gestures combined with the degree of familiarity of humans with 3D space in real world makes it easier to control the 3D space in more natural ways. Right after the Kinect was introduced in 2011, Kinect gesture-based interaction research began to grow rapidly. It is used in many areas, such as medical [12], education [13], and rehabilitation [14]. There is already a great deal of research pertaining to interaction with 3D space itself [15], gesture interaction for architectural design in 3D, and using gestures to explore 3D space [16, 17].
2.3. Towards a Highly Interactive Social Learning Paradigm
Social learning theory states that social behavior (any type of behavior that we display socially) is learned primarily by observing and imitating the actions of others. In the process of learning social behavior, we may construct corresponding knowledge. Also, social constructivism, proposed by L. Vygotsky, says that knowledge is constructed while learner interacts or cooperates with other social members in the society, instead of one-way instruction. In light of this, while we have social behavior, we are forming some knowledge at the same time.
Distributed constructionism [18] puts emphasis on the idea that, in the activities of design and construction, there needs to have multiparticipant to construct sharing and discussion, especially in the online learning environment, which takes knowledge and cognition as the result of the interaction between individual and elements in the online learning environment. Besides, Resnick thinks that, to learn effectively in social environment, it is necessary to have collaboration to design and construct meaningful knowledge and products. According to the theories, the core concept of social learning is having collaboration and interaction with others, and constructing related knowledge at the process of those social behaviors. The concept makes individuals absorb knowledge and learning experience in social behaviors, and, different from traditional e-learning, learner can receive complete learning participating experience from course constructing to after school assessment, especially in individual learning scenery. Other research [19] showed that interactive learning and collaborative learning can give user higher learning motivation and better learning efficiency than individual learning.
3. The Scenario
The scenario of proposed work can be simply summarized as follows:(1)one open application powered by script language is applied to facilitate the sign-in process of user’s own Facebook account;(2)the same application is then sent an acknowledgement to trigger the Facebook Graph API, which is utilized to obtain user data such as personal information, friendship, and applications, after authentication;(3)by using user ID and access token, system can make a call to get the data that we need for our system and visualized it through Unity3D engine on Web player;(4)after performing the visualization of the Facebook data into 3D graph based user interface, user can interact and explore it by using Kinect hand gesture through Kinect skeleton tracking that provide by Kinect
3.1. Data Acquisition
In order to visualize the data on Facebook, we first need to get the data from Facebook. As we mentioned before, Facebook has launched a software environment that provides third-party developers with the ability to create their own applications and data services on Facebook. With this service, we can easily integrate our application with Facebook or allow our application to directly access data from Facebook. This platform offers a set of programming interfaces and tools, such as social plugins, login capabilities using Facebook, Open Graph, and an SDK for accessing Facebook APIs. In order to access and use the data on Facebook, we need to create an application for obtaining the App ID/API key as well as the App Secret. These two methods are required for the SDK to obtain permission to fetch data from Facebook. There are several SDKs provided by Facebook for accessing the Facebook API, such as the Facebook SDK for PHP, the Facebook SDK for JavaScript, the Facebook SDK for iOS, the Facebook SDK for Android, and SDKs for other platforms or languages. In this study, we use the Facebook SDK for JavaScript to access data from Facebook since our application is a web application and also because it will integrate with Unity3D for 3D visualization.
In order to visualize user social networks and implement various functions from Facebook in our system, we define the types of data we need for our system below:(i)user data:(a)list of friends;(b)personal data (about users);(c)list of family members;(d)list of photos and albums;(e)data regarding users’ posts, statuses, and feeds;(ii)friend data:(a)personal data;(b)list of photos and albums;(c)data regarding friends’ posts, statuses, and feeds.
In order to obtain the data that we have listed above, we will need the user to login through our application using their Facebook account and give permission (authentication) for us to access the data. It will show up in form of a web dialog interface right after we successfully log-in from our application. The permission list is as follows:(1)email permissions: email is a protected property and access to it must be specifically asked for and granted;(2)extended permissions give access to more sensitive info and provide the ability to publish and delete data;(3)extended profile properties: nonoptional permissions for access to a user’s data and that of their friends;(4)open graph permission allows your app to publish actions to the Open Graph as well as retrieve actions published by other apps;(5)page permissions: permission related to management of Facebook Pages;(6)public profile and friend list are basic information types available to an app.
3.2. Design of Learning Support Mechanisms
One of the goals of information visualization is to visualize information, such as social information in our system, transforming textual descriptions to visual representations, thereby facilitating the perception and handling of hidden structures from underlying datasets. We choose a graph-based layout to visualize the information because it is widely used to depict data in which information is comprised of objects and relationships between those objects, such as relations between users. In our system, we choose to use 3D-based graph layouts because it provides more space for future growth of the graph and because humans are more familiar with 3D in real world compared to 2D, which will make exploration easier. The drawing of graphs is another challenge in information visualization. We use the Spring-Embedding (force-directed) Algorithm to draw graphs. This algorithm calculates the layout of a graph using only information contained within the structure of the graph itself, rather than relying on domain-specific knowledge. Graphs drawn by this algorithm tend to be aesthetically pleasing, exhibit symmetries, and are free of crossings. Although this algorithm is for 2D graphs, with some modifications, it still can be used for 3D graph layouts. In this system, we design several interaction and navigation features, mouse-keyboard interactions, and Kinect interactions to explore graph. For both types of interactions, we use common actions to explore 3D space, such as zooming and rotating. With mouse-keyboard interaction, we use a common method for control, such as a scroll button for zooming, right-click-hold for rotating, left-click-hold for panning actions, and W, A, S, and D for keyboard actions. We use Kinect because by using our own body movements as a controller, it is easy for users to perform control. It also provides a more immersive and realistic experience while using our system.
The other reason why we use Kinect interaction is because, in the future, we want to combine this system and virtual smart classrooms with Holodeck technology [20]. We have designed several gestures for exploring and interacting with the 3D graph. These actions are as follows.(i)Zooming action: the Kinect gesture for this action is conducted in two-handed mode, where both hands, left and right, are placed in front of the body and moved farther apart from each other on the x-axis to perform a zoom-out and moved closer to each other to perform the zoom-in action.(ii)Rotating action: the Kinect gesture for this action is conducted in two-handed mode by holding the right hand higher than the waist and with the left hand moving on the x-axis to control the rotation. Moving to the left causes the graph to rotate clockwise and moving it to the right causes graph to rotate counter-clockwise.(iii)Panning action: to perform this interactive action on Kinect, the user must first be in one-handed mode, placing the left hand in front of user’s body and holding it in place for a few seconds to enter panning mode. After entering panning mode, you can move your left hand up, down, right, and left to perform the panning action.(iv)Select object action: to perform this interactive action on Kinect, the user can use the object picking action by moving one hand to the object and changing the moving axis from x-y axis to z-axis. When an object is selected, the content of the object will be shown on screen.(v)Operations and features: our system provides several features. One of the features is called Social Graph, shown in Figure 1. This feature contains the graph of the current user as well as his social relationship. The current user node is displayed as a blue sphere, normal friends are presented as green sphere shapes with blue lines, acquaintances are displayed as box shapes with yellow lines, and best friend are represented as star shapes with red lines. Node information, including user ID, name, and taken-course list, are displayed and highlighted when the node is selected. Another feature is called Course Graph, which provides a course tree that contains learning history and scores for each lesson. This feature can help students to track their course records and histories.
We also provide the focus operation and the recommendation feature. The focus operation, shown in Figure 2, is basically the graph filter feature which can change the graph layout according to user’s keywords. It makes the graph clearer to see and helps users find what they want. We designed three filter attributes for this feature, as described below.(1)The friend similarity operation is a feature for finding similarities between a user and the user’s friends on a learning domain according to keywords. The system will change the graph node’s position and the greater the similarity the closer it will be to the user [21]. Nodes with no similarity will be removed. To make it clearer for the user, the line edge color of the user and their friends also changes based on similarity.(2)Same course operation: using this feature, users can find friends with similar learning progress in a specific course. Nodes will be presented with different distances and colors based on learning progress. Friends that are currently on the same lesson as the user will be placed closer to the user. Other friends will be placed at the same distance and be given the same color line based on the lesson they took.(3)Friend recommendation is a feature for helping users find new friends based on similarity of learning (courses taken, majors, and current learning progress). If people have equal or close similarity with the current user and are not friends with the current user, they will show up in the list of friend recommendations.(4)The goal of course recommendation is to give users suggestions regarding the next course that they can take based on their learning progress. This feature allows the user to get recommended courses from the system or from their friends. The list of course recommendations will be shown in the list (see Figure 3).
(a)
(b)
Figures, see Figures 4–8, are the proposed Kinect hand gesture implementation. As we discussed in the previous section, we divide the hand gesture interaction into two parts, exploration part and UI part. In the exploration part, user can perform zoom, panning, rotate, full screen, hide menu, mouse, click, and filter gesture. In the GUI mode user could only perform mouse, click, swipe up down, scroll up, down, and close gesture. To perform the gestures, first user needs to activate the Kinect mode by click the Kinect menu button (pink one) in the main view of our system. User needs to stand up in the front of Kinect device about 1,5 meters and system will detect and label all of the parts of human body by using skeleton tracking. Then user can start to perform the gesture to control the 3D graph and other UI in our system application. Here in Figures 4, 5, 6, 7, and 8 you can see our gestures implementation to explore and interact with the 3D graph based interface for social networks.
4. Implementation of a Facebook-Empowered Learning System
Current e-learning systems could only play the role of teaching assistance and seldom provide the social learning functionalities which makes learning not different from traditional classroom learning environment, where student feels bored during the learning time. In this research, the main goal of our system is using the power of social network site to provide an open, online social learning environment that gives student the ability to have social interaction and learning behaviors on Internet at the same. The following are the specific goals of this research:(i)integrate e-learning environment and Social Networks environment;(ii)open course ware is a pure online learning environment, and the choice of courses or social relationships becomes one problem for user to face. We acquire and analyze user’s learning portfolio from e-learning environment with social information and relationship from social networks environment, to build a recommendation system both in social networks and in learning activity parts and to help user to solve the problem of course and friendship choice.
In general, learning management system can be separated into two aspects.
4.1. Instructor Functionality
Instructor functionality is for teacher to manage learning related activities and corresponding content files; we can basically classify them into the following categories.
These functionalities can help teacher to manage the course or class state and give him the ability to monitor students or whole class learning situation, which can help teacher to adjust their instruction way or style, according to the condition of monitoring subjects.
4.2. Learner Functionality
Learner functionality is for student to participate learning activities and corresponding supportive and recording functionalities. We can basically classify them into the following categories.
These functionalities can help student access learning content, including course pages, assignment or assessment, and their related activities/behaviors support. One of the main functionalities of learning management system is the learning activities/behaviors monitoring and recording, and it will construct the learning portfolio of the student. With the help of learning portfolio, student can have self-examination on their learning result and adjust their learning attitude and skill to get better learning result or achieve the goal of learning topic, so do teacher in the same manner in instruction part.
The design of this learning management system focuses on the learner functionality aspect, and the functionalities are mainly to raise student’s learning activities. We design the system with the e-learning standard SCORM (Sharable Content Object Reference Model), which prompts it to use the sharable SCORM learning content packages and display the learning content to student on the web browser directly. We separate the learning content to several categories that represents different learning domains, and each category contains several courses. Student could choose the category that he is interested in and read the containing courses. The courses in specific category are basically independent for the reason that, in the Open Course Ware online learning environment, the course content comes from different teacher in different universities and it is hard to integrate them into a complete learning procedure. The courses in the specific category are related to some specific learning domain but, basically, are independent from each other and seldom have before-and-after relationship, to give the students the most freedom to choose the course they are interested in.
5. Experiment and Empirical Studies
To evaluate our system, we divide several experiments as discussed below. At first, we evaluate the performance of 3D graph based social network visualization tool on the browser. We checked the performance by using different numbers of friends on Facebook and chose some users with 50 friends until 800 friends (the average Facebook users have at least 700) and implemented on Chrome browser. See Table 1.
Results indicated that the number of 3D nodes with more than 800 and edge with total of 3D object rendered showed 1600 3D objects; this results in good performance around 23–25 frame rate per second on the browser since normal rate for flash player around 30 fps for addition; it also runs with Kinect devices and interactions. The next evaluation was to evaluate the accuracy of hand gesture recognition of proposed system. We do testing by performing each of the gesture interactions for 60 times per gesture. The result of gesture recognition is evaluated as in Table 2.
From the results of the total recognition for our hand gesture is 91% of recognition. It indicated that select gesture is the lowest recognition because of position of the nodes sometime it make it hard to perform the gesture for example, nodes position is to lower or in the corner of the screen.
We conducted the next experiment to check the user satisfaction of the system and the gesture design by system evaluation and questionnaire. The study was conducted at National Central University and collected from students. There are 83 participants that are Facebook users with at least using Facebook almost every day for 3 years.
After we test our system there are several results that we get from the experiment. From the experiment that we conduct, for the gesture part found, almost all of the gestures were easy to perform and easy to learn. But for some gestures, especially for the rotate gesture, it was found to be a little bit hard to perform. All of gestures were not awkward to use, but some of gestures made the user feel fatigue, for the zooming gesture. They said it was slow and needs to perform in the same position for a long time. The user said the gestures help the system well, especially for the filter gesture and close gesture. Those make it easy for the user, just like a shortcut. Overall they are satisfied with the gesture. Some of them also give some suggestions about the gesture design. For the system, overall, it is good, easy to use, comfortable, easy to learn, and they also really like the user interface theme. But because of the system only implement some few of the Facebook functions, they cannot stay in the system any longer. When we ask about the visualization can make them easy to explore their social network, they agreed. It makes it easy for them to see their social network in the physical ways. The 3D visualization combined with the gestures makes it more immersive, real, natural, and easy to explore their social network. Overall they satisfy the system, but some of them are confused about the use of the Kinect. When we explain it that in the future it will be improved and used in the smart room star trek holodeck like, they support and agreed that in the future they willing to use it.
6. Conclusion and Future Works
In this research, we proposed an interactive 3D Facebook visualization system using Kinect as a new interaction and using this interaction to integrate with the developed learning system, an open course ware, which is based on the integration of e-learning system and social network site Facebook, to provide a social learning environment. Our main objective of this paper was to provide a 3D visualization tool to social networking site (Facebook) and design the way of using new motion capture technology, Kinect, for interaction and exploration of our 3D visualization system. The results of the experiment indicated that our system had good using satisfaction and was easy to use, enhance learner’s interest, and help to visualize the social interaction. Otherwise, the proposed 3D visualization makes the user more immersive and naturally interact with the system. Also, the gesture design also helped user to interact and explore the system. But some of the gestures made the user feel tired of fatigue problem (zooming gesture), and our research will improve gesture design to overcome the problems. Overall the system and the gesture design performed well; also we can say that we nearly achieved the goal of this research. For the social learning system, the social behavior is an important issue in social learning, and we can have further analysis here. Moreover, we can integrate more social functionalities in our learning management system, like Facebook club for learning group or online forum for discussion; then we can have more records about social learning behaviors. By doing this, we can take the social interaction result or outcome of the user, like participation rate and interaction times, to build a social learning evaluation model to analyze user’s social learning behavior in a quantitative way, which can also be taken into consideration in the recommendation system. Adding this feature we can generate the recommended friends or users who are hospitable to help people and willing to join the social learning interaction.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.