Abstract
The process of producing new creative videos by editing, combining, and organizing pre-existing material (e.g., video shots) is a popular phenomenon in the current web scenario. Known as remix or video remix, the produced video may have new and different meanings with respect to the source material. Unfortunately, when managing audiovisual objects, the technological aspect can be a burden for many creative users. Motivated by the large success of the gaming market, we propose a novel game and an architecture to make the remix process a pleasant and stimulating gaming experience. MovieRemix allows people to act like a movie director, but instead of dealing with cast and cameras, the player has to create a remixed video starting from a given screenplay and from video shots retrieved from the provided catalog. MovieRemix is not a simple video editing tool nor is a simple game: it is a challenging environment that stimulates creativity. To temp to play the game, players can access different levels of screenplay (original, outline, derived) and can also challenge other players. Computational and storage issues are kept at the server side, whereas the client device just needs to have the capability of playing streaming videos.
1. Introduction
In the past few years the usage of video material has largely grown in popularity, fueled by an increasing number of websites designed to share video material. Watching, uploading, downloading, and sharing videos are nowadays common activities in the web scenario. The well-known YouTube, the third most visited website according to Alexa statistics [1], is coupled with several other video sharing sites like Vimeo, MetaTube, and Yahoo! Video, not to mention the high usage of videos in several social network sites like Facebook, MySpace, and Flickr. With no doubt, video applications generate the main source of traffic for the Internet backbone network: it accounts for 90% of the worldwide Internet data traffic [2], and several research studies predict that the popularity of all forms of video material (video on demand, Mobile, Internet, P2P, 3D, and HD) will continue to increase.
The gaming market experiences a similar success. Despite the current economic environment, the video game market continues to report promising results and is expected to have a significant growth in the next few years. Mainly due to the introduction of new devices (e.g., XBox’s Kinect, Nintendo 3DS) and to the turnover of software sales, a research analysis of IDATE [3] predicts that, worldwide, the gaming market will increase from 38 billion EUR in 2010 to 52.3 billion EUR in 2014.
The combination of both scenarios would likely create a success and popular environment. To this aim, in this paper we want to make the editing of a video a gaming experience. Motivated by the large usage of video material, by the success of the gaming market, and by the increasing presence of active users in the Web 2.0, our goal is to propose a MovieRemix game and an architecture able to support it.
Video remix is a popular phenomenon in the current web scenario: several websites with remixed video material (e.g., http://www.totalrecut.com/) are appearing on line, and the word “Remix’’ returns more than 1.5 millions of videos when entered in the YouTube search box. It is worth noting that the remix practice is not a new phenomenon, but what is new is the scale: nowadays multimedia and web technologies allow people an easy access to video material.
Roughly, a remix video is an audiovisual content obtained by editing, combining, and organizing preexisting material (e.g., professional video contents like movies, previews, recaps, or commercials). The remix is usually a creative video with new meanings with respect to the original sources [4]. More generally, a remix may come from catalogs or libraries of music, images, and audiovisual and multimedial cultural products [5]. The author of the remix is a bricoleur [6], and he/she takes advantage of the advent of digital technologies that facilitate the remix practices in many scenarios. For instance, in popular music we have cover versions and remixes of golden oldies as well as new songs. The transformation of the original sources seems to have no limits. The same happens in the video scenario, where remix videos are made from copying or editing several different movies.
The success of the Be Kind Rewind (http://www.youtube.com/user/bekindrewind/) initiative highlights that users want to manipulate professional videos to create personal and customized clips. In essence, using many different sources and remixing them, creative users aim at creating new cultural and artistic products [4].
Unfortunately, when dealing with audiovisual material the technological aspect plays a critical role and can be a burden for many creative users who do not have a sufficient technological background. The raw material of a remixer is usually a set of video shots, and these are obtained either by personally shooting them or by copying them from different sources. The shooting requires a video camera and limits the amount of audiovisual material that can be used (i.e., it is difficult to use a New York background if you do not live in New York City), whereas the extraction of video material from other sources requires knowledge of video editing applications and of low-level video characteristics (e.g., sources may be encoded with different encoding technologies like MPEG, DivX, Flash, etc.), not to mention that copyright laws protect from unauthorized usage many materials available in the web scenario. However, assuming that a user can have access to several video clips, he/she has to edit/organize them in order to create the remixed video. Once again, this process requires knowledge that many users may not have. As a result, a relatively small number of users may express their creativity within the video remix phenomenon.
In this paper we propose a game and an architecture that make the remix experience easy, fun, and pleasant. Similarly to many video games where a user plays a role different from the real one (e.g., soccer player, airplane pilot, music director), our goal is to allow people to act like a movie director, but instead of dealing with cast, camera shot, and so on, our player deals with a catalog of pre-existing video shots and with a catalog of screenplays. The objective of the game is to create a video starting from a given screenplay. Different levels of screenplay (derived, original, and outline) are given to players in order to increase game difficulties. Through a developed graphical application, a player can select the preferred video shots from the MovieRemix catalog, can add music, and transition effects so as to create a new video. Once done, the remixed video can be uploaded, shared, and voted by other players. To temp to play the game, charts and challenges are also possible.
It is worth noting that, to play the game, users are simply required to be creative and to have Internet access: no special knowledge is required as all technical details (encoding format, compression mechanism, storage space, etc.) are hidden from users. Similarly, the architecture is designed to keep computation and storage issues (i.e., encoding and storage of video material) at the server side, whereas the client device can be very simple (the only requirement is to play streaming videos).
To evaluate MovieRemix we set up an experimental scenario. The investigation involved a group of heterogeneous people that were asked to use MovieRemix and to answer several different questions about the proposed game. Using a mean opinion score evaluation, results confirmed that MovieRemix is considered an educational game and that players have almost no difficulties in understanding the given screenplay. Furthermore, results showed that MovieRemix is better using with news and music video remix.
The paper is organized as follows. Section 2 briefly describes related works in the area of video remix; Section 3 presents preliminaries whose reading facilitates the understanding of the proposed architecture; Section 4 introduces game and architecture details; prototype implementation is described in Section 5, whereas the experimental evaluation is shown in Section 6. Conclusions are drawn in Section 7.
2. Related Work
In the literature, several studies analyzed the practice of audiovisual remix from different points of view: sociological, philosophical, analytical, and technological (e.g., [7–12]). In the following, we present approaches related to the technological aspect, that is, proposals designed to facilitate the making of remixed videos.
Early works in the area of video remix propose systems developed to make up for a lack of adequate tools to support collaborative knowledge building around media material or creative thinking.
Pea et al. [11] present DIVER, a system which makes it possible to create an infinite number of new digital video clips and remix compilations starting from a single source video recording. The DIVER project was born at Stanford University to support collaborative analysis of learning and teaching video records in a distributed community of researchers and practitioners. DIVER works like a virtual camera which can zoom and pan through space and time within a video record. The virtual camera dynamically crops image clips to create a dive which is a set of reordable panels, each one containing a thumbnail that represents a clip as well as a text field that may contain an accompanying annotation. The user can upload the created dive to a website for interactive browsing searching and display of video clips and collaborative commentary on them.
Multisilta and Mäenpää [10] propose MoViE, a platform that enables users to create narrations and stories made with the mobile phone and for the mobile phone in a collaborative way and using narrative structures. Applying a narrative structure used in jazz music, a user defines a topic for the story and shoots a short video about this topic; then he/she uploads the video to the system and tags it with appropriate keywords. At this point, like in a jazz concert, another author watches the video using a mobile phone, and, using the video as an inspiration source, he shoots several clips (called solos) to express his ideas about the original video and story. Finally, the system automatically creates remixes of original video by randomly combining shots and solos based on tags to produce creative arts video stories. A slightly different version of MoViE appeared in [13], where the basic idea of the tool is the same, but in this version users can create remixes either manually (i.e., by selecting videos one by one and by adjusting their order and their start and end cues) or semiautomatically (i.e., by defining a list of tags that the system uses to search for corresponding clips and to create the remix).
Scheible et al. [12, 14] propose an urban storytelling game which combines a mobile client, a storytelling tool in the Web, and a large public display into a collaborative street art authoring system deploying ubiquitous multimedia. The aim of the game is the illustration of stories created by a web player (also by remixing sentences from a pool of already illustrated stories) in collaboration with mobile players. Selected best stories are displayed on a large public display. The design of the game triggers creativity in writing stories and taking photos and fosters collaboration and social interaction in the form of team play.
As video sharing becomes more and more widespread, the same happens with video remix designated for supporting users during the process of video editing. The market offers a wide range of professional video editing softwares like Adobe Premiere (http://www.adobe.com/products/premiere/), Apple’s iMovie (http://www.apple.com/ilife/imovie/), and Windows Live Movie Maker (http://explore.live.com/windows-live-movie-maker?os=other). During the last years also a lot of online tools proliferated. Examples are JayCut (http://jaycut.com/), Eyespot and Jumpcut (the latter two online video tools were quite popular in the past few years, but has been recently shut down), which allow users to upload home videos and edit them on the Web, providing an alternative to simple desktop video editors. Others, like Cuts (http://www.cuts.com/) and Sweeney Todd Trailer Editor powered by GorillaSpot (http://research.yahoo.com/Yahoo_Research_Berkeley), have been released to create video mashup and provide users with the ability of selecting pre-existing contents to create personalized audiovisual material.
Diakopoulos et al. [7] present a qualitative case study of JumpCut, illustrating how collaborative authorship in remix culture is being affected by the composition of environmental constraints, which include legal codes, community and social norms, physical and architectural design, and economic factors. Authors suggest also some potential design implications based on their analysis. In particular, they say that tools to support creativity could be leveraged in the interface to enhance a remixer’s ability to find interesting juxtapositions of clips by, for instance, providing a palette of suggested clips based on loosely related tags. Furthermore, reducing the time and efforts of searching for and importing contents would enhance the ability to rapidly test and evaluate creative remix ideas.
Same findings are reported by Shaw and Schmitz [15] from the analysis of the user behavior during a pilot deployment, in association with the San Francisco International Film Festival, of a web-based platform which allows users to select, annotate, and remix material from a shared media archive.
Cesar et al. [16] describe an architecture of an inherently more social approach to viewing and sharing media. Authors promote the introduction of advanced user features (e.g., facilities to fragment a video in one or more ranges of clips, or to add annotations to video and its fragments, or to enrich video adding subtitles, captions, remixing, repurposing, or voice to a baseline object) as a spontaneous activity in order to enhance social sharing of video.
A recent tool of video mashup which reflects many of the suggestions given in works cited above has been presented by Cardillo et al. [4]. The tool allows users to navigate and interrogate a video repository structured following an ontology which mirrors the personal cinematic world of the audience and returns as result of the user query an automatic editing of the requested clips exploiting metadata (high- and low-level features) and tags. This process provides the user with a collection of clips having semantic coherence and stylistic homogeneity. Once the requested clips have been found, users can modify the remixed video, for example, changing the order of the clips, their start and end points, or the audio properties of clips.
Finally, in the area of remix, it is worth noting a project initiative between Yahoo! and Research Berkeley (http://research.yahoo.com/Yahoo_Research_Berkeley). The partnership has a declared scope of exploring and inventing social media and mobile media technology and applications that will enable people to create, describe, find, share, and remix media on the web.
MovieRemix differs from the above proposals in several different ways. First of all, although it may recall a video editing tool, it is not, neither is a tool to promote creative thinking like Diver [11] nor a tool to support collaborative story narration as MoViE [10, 13] and Story Mashup [12, 14]. Conversely, MovieRemix has been designed with two main goals. First, it aims at creating an exciting environment where the production of creative videos has to be easy, fun, and pleasant; to this aim, MovieRemix exploits experiences of the several proposals in the field like [7] and [15]. Second, by giving players a real screenplay, it aims at improving competences and abilities of people who would like to act like a movie director. This means that MovieRemix allows players to produce remix of videos from scratch and therefore is different from proposals (like the one in [4]) that mainly focuses on automatic production of video remix.
3. Preliminaries
MovieRemix is designed to produce a creative video product called remix. This product is based on a given screenplay (original, outline, or derived); once received, a player has to find and organize video shots so as to meet the screenplay guidelines. In the following, we briefly describe what is a remix, a screenplay, and a derived screenplay. In addition, since this paper also proposes an architecture to support the game, in the following we also briefly review basics of the standards used in the architecture.
3.1. Remix
The advent of new digital technologies has opened up whole new world of replica (or remix) practices. For instance, in popular music we have cover versions and remixes of golden oldies as well as new songs. A similar trend is true for music videos and movies. In essence, the transformations of the myth of the “original” or source text seem to have no limits. According to Lessing [17] and Manovich [18] remix has nowadays a broad meaning of transformation, reediting, bricolage, junctioning, or overlapping the original with other pictures, sounds, videos, or music [19]. Remixing aims to create something new with practices of bricolage and recycle. Manovich [18] states that we are living in a remix culture society: music, fashion, design, art, user-generated contents, media sharing, and even food are mostly remix, mashup, and collage.
Roughly, we can define a remix as the art of reusing catalogs of music, images, audiovisual, and multimedial cultural products [5]. It is worth noting a difference between video remix and video mashup: the former is a whole rework of another video given as a single source, whereas a mashup is a rework of just some fragments of many existing videos.
It is inevitable that the practices of interpretation will revise known texts and generate new texts. This is easy to qualify as the multitude of films containing variations, or sequels, remixes, cover versions as in Von Trier’s movie The Five Obstructions. Furthermore, some texts or parts of a text become a sort of “matrix” generating other texts, other versions and practices, new interconnections, and so on. For example, Run Lola Run (Lola rennt) by Tom Tykwer is a film about variations that invite us to compare the film proper with its teaser preview and video clip preview by putting both clips in the same DVD [20]. Texts can generate a variety of practices from “bastard pop” to “mashup” in music and from reworking to remaking in film and video. This is exactly what happens in Be Kind Rewind by Gondry (2008) which is a film about “how to swede” a movie, that is, how to make a remake [21]. The film gives details of the practice and so does the film’s website, which shows lots of fans “sweeded” short films. Talking about repetition and internal variations, Run Lola Run has an incipit that becomes a matrix of invariants to provide three versions of the same narrative. Each version replicates the same forms of content and expression as the first one, changing only the representation and some moments of the action. Though different, each story is bound to the others: it employs the same logic of a videogame, and hence the three versions are a sort of implicit sequel, in which the hero is gaining new strength and new skills. Like the spectator, the hero is also learning in the repetition, so by the second and the third games Lola knows and takes advantage of what is about to happen. Also Gondry’s Be Kind Rewind uses remake and remix practices as a subplot. A cult movie scene deals with a video tape from a rental store that does not work because it has been canceled. The whole story is exploited to lament the end of an era that started in the 1990s with the first DVDs. They talk about zero-budget short remakes. The movie exemplifies the homemade movie as a form of art, the reuse of some key scenes as a way to recreate the source film and the practice of the remix. Although neither Be Kind Rewind nor The Five Obstructions shows a whole homemade remix, Gondry does make explicit references to the various original movies using markers like the source movie credits. It is thus still a postmodern aesthetic of fragments and variations. The film becomes a myth in the fan’s affective memory, and the original movie is considered as a series of instructions of setting, costumes, characters, music, shots, and so forth. These instructions will be mixed with a clever bricolage to create new short movies. Be Kind Rewind has also a rich website, with video of instructions related to how to recreate “your own film.” Anybody can shoot a digital short film and post it on this site. As a result, more and more people are making home-made remix videos.
3.1.1. Screenplay
A movie is scheduled through a screenplay and a shooting script. A screenplay may be described as a writing technique to plan and preview a movie. Its composition is unique because the screenplay is a text that must have expressive, dramatic, and aesthetic qualities as well as practical and functional utilities. A screenplay contains the dialogs the actors have to play but may also contain psychological and aesthetic aspects of the story that are necessary when playing particular scene as well as when preparing the set or the cast costumes. Shooting choices and other technical instructions could be given aside in a shooting script.
3.1.2. Screen-Derived Screenplay
After every possible variations given by actors and set problems or by other improvisations or choices that occur during the shootings there is the phase of editing and postproduction. When a spectator watches a movie, he/she does not know how different this is from the first outline and even from the final screenplay. That is the reason why scholars who analyze movie propose to write down an inferred or derived screenplay, which is a screenplay described directly from the screen and realized only after the movie release. Therefore, the original screenplay is written in the planning phase of the film, while the derived screenplay is an analytical rewrite after the film has found its final form, has reached its audience, and, perhaps, has become a classic. A derived screenplay is usually a transcription with two columns: in the left column are provided all descriptions concerning sound, dialog, voice-over, music, and so forth, whereas the right column usually presents the number of the shots, a brief description of images and actions, technical data such as the type of shot and the cinematic effects (like fade-in or fade-out), and every camera movement (like pan shots, dolly, etc.).
3.1.3. The MPEG7-MDS
From the technological point of view, to manage a media content, it is useful to use a representation language capable of describing with metatags the semantics of the contents. For instance, apart from video data, it is necessary to have additional information like title, author, initial and ending points of a video segment, and so forth. MPEG-7 Multimedia Description Schemes [22] have been designed to this aim. It is the core part of the MPEG7 standard and was proposed to describe multimedia contents with a set of textual tags. It is a markup description language based on XML Schema that allows producing a description of the spatial layout of different media objects (e.g., audio, video, text, graphics) as well as the temporal order in which these objects will be played out. The description is based on tags, which define the purpose of the media object description. A tag usually has attributes and values that define the media object aspect (e.g., position and color) and has the form <tag attribute=value> (with the exception of tags that do not have attributes).
Details of MPEG7-MDS are outside the scope of this paper, and we refer the readers to [22] for such details. However, to appreciate the power of MPEG7-MDS, we report in Figure 1 an example of MPEG7-MDS description, where a video segment is described with some basic information like title and time length and with some textual description like screenplay and tags.
3.1.4. The MPEG-4 Standard
The encoding mechanism is another important technological aspect, very important when dealing with video streaming. As earlier described, our proposal uses Internet to stream the video to the client. Therefore, we need an efficient coding mechanism that provides acceptable quality while offering low bitrates. With such constraints, MPEG-4 [23] is probably the most used coding algorithm. It is a standard defined by the Moving Picture Experts Group to handle audio/video material. It is composed of several parts that deal with different aspects of the audio/video encoding. In our proposal we consider Part 2 (encoding of video material) and Part 3 (encoding of audio material). The former has different profiles in order to accommodate needs of different applications (from low to high quality). For the purpose of this paper, we focus on Simple Profile, a profile designed to provide videos in devices with limited system resources (e.g., cellphones and iPods). Part 3 (also known as Advanced Audio Coding) specifies audio encoding algorithms and provides higher quality with respect to previous released versions. Details of these encoding algorithms go beyond the scope of this paper (interested readers can refer to [23]).
4. Our Proposal
In this section we present details of the MovieRemix game and of the architecture we propose to support it. The main motivation behind MovieRemix is to create an environment able to stimulate the production of creative contents starting from pre-existing video material like professional or amateur video shots. MovieRemix creates a social space where general audience can produce creative videos. Needless to say, in addition to the game, it is necessary to design an architecture capable of supporting MovieRemix.
In essence, the goals of our proposal are to (i) create a game environment where the making of a video is easy, fun, and pleasant, (ii) stimulate creativity by providing different levels of game complexity, (iii) avoid the need to have special or particular devices (a device with an Internet access is sufficient), and (iv) design an architecture that keeps all the technological issues at the server side (and hide them from users).
Before presenting details of our proposal, let us depict a possible scenario.
Paul is fond of technology, and one of his favorite hobbies is to create new videos starting from pre-existing video shots. He gets videos from the Internet and uses a video editing application to extract video shots from long videos, to combine them in a particular order, and to add sound effects and a soundtrack. When Paul is satisfied with his creation, he uploads his video to popular video sharing websites in order to make it visible to other people. Alice is a journalist, and she uses a computer everyday, both for work and for fun. She loves watching remixed video from YouTube, and she would like to create her own videos, but unfortunately, she does not know how to use a video editing application. She tried, but never succeeded. One day she finds a game called MovieRemix. The challenge is exciting: make a movie following a given screenplay, using an easy-to-use video editing application. The game provides her with a library of video shots, each one described with high-level features (like title and actors), low-level features (like number of colors and time length), screenplay details, and users’ tags. She decides to play, and she gets an outline screenplay of what to do. MovieRemix gives her just a theme: “Bicycle in the traffic.” She starts searching for video shots using several different keywords: “bicycle,” “traffic,” “New York City,” and “critical mass.” After a while, she finds what she needs, combines them with transitions, music, title, and credits, and she uploads her final work to the MovieRemix video gallery. The day after, her remixed video results the most viewed video. One day she decides to play the “challenge” mode. The game server selects two players and gives them the same screenplay. Alice plays against Paul with the goal of making a video based on the theme “Plastic bottle.” After a while, they both upload their creative works and wait for other players’ responses.
Although simple, the above scenario is common in the current Web 2.0 scenario, where users want to play an active role. The contribution of this paper is to remove most of the technological burdens that may cut off a portion of our society. By coupling the architecture with a novel game that stimulates the production of creative videos, our proposal aims at creating an exciting environment where people can have fun playing with videos.
4.1. The Game
To stimulate users in the production of creative videos, we set up a game with three different complexity levels. As mentioned the game’s goal is to create a remix from preexisting video shots. The production guidelines are given to each user in the form of screenplay (original, derived, or outline). The three levels are the following.(i)Director. The player is provided with a derived screenplay. The player acts like a movie director and is required to follow the guidelines of the derived screenplay. Therefore, he/she has to select carefully different video shots stored in the MovieRemix catalog depending on the characteristics specified in the derived screenplay. Figure 2 shows an example of a derived screenplay.(ii)Apprentice Director. The player is provided with an original screenplay. The player can introduce a subjective interpretation of the screenplay, and therefore he/she is more free to express his/her creativity. Figure 3 shows an example of an original screenplay.(iii)Bricoleur. The player is provided with an outline screenplay (e.g., a simple theme or a movie title); therefore, he/she is free to create the remix as he/she does not have detailed guidelines to follow. Figure 4 shows an example of an outline screenplay.
After selecting the game level, a player may select the challenge option to play “against’’ another player. If so, the same subject is given to two different players (randomly selected by the game or specified by one player). During the remix process, MovieRemix allows applying transitions effects between video shots, soundtrack, title, and credits. When satisfied with the remix, a player can upload it to the MovieRemix gallery. If playing in the challenge mode, the two videos are compared one against the other and the winner will be the most voted by the MovieRemix community.
4.2. The Architecture
Figure 6 shows the architecture we designed to support MovieRemix. It is composed of a Game Server, a Game Client, a Streaming Server, and a Movie and Screenplay database where video shots and movie descriptions are stored.
The architecture is designed to keep all the complexity processes (e.g., video encoding, video retrieval, video description) at the server side. In this way, no technological skills are required to players, and members of the MovieRemix community are free to express their creativity.
Before entering into details of the architecture, it is worth recalling here that our proposal uses standard techniques to manage video editing and to produce portable Remix videos. Portability is an important characteristic as it ensures that the produced file can be used over several different devices. To achieve portability our proposal uses MPEG7-MDS for describing and organizing audiovisual data. Similarly, to encode video data, our proposal uses MPEG-4, a standard largely used in the Internet scenario to manage audio and video data.
In the following, we present details of the MovieRemix architecture.
4.2.1. Movie and Screenplay Database
Video material and screenplays are stored in a database accessible both to the Game Server and to the Streaming Server. Each video (e.g., video shots, short videos, and remix video) is coupled with textual information to better define the video contents. In particular, each video is provided with an MPEG7-MDS description of its high- and low-level features:(i)High-level features. Title, director, cast, and screenplay are entered by who releases the video and makes it available to the Movie and Screenplay database.(ii)Low-level features. Number of colors, resolution, duration, and number of frames per second are automatically extracted from each video shot when it is released to the Movie and Screenplay database.(iii)Tags. Keywords that describe video content may be entered by both who releases the video to the database and by players who watch the video. Note that tags may be very useful in describing multimedia contents as high- and low-level features may not completely describe the video content and a taxonomy-based approach could be too rigid for video browsing.(iv)Thumbnails. A set of keyframes that represent a sort of static summary. In this way, when browsing the video catalog, a player can understand the video content by simply hovering the mouse on the video shot, instead of watching it entirely.
Figure 7 depicts the information associated with every video shot stored inside the MovieRemix catalog. Note that textual information is described through MPEG7-MDS as shown in Figure 1.
4.2.2. Streaming Server
A streaming server, or a farm of streaming servers, is in charge of retrieving and streaming a requested video to the client. We recall here that the proposed architecture does not allow clients to download video. From the client point of view, when a video (or a video shot) is played in streaming, it is indistinguishable from an ordinary media file stored in the local computer device. By using this approach, all the video operations are done at the server side, whereas everything stored and managed at the client side is text based. This choice is twofold: it avoids unauthorized storage (and possible unauthorized redistribution) of video material, and it does not require clients’ devices to have huge storage space to keep video material at the client side. As previously mentioned, all the video material MPEG-4 encoded.
4.2.3. Game Client
The game client interacts with game and streaming servers and creates an environment where the production of remixed video is easy and fun to play. In addition, the game client has to provide the following facilities.(i)Authenticate. At startup every player needs to login to the game server; after that the MovieRemix game can begin with the downloading of a screenplay (derived, original, or outline).(ii)Search and Browse. Every player should be able to look for specific video shots by entering textual keywords. The game server should reply with a list of video shots that the player should browse in an easy way.(iii)Retrieve. Once an interesting video shot is identified, it should be easily added to the personal bin of video shots for a possible usage in the remixed video. Note that video shots should be virtually added to the personal bin, as they physically remain at the database side.(iv)Edit. Different video operations should be available: extraction of a part of a long video by specifying an initial and an ending point; availability of tools to facilitate the creation of titles, credits, and subtitles; availability of several possible transitions to be used when a video shot ends and another one begins; availability of a tool to add soundtrack.(v)Describe. The player should be able to define and add his/her own tags for every video when playing it.(vi)Organize. The player should be provided with a storyboard, simple and easy to organize.(vii)Rate. The player should be able to rate a video remix and to view a video remix chart.(viii)Upload. The player should have the capability of uploading his/her remix to the MovieRemix gallery in order to make it visible to other MovieRemix players (Figure 5).
Note that all complex and time-consuming operations (e.g., encoding and visual effects) are done at the server side. The client simply needs to describe the operations that the server has to perform. For instance, let us suppose that a player applies a dissolve transition effect of 10 frames between clip 1 and clip 2. The client will simply write into the remix file an instruction like Dissolve (clip1, clip2, 10), and the streaming server will do all the necessary work.
4.2.4. Game Server
The game server is the core of the MovieRemix architecture: it interacts with the game client and gives instructions to the streaming server of how and where to find specific videos. In particular, the game server is in charge of the following tasks.(i)Authenticate. When a player performs the login process, the game server has to check whether the player is registered to play the game. If ok, the game server sends the client the screenplay (derived, original, or outline).(ii)Summarize. Every video stored in the MovieRemix catalog needs to be summarized. This allows players to save time when browsing the video catalog. In essence, by watching a video summary (either static or dynamic) players may avoid wasting time watching entire (and eventually) useless videos.(iii)Extract. Every video stored in the MovieRemix catalog needs to be described through low-level features. This allows a better description of the video material and facilitates the retrieval process.(iv)Group. When players ask for video shots, the game server may present “similar” video. This can be done by grouping together similar videos. If grouped according to low-level features, clustering algorithms are usually employed. If grouped according to players’ tag, algorithms based on folksonomy are usually employed.(vi)Retrieve. When players look for videos, they enter textual keywords: the goal of the game server is to retrieve the most relevant videos according to the specified keywords. Keywords can be screenplay words, high- or low-level features, or tags.(vii)Communicate. The game server interacts with the streaming server to facilitate the stream of the requested video towards the game client.(viii)Update. Players can introduce additional tags to every video shot. Therefore, the game server needs to update the information associated with every video shot.
5. Prototype Implementation
In this section we present a prototype implementation of MovieRemix.
5.1. Streaming Server
The goal of a streaming server is to stream a video toward the game client. Several applications (either proprietary or public) are available to this aim. In our prototype we use AviSynth (http://www.avisynth.org/) for three main reasons: (i) it is released under GNU GPL license, (ii) it has a powerful and scalable scripting language that can be used to provide transition and video effects, and (iii) it is simple to use as it takes a text file (written with AviSynth scripting language) in input and creates a video file that can be read through any media player.
5.2. Game Client
The Game Client is developed in JavaFX (http://javafx.com/), a platform for creating cross-platform media applications that can easily provide media playback in the desktop window, within a web page, or on the mobile device. Based on an object-oriented scripting language, JavaFX is extremely practical with the development of graphical applications.
MovieRemix game client allows users to select, describe, and remix material from the MovieRemix Video Catalog.
Figure 8 shows the interface. The top menu allows players to select a specific medium of the remixed video (Audio, Video, Transitions) and to operate on it (e.g., the submenu Video contains the extract feature that allows extracting a video shot from a long video), to browse the MovieRemix Galleries (one for each game level), and to explore the MovieRemix charts (one for each game level).
On the left side, the Game goal box contains the screenplay (original, derived, or outline), whereas the preview box provides all the standard video player controls a player might need.
The right side presents two boxes: Find video shots and Shot info. The former allows players to search video shots by entering textual keywords. When a player hovers the mouse pointer over a remix, a set of thumbnails appear one after the other. By selecting a shot among the ones returned by the game server, players can access detailed information about the video shot. This additional information is presented in the Shot info box and includes high- and low-level features and tags.
The storyboard is located at the bottom side: here players can organize video shots and can add video shot transitions (choosing them from the submenu Transition), can introduce title and credits (through an option of the submenu Video), and can add soundtrack to the remixed video (through the submenu Audio). Note that to preclude any possible conflict between soundtrack and shot audio, the audio of each video shot can be silenced.
Figure 9 shows a MovieRemix chart (in particular, the one of the Bricoleur level). Members of MovieRemix can play and vote the preferred video remix. Once again, to save players’ time, when a player hovers the mouse pointer over a remix, a set of thumbnails representing the video summary appear one after the other.
5.3. Game Server
As previously mentioned, the game server is the core of the MovieRemix architecture and is in charge of several tasks. In the current version of the prototype, the game server uses a simple algorithm to extract thumbnails from a video shot. These thumbnails are taken at video cuts (i.e., when two consecutive video frames are very different) identified using a combination of luminance and chrominance values. Also low-level features are taken using a simple algorithm that retrieves such information from the header file. With respect to the grouping algorithm, in the current version of the prototype, grouping is available only according to players’ tag, that is, an algorithm based on folksonomy is employed; the grouping according to low-level features is not currently implemented.
6. Experimental Evaluation
To assess our proposal we set up an experimental scenario where a group of people were asked to play with MovieRemix. We used a mean opinion score (MOS) technique, which is widely employed in testing products or services. Roughly, it works as follows: evaluators need to rate several questions with a scale expressed from 1 to 5. This type of test can be considered effective as long as the obtained results do not present a large statistical difference. Our experimental scenario consisted of 16 people with different backgrounds (computer science and social sciences) and different work experiences (academic and private employees).
We asked them to answer several questions with respect to the MovieRemix experience. It is to note that the obtained MOS results did not present a large statistical difference.
Figure 10 presents the different game levels selected by players. Most of the players chose the Bricoleur mode, whereas few of them played in the Director mode. To better understand the reason for this behavior, we asked them the motivations that led to such choice. Figure 11 highlights that two were the main reasons: one is the difficulties of playing with the other levels, whereas the other is the easiness of translating the proposed screenplay into video. Figure 12 shows that personal creativity is another important reason for the game level selection. As showed, most of the players think that the outline screenplay (the one of the Bricoleur level) is the most suitable to free player’s creativity.
Figure 13 presents results obtained from asking players when they are satisfied with their work, meaning when they consider their remixed video done. Most of them like to personalize the given screenplay. Although this may be surprising, it is not. Indeed, many movie directors like to change the given screenplay according to their inspiration.
At the end of the game, we asked players what the main difficulties they encountered were. Figure 14 shows that players had no problem in understanding the given screenplay and also highlights that there are two main burdens in making a remix video: one is related to the technological aspect (either editing the different media streams or browsing the video catalog), whereas the other is more semantic as it is related to the translation of the given screenplay into video.
Figure 15 presents results obtained from asking players the genre of video that is more suitable to produce with MovieRemix. Players are happy with all types of videos, with a slight preference for music and news videos.
Figure 16 presents results obtained from asking players how they would improve the video retrieval. Sample videos (i.e., related videos suggested by the systems) seem to be very important for players when looking for video clips to use in the video remix. Tagclouds and moving videosummaries are considered equally important.
Figure 17 presents results obtained from asking players what type of game is MovieRemix. Players considered MovieRemix both an educational and a serious game, which confirms the goal of our proposal.
7. Conclusions and Future Directions
MovieRemix is both a novel educational game and an architecture that supports the production of video remix. It allows players to deal with real screenplays and to use them as a movie director. Through a developed prototype we evaluated the proposed approach and results showed that players like to use MovieRemix to produce different types of video (e.g., movies, documentaries, news, music, commercials, and recaps). An interesting result was that players had almost no difficulties in understanding screenplays. On the contrary, the main burdens were related to the technological (e.g., browsing the video catalog) and to the semantic (e.g., translating the given screenplay into videos) aspects. Players also pointed out possible improvement of MovieRemix by suggesting a more deep usage of sample videos (e.g., videos related to the given screenplay and suggested by the system), of tagclouds, and of moving videosummaries.
Acknowledgments
The authors wish to thank all the people who helped doing the subjective evaluation. The work of M. Federico has been partially supported by Disabled Students Services under the project. Design and Analysis of Mechanisms to Improve Multimedia Contents Accessibility within the Learning Scenario.