Abstract
The application of virtual reality (VR) technology has revolutionized the aesthetic concept of traditional movies, which especially causes the evolution of the concept and form of time and space in movies that the space-time structure and narrative form of traditional movies are no longer suitable for VR movies. Therefore, in this paper, the space-time of VR movies is deconstructed and reconstructed, and the space-time consciousness is taken as the research background. From the perspective of creative subject and audience experience, the space-time narrative characteristics, structure, and methods of virtual reality movies are discussed. At the same time, based on the dynamic scheduling principle of VR images, a multisource scheduling model is established with narrative space, intention space, aesthetic empathy, emotional identity, time deconstruction, and music expression as the original data sets, which is of guiding significance to the creative practice of VR movies.
1. Introduction
VR film is the application of VR technology in the field of creation in film and television. As a brand-new art form, it subverts the audio-visual experience that traditional films bring to the audience and provides more possibilities for the audience in terms of immersion and freedom. Different from traditional movies and TV shows, it provides viewers with a sense of “happening” time and has the “interactivity” [1–3] that traditional film and television works lack. Physical features such as color, sound, and environmental features in VR space can resort to sensory experience. Moreover, through technical means, the audience’s sensory system can be enhanced and extended, so that it can interact with the virtual space and achieve an immersive mental state when people watching movies.
However, with the deepening of creative practice, among the video works mainly produced by shooting, VR images are shot by panorama camera with almost 360-degree viewing angle, such as industrial-grade panorama camera Insta360TITAN or consumer-grade panorama camera Insta360ONEX [4, 5]. However, traditional cameras have borders and boundaries, and the display of final picture is influenced by lens focal length, picture scene, and other factors. The traditional way of watching movies requires the audience to be quietly in a dark environment, while the viewing method of VR images requires the audience to wear helmets and keep moving. These differences have led to the fact that some of the experience and theoretical achievements accumulated by traditional images over the years cannot be perfectly applied to VR image creation [6, 7]. Due to the long-term lack of theoretical guidance, VR movies have some problems, such as insufficient narrative ability and imperfect creative techniques, which leads most of the current VR works are blindly creating spectacle images and pursuing visual experience [8]. At the same time, the content of narrative theme is gradually weird and absurd, where language confusion in creation, vague definition of category of works, serious homogenization, and other problems often exist [9–12].
The previous theoretical framework of creation does not adapt to the new environment, forcing VR movies to combine their own characteristics to find a new way to construct a narrative theoretical system based on spatial narrative theory.
Common load balancing scheduling algorithms mainly include FCFS (First Come First Served), SJF (Shortest Job First), RM (Rate Monotonic), LSF (Least Slack First), and so on. These scheduling algorithms can be judged by the remaining idle time of a task. The less idle time, the task is scheduled to be executed by the CPU as soon as possible. However, due to the change of priority, the system schedules frequently switch, resulting in severe system turbulence.
Among them, scene scheduling, as an important spatial narrative technique, is the key to the new system of VR image narrative theory. Based on scene scheduling, a spatial narrative theory system for VR movies is built, which is not only feasible but also has certain academic and practical value.
2. Temporal and Spatial Structure of VR Movies
Compared with the traditional cinema pattern, the mode of emerging movie pays more attention to the visual impact and the immersive entertainment experience of consumers, but most of them are fragmented, incoherent, and gamified narrative structures. It is difficult for a single frame to attract the audience’s attention for a long time, which requires the director to control the development of plot more comprehensively and add more plots and elements to the story. Invisibly, the audience’s attention becomes distracted, the continuity of the plot is inevitably affected, and the narrative becomes fragmented.
2.1. Based on the Creative Subject
The expression of movie languages must be closely related to the theme, and the basic elements such as character setting, plot development, and image style will serve for the promotion of the main plot line. When the architecture of new space and time is adopted to separate multiple space and time independently, how to make it clear to the public without causing confusion is particularly critical, which inevitably requires the creator to set the main emotional clues throughout the creation. No matter whether the time or space changes are the connection points, there will be causality and formulaic plot setting of rewriting rule [13, 14]. Based on the spatiotemporal reconstruction of creative subjects, the rewriting rules are shown in Figure 1.
The rewriting rules of “ Space-time reconstruction” are divided into two categories. The first category is that stories can be rewritten through time and space, and the second category is that stories cannot be rewritten. In the rewritable category, it is divided into complete rewriting and partial rewriting, and partial rewriting can lead to changes in the process and results. Under the unrecoverable results, a sightseeing summary means that the protagonist cannot make any story changes in the reconstructed time and space.
2.2. Based on Audience Experience
Within the VR system, the user’s interactive experience is one of the most important parts. Therefore, free and creative viewing is not only the lifeblood of movie creation but also the focus of movie research [15]. Time in real life is linear and continuous, while time in movies is fragmented and continuous in reasonable imagination. The abstract time is attached to the spatial structure, and the changes of time and space recorded in the film can make people the same. In the narrative structure of film and television, the movie with the theme of “time and space reconstruction” breaks the conventional narrative and reconstructs the time and space in the movie, which is different from the traditional time and space of movie. According to the screen felt by the audience, time can be divided into three forms: present, past, and future, as shown in Figure 2.
The way of image in space weakens the linear feature of narration and strengthens the sense of existence in space. People exist inside the screen and become the “core” of the content. In the space wrapped by digital images, the visual wandering is extremely free, which also leads to the main contradiction between space and visual characteristics [16, 17], that is mismatch between unlimited space and visual characteristics with only one focus. The movement of visual focus can no longer keep up with the speed of narration of traditional movies which results in spatial redundancy.
3. Rules of Interactive Spatial Scheduling
The characteristics of scene scheduling in VR video can be summarized as audience-centered scene scheduling with complete scheduling space and interactive features. As the “original material” of VR image scene scheduling, movie space has integrity in both static frame space and dynamic space, which also weakens the temporal narrative of VR images. At the same time, with the increase of space display, there is a risk of confusing spatial information.
As shown in Figure 3, when analyzing spatial scheduling of VR movie, it can be divided into static and dynamic aspects. On the one hand, static scheduling is in static frames; the category of static scheduling includes all static symbol elements and static space. On the other hand, dynamic scheduling is in dynamic space, and the category of dynamic scheduling is the relationship between all movements and movements of symbol elements in space.
3.1. Static Scheduling
Static scheduling is mainly divided into ontology symbol scheduling, intersymbol correlation scheduling, and synchronic scheduling [18].
3.1.1. Ontology Scheduling
It is mainly reflected in the choice of symbols and the spatial position of symbols. On the one hand, the selection of visual symbols is the beginning of VR video scene scheduling and the confirmation of scheduling materials. The establishment of characters and the arrangement of props are important links in the choice of image narrative symbols.
3.1.2. Intersymbol Correlation Scheduling
VR image art is a dynamic art but exists in the static picture space, apart from the meaning of each symbol; the relationship between symbols also conveys information for the audience. More importantly, in VR movies, the complete spatial display makes the number of symbols in the space increase greatly, and the relationships among symbols become more numerous and complicated.
3.1.3. Synchronic Scheduling
In the frame space, the symbols brought by time and the diachronic scheduling of space have been put on hold for the time being, while the display of symbols in space presents synchronic characteristics, which means that the frame space is not the linear display of a single image, but the simultaneous concrete display, the juxtaposition of many images, and the juxtaposition of various spatial relationships. Therefore, in the static scheduling, the scheduling of space and symbols is the scheduling of multisymbol synchronicity and the juxtaposition of their multispatial relationships, which is simply the scheduling of spatial levels in VR static space as a whole.
3.2. Dynamic Scheduling
Dynamic scheduling is also divided into ontology symbol scheduling and intersymbol correlation scheduling, but compared with static scheduling, it adds diachronic scheduling, which no longer separates time from space, but discusses the spatial scheduling of VR images in the continuation of time. So it is necessary to take dynamic changes as the main scheduling premise.
Narration has little influence on static scheduling, and the focus of scheduling is to show the meaning of symbols in a reasonable and orderly way. In dynamic scheduling, action is the noumenon of movie narrative, and the dynamic scheduling of images is greatly influenced by narrative, which makes the focus of dynamic scheduling no longer the presentation of the symbols themselves, but the fitting of narrative. All designs of dynamic scheduling should focus on narration that gives new meanings to symbols in space through scheduling to generate new connections.
The dynamic meaning scheduling among symbols mainly includes the scheduling of ontological meaning relations between symbols and the scheduling of extended meaning relations. In addition, the diachronic nature of dynamic meaningful scheduling is to increase the change of time dimension based on synchronic scheduling. Therefore, dynamic meaning scheduling and static meaning scheduling are not two parallel scheduling methods, but an evolution that influenced by narrative, VR dynamic meaning scheduling has a strong story. In narrative, there are two different relations between symbolic meaning and narrative expression. One is narrative expression where the scheduling of VR image meaning is mostly the evolution of dynamic meaning in symbolic ontology, which is the so-called display of symbolic “arc light” [19]. The other is that symbolic meaning is not the meaning to be expressed in narrative where narration will extend the meanings of symbols, and the extension is the object of dynamic meaning scheduling.
4. Scheduling Model of Multisource Data
4.1. Data Processing
The existing data of VR movies are collected from different ways, resulting in a wide range of data sources and inconsistent formats. Therefore, it is necessary to standardize and parameterize the multisource data that does not meet the loading format requirements, as shown in Figure 4.
Geometrical objects of GeoJSON include points, lines, faces, and composite geometric figures composed of the above types to represent vector elements. PAL files of basic image data including narrative space, intention space, aesthetic empathy, emotional identity, time deconstruction, music expression, and symbol language are converted into FPGA format by the data processing tool Cesium Lab. And then, the obtained vector data are analyzed according to the hierarchy of TopoJSON. Afterwards, the loading method of TopoJSON is adopted, and the creation object is added in its dynamic object collection, so that the converted vector data is stored in the data source collection of Cesium through the function Cesium.Topojs.load. Finally, according to the geometric types of vector elements, the corresponding geometric objects are created and added to the collection property of the 3D scene viewer.scence, so as to complete the conversion of vector data to TopoJSON format and 3D visual loading.
4.2. Establishment of Model
4.2.1. Octree Structure
Based on the octree pyramid structure, the data is divided into layers and blocks, and the preprocessed data is stored in the database in binary mode. According to the indexes, the image data in the resource management queue is extracted, and the 3D scene rendering of VR data is realized according to the request.
The principle of constructing the octree structure is similar to that of quadtree structure. The difference lies in that the basic unit based on octree processing is cube instead of rectangle; that is, the operation unit for merging and dividing is cube. As shown in Figure 5, the nodes of each layer from top to bottom are the pixel synthesis of 8 nodes of the next layer. Under the condition that the representation range is unchanged, the resolution is getting lower and lower, and the size is getting smaller and smaller.
Taking the original VR video data as the bottom layer of the octree pyramid structure, a range of completely surrounding the terrain size can be determined according to the size of the space-time sequence. By obtaining the longest edge the maximum value dem_max in the elevation data, the number of included elevation subblocks is calculated. Then, it is found that the pixel value of falls on a specific level, and a positive integer is taken, so that the size of the data packet can be obtained. On the premise that the resolution of static image of the next level is 1/8 of that of the previous level, it is repeatedly implemented by the octree processor of Cesium Lab and stops until the top-level data block cannot be divided by subblocks.
4.2.2. Scheduling with Multisource Data
In the process of real-time rendering of 3D scenes, it is necessary to schedule the data involved in rendering according to information such as viewpoint position while performing operations of specific rendering. The efficiency of data scheduling is very important in the process of large-scale 3D scene real-time roaming. On the one hand, the efficiency of data scheduling depends on the times of scene data scheduling that change with the change of viewpoint information. On the other hand, it depends on the time when the relevant mapping is performed. Double-thread data scheduling strategy includes two parallel scheduling modes, in which drawing thread is the main thread and scheduling thread is the secondary thread, as shown in Figures 6 and 7. (1)In the scheduling thread, the list of task first generates corresponding parameter requests according to the real-time information and updates the position coordinates of the current viewpoint to initialize the loading area. Then, according to the buffer read-write principle, it is judged whether the corresponding tile data needs to be scheduled to be loaded into the buffer. Finally, that correspond tile object is transferred to the tile queue before data load, according to the mechanism of tile classification management, where the tiles are hierarchically piped and controlled to complete the data scheduling task(2)In the mapping thread, firstly, the initialization area is updated through the change of viewpoint clipping, where the coding and interactive data of the scene tile nodes in the buffer are transferred to the application queue of the visible area. Then, according to the viewpoint distance, it is judged whether the tiles in the traversed visualization area need to be further divided, if not, the corresponding tile data is directly transferred from the buffer; otherwise, create another stack, push the tile into the stack, and divide it into octree, and the node pops up after division. For the newly generated eight nodes, repeat the division steps counterclockwise until all nodes become leaf nodes completely. Finally, according to the index information of the tile where the leaf node is located, analyze the corresponding information such as hierarchy, line number TileX, and column number TileY, and load it into the video memory for drawing. After all tile requests in the queue completing the above steps, empty the queue and take it as the request queue starting from the next frame
5. Test of Model
5.1. Experimental Parameters and Environment
The VR image data of this paper comes from the vision laboratory of a university. VR scenes were constructed based on the film and television work “Dear Angelica,” and the interactive space scenes include traffic, water system, and vegetation. Due to the inconsistency of some image data formats, it is necessary to integrate the multisource data that do not meet the loading format requirements in the way described above and carry out parametric integration. 10 HP Z600 workstations were interconnected through LAN network for testing. At 10th, 20th, 30th,...,50 frames, the CPU and memory utilization are recorded and compared.
5.2. Fluency
In order to verify the fluency of the data scheduling strategy and caching scheme in the rendering process of VR visualization scene with multisource data integration, relevant experiments were carried out in the roaming process of Cesium 3D visualization scene. The frame persistence per second (FPS) is an effective method to detect the rendering performance. The value of FPS represents the times that the image is rendered and refreshed per second, which reflects the smoothness of the scene. When FPS is less than 15, the scene will display stuck, and when FPS is more than 30, the human vision will be smooth. It is generally believed that the FPS value should not be less than 30 in the roaming process of 3D scenes [20]. By recording the FPS and frame number of scene rendering after implementing the dual-thread scheduling strategy and the second-level cache mechanism in the roaming process of 3D scene real-time rendering and comparing with the loading results of traditional methods, the statistical results of Web page frame rate are shown in Figure 8.
According to the changes of frame number and real-time frame rate in the process of roaming real-time rendering and rendering of spatiotemporal 3D visual scenes, the frame rate gradually increases with the request time and rendering preparation in the initial stage of the page. When the scene data is loaded and consumed, the FPS is basically maintained at around 45, which is not affected by the number of FPS. Compared with the drawing results of traditional methods, each frame is optimized to a certain extent, which achieves the condition of completely smooth roaming, that is, meets the balance requirements of data scheduling and real-time rendering.
5.3. Loading Efficiency
In the scene roaming process around 45, the changes of data load request, real-time data load, and scheduling time of Web scene were recorded with the slow and no long distance jump movement of viewpoints, as shown in Table 1.
It can be seen from Table 1 that with the start of the first frame, all data in the buffer is empty before real-time loading, and the amount of data loaded in real time is equal to the amount of data requested to be loaded. Data scheduling takes the longest time in the first frame of scene scheduling, because there is little change between the requested loading data and the real-time loading data during the slow movement of the viewpoint. At the same time, the amount of data loaded in real time determines the scheduling time. Compared with the average scheduling time of frames after the first frame, the average scheduling time is 0.161 s, while the persistence time of human vision is 0.05~0.2 s [21]. Therefore, the systematic scene in this paper can achieve smooth roaming conditions.
5.4. Scheduling Accuracy Test
There are 10 groups of 3D image samples used in the experiment, and the 3D image data of each group are shown in Table 2.
In actual processing, the following three situations will occur: First, there are many texture texels in texture space, in which the smallest single bit of 2D texture corresponds to a 3D model pixel. Secondly, each texel corresponds to a plurality of 3D model pixels. Finally, each texture texel corresponds to a 3D model pixel. For these three cases, targeted treatment must be carried out separately. The specific treatment methods are as follows: compression filtration treatment, amplification filtration treatment, and no additional treatment.
The video before and after scheduling is captured by 3D images, and its reconstruction accuracy is tested. The calculation formula is as follows. The comparison results are shown in Figure 9.
where represents 3D image reconstruction accuracy, represents the cross-section of three-dimensional image, stands for three-dimensional image vector, represents the collection of three-dimensional image feature points, and represents the original feature point of 3D image.
It can be seen from the figure that after the spatial scheduling, the reconstruction accuracy of the 3D images captured from the original video has improved remarkably; nearly 100%, which indicates that under this scheduling model, the system can realize the scheduling of 3D images with higher accuracy.
6. Conclusion
The enhancement of spatial display ability makes expression of spatial narrative the key of VR movie narrative, and then, scene scheduling, as an important means of spatial narrative, is given more narrative responsibilities and becomes the key of VR image narrative research. Therefore, based on the space-time structure of VR images and the scheduling principle of interactive space, the efficiency of data scheduling and scene rendering is improved, and the scheduling model of VR scene based on spatiotemporal analysis is developed. Theoretical analysis shows that VR movies have strong ability of space narrative due to the enhancement of display. The test results show that in this paper, the systematic scenario of the scheduling model can achieve smooth roaming conditions.
Data Availability
The dataset can be accessed upon request.
Conflicts of Interest
The author declares that there are no conflicts of interest.
Acknowledgments
This work was supported by the 2020 Humanities and Social Science Research Project Planning Project of Universities in Jiangxi Province, “Research on digital Visual Art Application of Jiangxi Red Cultural Heritage—Taking Jiangxi Red Culture as an example”; the project number is YS20123.