Abstract
Defining spatiotemporal relations and modeling motion events are emerging issues of current research. Motion events are the subclasses of spatiotemporal relations, where stable and unstable spatio-temporal topological relations and temporal order of occurrence of a primitive event play an important role. In this paper, we proposed a theory of spatio-temporal relations based on topological and orientation perspective. This theory characterized the spatiotemporal relations into different classes according to the application domain and topological stability. This proposes a common sense reasoning and modeling motion events in diverse application with the motion classes as primitives, which describe change in orientation and topological relations model. Orientation information is added to remove the locative symmetry of topological relations from motion events, and these events are defined as a systematic way. This will help to improve the understanding of spatial scenario in spatiotemporal applications.
1. Introduction
Automatic event detection is gaining more and more attention in computer vision and video researchers community. Visual scene description takes into account ontological viewpoint of relative object positions. It is sufficient to emphasize the model of moving objectβs spatial relations such as modeling video events [1, 2]. Modeling spatiotemporal relations between moving objects involves the modeling of motion events such as durative events. These events are the union of primitive events, which hold for each snapshot during the interval with a particular temporal order.
Spatiotemporal features could be used for modeling the spatiotemporal events [3]. Defining spatiotemporal relations have two main domains of research: spatiotemporal object and spatiotemporal relations modeling. Cuboid object approximation or three-dimensional geometry is used to model the former and for lateral two-dimensional objects occupy different spatial locations at different time points [4]. Several types of logics for mechanizing the spatiotemporal relations and reasoning process are used like interval temporal logic [5, 6], point temporal logic [7], and propositional model logic [8].
The point temporal logic supports instantaneous snapshots of the world. A snapshot represents the current situation, and a spatiotemporal relation is defined if a particular spatial relation holds for every snapshot during that interval. It is considered that time and space are bounded to each other. Spatiotemporal relations are modeled between moving objects by taking transaction from one snapshot to the next snapshot.
Dimiter Vakarelov, in [9, 10] provided strong mathematical and logical bases for defining the general spatiotemporal topological relations and divided them into two categories: (i) stable spatiotemporal topological relations and (ii) unstable spatiotemporal topological relations. A stable spatiotemporal topological relation is a relation which holds for every frame or snapshot in the interval and unstable spatiotemporal topological relations are those which hold at least for one snapshot during the temporal interval. Some spatiotemporal relations are strictly stable such as disjoint (D), nontangent proper part (NTPP), nontangent proper part inverse (NTPPI), equal (EQ), and others may be stable or unstable like meet (M), partially overlap (PO), tangent proper part (TPP), and tangent proper part inverse (TPPI). This provides a way to use the spatiotemporal relations in linguistics and their use as motion events modeling.
Most of the existing theories of spatiotemporal relations are domain-based, where domain knowledge imposes the conditions to a spatiotemporal relation to be topologically stable or unstable. A domain where the spatiotemporal relation is topologically stable, there the directional or distance relations are unstable such as spatiotemporal relations on the road networks. In defining motion events or verbs which represents the transitive movement, stability or unstability of topological relations plays important role and directional relations along with the topological relations remove certain symmetries and helps the user to understand real-scene situation. Consider the following two examples.(1)Mr. John (object, name of a person) crosses (relation) the football ground (object);(2)Mr. John (object, name of a person) crosses (relation) the football ground (object) from north to south.
In proposition 1, there is no confusion about the topological relation that object (John) has certain topological relation with object (football ground). But there is a symmetry about the directions, and user did not know the exact direction from object to object before and after the occurrence of cross event. But in proposition 2, when directional constraints are added, they remove the confusion about directions and symmetry of topological relations that object (john) crosses the object (ground) from north to south. It justifies that how the topological relations in two objects change, and what was the temporal order of occurrence the primitive events.
In our approach, we used a method of combined topological and directional (CTD) relations [11], more suitable for reasoning about the moving objects and developing the motion events. Situation is represented by relationship between the considered entities. It is natural to represent the information using relations. Events can be expressed by interpreting collective behavior of physical objects over a certain period of time. The main focus of this work is to formalize the spatiotemporal relations and the spatiotemporal events in a systematic way.
This paper is arranged as follows. Related work is discussed in Section 2 and Section 3 composed of preliminary definitions. Section 4 explains the combined topological and directional relations method, spatiotemporal relations are defined in Sections 5 and 6 compose on geometric representation of some motion events and these motion events are defined in Section 7. Section 8 concludes the paper.
2. Related Work
A moving object occupies different positions at different time instants. Relative motion means that the object changes its position with respect to another object. This relative motion can be studied through different aspects of space, and spatial relations are one of them. A set of spatial relations which hold for one snapshot is considered as a primitive event, then spatial relations between moving objects for an interval are characterized as spatiotemporal relations or spatiotemporal events.
Commonly adopted approaches are qualitative and domain-based such as qualitative trajectory calculus (QTC) [12, 13]. This describes relation between moving point objects. Hornsby and Egenhofer [14] modeled the different spatiotemporal relations between moving objects on road network. All these relations represent certain class of motion and objects are approximated as point objects. When objects are under motion, especially on road networks, the relations are purely directional relations, where the objects change their position, but do not change the topological structure of scene.
A mereotopological approach is extend to define spatiotemporal relations and a notation of temporal slice is used, where temporal slice is called an episode of history for a given interval [15, 16]. The primitive events can be defined using the Allenβs temporal logic and defining relation (P,i) (property P during the time interval i) [17]. In this method, interval temporal logic is used, and a primitive temporal interval is defined, the smallest interval where the relation does not change. For composite events another property βoccurs,β defined as occurs(e,i) = event , occurs during the interval , and different hold predicates are combined through logical connectors in a sequential order.
Ma and Mc Kevitt [18] described a method based on continuous transection from one state to another state. In this approach, topological relations are computed by the 9-intersection method [19]. This method supports instantaneous point temporal logic, which detects only changes in topological structure of scene at different instants of time. This method is based on point set topology and uses the snapshot model for spatiotemporal data [20β22]. This model of topological relations is used by Muller [23, 24] to model the motion events or motion classes. They model the different motion events which involve the topological changes at each analysis frame. Spatiotemporal relations between moving objects are also effected by the environment regarding its application domain such as modeling the relation cross, enter, leave shows that one object is only on concept level, that is, a region of interest. They are defined for a network, visual tracking, image understanding and activity recognition, or freely moving objects like modeling movement behavior of animals.
In [10] provided the strong mathematical and logical bases for defining the general spatiotemporal topological relations and divided them into two categories, namely, stable and unstable spatiotemporal relations. Both stable and unstable spatiotemporal relations play an important role in modeling the spatiotemporal events. World is represented as situation (a primitive event), and action is simply a function from one situation to another. A single snapshot represents a primitive event at an instant . Events are embedded in time, they have temporal boundaries, they have their relationship to time. They do not occupy space, but they are related to space.
Spatiotemporal events are defined as composite events, how their different parts (primitive events) are interrelated. A property (P,t) (property P holds at time instant ) is used along with the instantaneous temporal logic. The primitive events are defined for each snapshot during an interval using the Allenβs temporal logic and defining the relation (P,T) (property P holds during the time interval T) [17], and a relationship between holds and _at can be represented as , . This provides us that a property holds for an interval if it holds for every point during the interval. In this method, interval temporal logic is used, and a primitive temporal interval is defined, if is a zero duration interval, then it represents a snapshot.
Motion events are the subclasses of spatiotemporal relations with a temporal ordering in a primitive actions. Motion events, they do not formulate the necessitate of a calculus, they are only logical representation and temporal ordering of existence of different primitive events. Modeling the motion events, where property (P) changes at each instant, it is more suitable to use the sequential logic and use relation, (event occurs before in S during time ). Composite events are the initial conditions dependant, when an initial primitive event occurs at a certain time point , it set up the superclass and name of the possible composite event to be happening.
Topological relations have a certain type of locative symmetries, they do not explain the symmetric location of path and motion direction of argument objects. To remove the symmetry of spatial relations about the locative perspective, relevant spatial orientation is added. In language semantics, motion events are divided into three classes: an initial, the median, and terminal [25]. Some sentences can be explained with the help of a single directional relation such as enter, release, touch, and some need two directions like cross and graze.
We used CTD method [11] to develop such motion events, where topological components play role for defining the motion events, directional components are used to overcome the locative symmetries and locative perspective and for other class of motion events, topological components can be used for controlling variables, and directional components become important. We hope this paper will create a bridge between the two approaches of modeling the spatiotemporal events, approach based on interval logic, and that of point logic.
3. Preliminary Definitions
In this section, we recall some basic definitions which are frequently used throughout the remainder of the paper.Fuzzy set: a fuzzy set in a set is a set of pairsFuzzy membership function: a membership function in a set is a function . Different fuzzy membership functions are proposed according to the requirements of the applications. For instance, Trapezoidal membership function is defined as it is written as , where .Force histogram: the force histogram attaches a weight to the argument object that this lies after in direction , it is defined as The definition of force histogram , directly depends upon the definition of real-valued functions , and used for the treatment of points, segments, and longitudinal sections, respectively [26]. These functions are defined as where and represent the number of segments of object and object , respectively, and variables (x, y, z) are explained in Figure 1. These are the definitions of Force histograms, directly depending upon the definition of function . is actually a real-valued function.
4. Combined Topological and Directional Relations Method
In this section, we explain different steps of the combined topological and directional relations method. This explains different terms used in computation of combined topological and directional relations.
4.1. Oriented Lines, Segments, and Longitudinal Sections
Let and be two spatial objects and , where is any real number and . is an oriented line at orientation angle , and is the intersection of object and oriented line . It is denoted by , called segment of object and length of its projection interval on x-axis is . Similarly for object , where is segment and length of its projection interval on x-axis is , is the difference between the minimum of projection points of and maximum value of projection points of (for details [27]).
In case of polygonal object approximation can be calculated from intersecting points of line and object boundary, oriented lines are considered which passes through at least one vertex of two polygons. If there exist more than one segment, then it is called longitudinal section as in case of in Figure 1.
4.2. Allen Temporal Relations in Spatial Domain and Fuzziness
Allen [5] introduced the 13 jointly exhaustive and pairwise disjoint (JEPD) interval relations. These relations are with meanings before, meet, overlap, start, finish, during, equal, during_by, finish_by, start_by, overlap_by, meet_by, and after. All the Allen relations in space are conceptually illustrated in Figure 2.
These relations have a rich support for the topological relations and represent the eight topological relations in one-dimensional spatial domain. Fuzzy Allen relations are used to represent the fuzzy topological relations, where vagueness or fuzziness is represented at the relationβs level.
Fuzzification process of Allen relations do not depend upon particular choice of fuzzy membership function. Trapezoidal membership function is used due to flexibility in shape. Let be an Allen relation between segments (segment of an argument object) and (segment of an reference object), is the distance between and its conceptional neighborhood. We consider a fuzzy membership function . The fuzzy Allen relations defined in [28] as where , is the length of segment (), is the length of segment (), and are computed as described in Section 4.1.
Most of relations are defined by one membership like , , , and . In fuzzy set theory, sum of all the relations is one, this gives the definition for fuzzy relation . These are the topological relations which represent the fuzziness at relationβs level, for example, here Meet topological relation is represented based on nearness, and length of the smaller interval defines the smooth transition between the Meet(Meet_by) and before(after) relation. In spatial domain, before(after) are called the disjoint topological relations. These relations have the following properties:
Eight relations are possible combination of eight independent Allen relations in one-dimensional spatial domain. These relations and their reorientation show that the whole 2D space can be explored with the help of 1D Allen relations using the oriented lines varying from (0, ).
4.3. Combining Topological and Directional Relations
Eight topological relations represented in point set topology or point less topology between object pair are represented in one-dimensional space by the Allenβs temporal relations in spatial domain. We extend these Allen relations for two-dimensional objects through logical implication, where a object is decomposed into parallel segments of a 1D lines in a given direction, and the relation between each pair of line segments is computed.
The process of object decomposition is repeated for each direction varying from 0 to , two-dimensional topological relations are then defined as it provides us with information about how the objects are relatively distributed. These relations are not jointly exhaustive and pairwise disjoint (JEPD), to obtain JEPD set of topological and directional relations an algorithm was advocated in [11], it provides us with the JEPD set of relations. Objects are approximated through the polygon object approximation. Different steps of computing the combined topological and directional relations are(i)fix angle and draw lines passing through the vertices of polygons representing the objects;(ii)for each line, compute the variables as depicted in Section 4.1 and compute Allen relation for each segment as given by (5). In case of longitudinal sections, use fuzzy operators to integrate the information, usually the disjunction operators are suitable. These relations are computed for each line in a direction, then obtained information is integrated into a single value. Normalize these relations for a direction by dividing sum of all Allen relations to each Allen relation;(iii)these normalized fuzzy Allen relation is then multiplied to a fuzzy directional set to find the degree of an Allen relation in a direction;(iv)for qualitative directions, this information is summarized, and different topological relations with directional contents are defined, such as where represents a topological relation, and represents the direction is the reorientation of ; (v)this information is represented in a matrix, this matrix represents fuzzy spatial information;(vi)these fuzzy spatial relations are defuzzified by an algorithm, this provides us with final topological and directional relations between the object pair. These topological and directional relations are JEPD.
This model describes well the possible topological relations between every sort of objects.
5. Spatiotemporal Relations
Spatiotemporal relations can be defined as spatial relation holds for an interval, that is, relation holds for a certain time interval, and it does not change. In spatiotemporal object theory it is defined as (P) a spatial relation is a relation holding between all temporal slices of two entities during the relevant period. All eight spatiotemporal relations are defined in terms of theorems.
5.1. Spatiotemporal Relation
Theorem 1. A spatiotemporal disjoint relation between object pair holds during the interval , that is, holds.
Proof. A spatiotemporal relation is defined as object pair are disjoint during the interval if ( is temporally equivalent to ). Now let us consider the partition of interval , then its partition can be taken as . Each represents discrete points of interval , and this representation is equivalent to a snapshot. Typically a snapshot is a sampling process, which represents zero duration temporal slice of a spatiotemporal object. There are snapshots in interval , as a result a disjoint topological relation exists for each snapshot separately. Thus, for all ) holds.
Let us consider snapshots where the temporal ordering holds, that is, such that and all these points form partition of an interval . If the disjoint topological relation holds at discrete points, it means that . If the disjoint topological relation holds between object pair, it means that both the objects are temporally equivalent (). Hence, holds during the whole interval .
Theorem 2. A spatiotemporal relation Meet M(XY,T) holds such that for all holds.
Proof. A spatiotemporal relation meet holds between object pair over interval , where . Let be partition of interval , if holds, then a stable topological relation holds. We consider on contrary, that , where the topological relation does not hold, but it holds at , then according to the temporal logic and continuity of topological relations . This shows that any of the three relations is possible (stands for future position). If holds, then the whole spatiotemporal relation is changed, and it becomes the spatiotemporal partial overlap relation. This possibility is ruled out. In other case, spatiotemporal relation remains meet, as is an arbitrary variable, this shows the minimum condition. Hence, s.t. holds.
Let us consider that there are snapshots in an order, which construct an interval . Now consider that there exists at least one snapshot during whole interval, where spatial meet relation holds, and for all other snapshots, the spatial relation is disjoint. This shows that during temporal interval , the unstable spatiotemporal meet relation holds. It satisfies the minimum conditions for a spatiotemporal meet relation, hence holds during interval .
Theorem 3. A spatiotemporal partial overlap (PO) relation holds over interval , that is, .
Proof. Spatiotemporal relations have the spatial and temporal boundaries. A stable spatiotemporal relation holds during the temporal slice, If it holds at every point of the interval. As temporal slice is the union of finite points of temporal domain, spatiotemporal partial overlap holds during the whole slice, if this relation holds at least one sampling point (snapshot), at remaining points any of the spatial relation may exist. Hence (CO stands for complete overlap of objects (), s.t., ). If there does not exist such a , then the relation holds for every , which shows that a stable PO topological relation holds.
We suppose on contrary that holds. It means that at all points either the binary topological relations are complete overlap or disjoint and meet. If the relations are complete overlap, that is, for all holds, then the spatiotemporal relation will be a part of complete overlap. In case of other choice that , s.t. or for all s.t holds, then the spatiotemporal topological relation will be stable or unstable meet and for case for all s.t., the topological relation will be disjoint. The choice, that s.t. and s.t. holds is impossible because in a such a case , s.t., holds (continuity of spatial relations).
Theorem 4. A spatiotemporal tangent proper part (TPP) relation holds over interval , that is,TPP,such thatTPPfor all , NTPP holds.
Proof. A spatiotemporal topological relation holds between the object pair during the interval . Now let us consider that interval consist of n snapshots, if this relation holds for every snapshot then a spatiotemporal stable topological relation holds. In other case, there are two possibilities that for all , there exists a topological relation , and for either the topological relation is or due to continuity of topological relations between moving objects. For , the spatiotemporal topological relation is changed, and it becomes the spatiotemporal PO topological relation, this possibility is ruled out. It remains that , if this relation holds and is an arbitrary point, so the relation becomes the unstable spatiotemporal .
Consider that such that holds. We consider on contrary that such that or does not holds. Then, possible topological relations at are similarly for . Other possibilities are ruled out due to the continuity of topological relations, and does not hold because objects are considered under motion, and expansion or zooming of one object is not allowed.
In case the topological relation holds, then the whole spatiotemporal relation over the interval becomes partial overlap. Similarly for instant and is an arbitrary point, so this is impossible for whole the interval . For the topological relation , the spatiotemporal relation becomes the stable spatiotemporal TPP.
Theorem 5. A spatiotemporal nontangent proper part (NTPP) relation holds over interval , that is, NTPP for all,NTPP holds.
Proof. Let us suppose on contrary that s.t. does not hold, and at temporal points the relation , holds. Then continuity of spatial relations forces the existence of or spatial relations. This contradicts the existence of the spatiotemporal relation. Hence, holds.
It is given that holds. If a spatial relation between object pair holds at every point of the interval, then it means that it holds throughout the interval, that is, holds.
Theorem 6. A spatiotemporal tangent proper part inverse (TPP) relation holds over interval , that is, s.t. holds.
Proof. Proof is similar to the , just replace by and by .
Theorem 7. A spatiotemporal nontangent proper part inverse (NTPPI) relation holds over temporal interval , that is, holds.
Proof. Proof is similar to the .
Theorem 8. A spatiotemporal relation equal(EQ) holds between the object pair , , s.t., holds.
Proof. We suppose on contrary that there exists a , where the relation does not hold. It shows that there are two possibilities that either the relation at is a complete overlap or partial overlap. If the relation at is complete overlap, then the spatiotemporal relation becomes or . In the second case, the spatiotemporal relation becomes the during the whole interval. Thus, both cases prove the contrary conditions, hence does not hold, that is, for all holds.
() Converse of this proof is very simple and straight forward. Let be the interval for which we have to define the spatiotemporal relation, both the objects are temporally comparable . Let be an arbitrary point of the interval and relation holds. Since is an arbitrary point so, the relation holds throughout the interval , that is, holds.
6. Visual Interpretation: A Three-Dimensional View
Geometrical figures can better elaborate concepts, a moving object changes its position at each instant . These objects in a spatiotemporal domain can be represented by their envelops, a two-dimensional object becomes volume. Here, spatiotemporal meet and partially overlap relations are represented by their envelops in Figures 3(a)β3(d) and 4(a)β4(h). These are possible representation of motion events. Spatial relations between moving objects are used in modeling the motion verbs or motion events in natural language processing. A set of motion relations is introduced that capture semantic between pairs of moving objects. This information is useful about reasoning the moving objects.
(a) touch
(b) excurse
(c) snap
(d) release
(a) enter
(b) leave
(c) cross
(d) graze
(e) into
(f) out of
(g) melt
(h) spring
7. Modeling Motion Classes
Visual images may illustrate cases of a definition, giving us a more visual grasp of its applications. They may help us understanding the description of a mathematical situation or steps in reasoning. These relations can be defined as the transection of relations at time to . This change may be in topological or metric relations, and different classes of spatial relations, between moving objects have been defined [12, 13, 29, 30]. Motion classes based on intuitive logics or motion verbs have been defined in [31] by Phillipe Muller and Ralf H. GΓΌting and Markus Schneider used in database. We define in this paper only the motion events, where topological relations capture changes between situations. These motion events can be defined using the predicates , ,, β and . In next section, represents the sequence event.
7.1. Unstable Meet Spatiotemporal Relation
Unstable spatiotemporal relation is a relation where objects changes their states at each time instant. A spatiotemporal meet relation is characterized by different motion events depending upon the logical and temporal order of different states or primitive events. Touch(XY,T): A spatiotemporal meet relation can be characterized as a motion event Touch, s.t. , where primitive events occur in an order and defined as
An institutive view of this spatiotemporal relation is shown in Figure 3(a). This relation can be expressed by a single direction, where a meet topological relation holds. It meansSnap(XY,T): A spatiotemporal meet relation is called a Snap if such that
A geometric representation is shown in Figure 3(c). This relation can be expressed by a single direction, where a meet topological relation holds. It meansRelease(XY,T): A spatiotemporal is called , read as releases during interval if it has a certain temporal ordering, such that
A three-dimensional geometric view of this relation is shown in Figure 3(d). This relation can be expressed by a single direction, which is the destination direction. For example, object ββ (motion event) object towards East (destination direction). Direction for such a relation is defined asBypass(XY,T): A spatiotemporal is called a , read as bypasses during interval if it has a certain temporal ordering, that is, such that<<<t4
This relation can be expressed by a single direction, where a meet topological relation holds. It meansExcurse(XY,T): A spatiotemporal Meet(XY,T) is called a , read as excurse during interval if it has a certain temporal ordering, an intuitive view of this relation is shown in Figure 3(b).
This relation is expressed by an initial and destination directions, the direction for this relation can be defined as
7.2. Unstable Overlap Spatiotemporal Relation
Enter(XY,T): An unstable spatiotemporal overlap relation is called Enter, generally denoted by Enter and read as β enters in during interval .β If , then relation is defined asAn intuitive view of this relation is shown in Figure 4(a). This relation can be expressed by a single direction because the destination point is inside and can be expressed without direction, a direction for the Enter spatiotemporal event is the direction where a meet topological relation holds, that is,: A spatiotemporal partial overlap relation is called Leave, denoted as Leave(XY,T) β leaves during interval β. If , then relation is defined as
An intuitive view of this relation is shown in Figure 4(b). This relation can be expressed by a single direction which is the destination point, that isCross(XY,T): A spatiotemporal relation Cross(XY,T) β crosses during the interval .β Its geometric view is given in Figure 4(c). If , then relation is defined as
This spatiotemporal relation is expressed by a initial as well as destination direction such as object crosses (motion event) object from north (direction) towards east (direction) during the interval :: A spatiotemporal relation(XY,T) read as β get into during the interval .β If , then relation is defined as
Its three-dimensional geometric view is given in Figure 4(e). This relation can be expressed by a single direction in language semantics, where a meet topological relation holds. For example, object get into (spatiotemporal event) object from north (direction). It means.: A spatiotemporal relation (XY,T) read as β comes out of during the interval ,β its intuitive view is considered in Figure 4(f). If , then relation is defined as
This relation can be expressed by a single direction. Object go out_of (motion event) object towards east (direction), where a meet topological relation holds. It means : A spatiotemporal relation (XY,T) read as β melts during the interval β. If , then relation is defined as
An intuitive view of this relation is shown in Figure 4(g). This relation can be expressed by a single direction because its destination point is dimensionless. This can be its direction, where initial spatial relation holds:: A spatiotemporal relation Spring also called Separate read as β separates during the interval .β If , then relation is defined as
Its three-dimensional geometric view is given in Figure 4(h). This relation can be expressed by a single direction because its destination point is dimensionless. This can be its direction, where terminal spatial relation holds:: A spatiotemporal relation Graze read as β grazes during the interval .β If , then relation is defined as
This relation is represented in a three-dimensional perspective in Figure 4(d). This spatiotemporal relation is expressed by an initial as well as destination direction such as object grazes (motion event) object from north (direction) toward east (direction):
8. Conclusion and Future Work
In this paper, we define spatiotemporal relations, where the discrete time space is used. These spatiotemporal relations are topologically stable or unstable. Motion events represent the subclass of spatiotemporal relations, and certain number of motion events represent the class of a topologically unstable and stable spatiotemporal relations. In these spatiotemporal relations temporal order of holding a primitive event is more important, and this order has a pivotal role in natural language semantics. Topological relations have a locative symmetries, to remove these symmetries we add a directional components. In this paper, CTD method [11] is used to model the motion events, where topological and directional information for a snapshot are captured at the same abstract level. Hopefully this work will bring a significant change in video understanding, modeling video events, and other related areas of research.