How Do We Know Which Object Went Where? Investigating Higher-Level Influences on Object Correspondence

Abstract

Our visual system enables a coherent and stable perception of the world around us. Because our environment changes constantly, the visual system has to solve the correspondence problem, that is, determine which object went where through space and time. Our visual system's ability to solve this problem is remarkable because it has to infer our three-dimensional world from information projected on our two-dimensional retina. This information is ambiguous because the same two-dimensional retinal image can correspond to multiple different images in our three-dimensional world. In addition, the world around us is constantly in motion, and objects often occlude each other. Furthermore, the correspondence solution has a strong influence on how we perceive the world around us because it determines whether we perceive objects as one and the same object over space and time, that is, in motion, and in turn determines how we interact with our environment. To enable perception (and in the end, recognition) the visual bottom-up processing starts at a low level, where features are extracted from image-based information such as luminance constrast and orientation of edges. The information is further processed at an intermediate level, in which grouping of the features takes place, leading to object-based information, e.g., representation of an object with its true color (abstracted from the illumination of the surrounding) and combined contours represented in 3D. Finally at a high level, object recognition takes place by complementing perception with top-down information, such as semantic information. Regarding the correspondence process, it is still unclear which levels of visual processing are involved in solving the correspondence problem, notably whether and, if so, how higher levels of visual processing contribute to this process. The object-based correspondence theory suggests that correspondence could be solved at an intermediate level of visual processing using object-based information. In addition, attention is suggested to be a mechanism for establishing correspondence between objects based on their identity. To test the object-based correspondence theory within this thesis, three studies, including several experiments, were conducted. The Ternus display, an ambiguous apparent motion display, was used as a measure of correspondence. The Ternus display consists of three aligned elements, shifted by one element position from one frame to the next. Depending on how correspondence is established between the elements, different types of motion can be perceived (element or group motion). Study 1 investigated whether our viusal system uses object-based information, available at an intermediate level of visual processing, to solve correspondence. To do so, a (Ponzo-like) depth illusion background was used to manipulate the perceived size of the Ternus display, while keeping the (low-level) retinal size the same. Results showed an influence of the perceived size on the correspondence solution, which is evidence that, after size constancy is processed, mid-level information is used. Study 2 addressed the question of whether object-based information that is presented using different object histories prior the presentation of the Ternus display can also influence correspondence. Results showed that this is the case, suggesting that even information that is stored in object representations and not present in the image at the moment when correspondence is solved can influence correspondence. Study 3 investigated whether voluntary attention is able to influence how correspondence is established, as the object-based theory suggests an attention-mediated mechanism to track objects across space and time. A modified Ternus display was used, in which the elements within a frame were presented in different colors in such a way that they were compatible with group and element motion (group and element bias) at the same time. Particpants' task was to direct their attention toward a specific element. The results showed that the correspondence solution was shifted toward the motion percept matching the bias of the element that was attended, therefore suggesting an influence of attention on correspondence in line with the object-based theory. In sum, this thesis provides new evidence for the influence of object-based information at an intermediate level of visual processing and for an attention-mediated correspondence mechanism in line with the object-based correspondence theory. Based on these findings and previous research, a two-level correspondence process model is proposed, incorporating both low- and higher-level visual processing. Finally, open questions and research ideas with regard to the studies conducted and the suggested model of the correspondence process are discussed.