In the Beginning was the Deed: From Sensorimotor Interactions to Integrative Spatial Encodings

Abstract

Goal-oriented behavior requires reliable predictions regarding action outcomes. The theory of event segmentation and the free energy principle allow to derive hypotheses regarding the formation and maintenance of predictive models and their representational format. According to the free energy principle, cognitive systems constantly try to infer the causes of perceived sensations. This results in the formation of predictive models based on sensorimotor experience. Even if there is an ongoing debate regarding the representational format of these models, an integrative spatial code, which integrates different modalities in an abstract representation seems plausible. The integration process is assumed to be biased towards behaviorally relevant modalities. Moreover, a striving for consistency is assumed to maintain unambiguous states. Besides the representational format, the prediction process itself is of central interest. According to the event segmentation theory, cognitive systems segment the stream of sensorimotor information along significant changes, so-called event boundaries. Hence, it seems likely that predictions are carried out in terms of a simulation of the next, desired event boundary within the proposed integrative spatial code. The spatial code might support mental simulation in general, providing sensorimotor grounding to higher cognitive functions – as proposed by theories of embodied cognition. The proposed properties of the integrative spatial code were investigated in four studies, concerning the questions (i) whether multisensory integration is biased towards action-relevant modalities, (ii) how representations are kept consistent across frames of reference in case of multisensory conflict, (iii) if predictive models provide an anticipatory, event-like structure in the service of behavior control, and (iv) how different modalities are combined through a spatial code in the service of predictive simulations. The obtained results confirm the assumptions regarding the proposed integrative spatial code. The combination of the free energy principle and the theory of event segmentation seems a viable approach to account for the emergence of a predictive, integrative spatial code from sensorimotor interactions. The results allow the derivation of design principles for an artificial spatial reasoning system and the developed experimental paradigms allow further investigations of the causal role of spatial models in higher cognitive functions.