With recent advances in machine learning, automated dialogue systems have become more able to produce coherent language-based interactions. However, most work on automated spoken language understanding uses still only text transcriptions, i.e., just the lexical content of speech. This ignores the fact that the way we speak can change how our words are interpreted. In particular, speech prosody --e.g. pitch, energy, and timing characteristics of speech -- can be used to signal speaker intent in spoken dialogues. In fact, prosodic features can help automatic detection of both dialogue structure and speaker affect/states. In this talk, I will discuss our recent work on how we can combine non-lexical and lexical aspects to speech to improve speech understanding tasks, such as emotion recognition, and how new approaches to self-supervised learning from speech might be able to help us make the most of the true richness of speech.

That cortical activity tracks the dynamics of auditory stimuli is reasonably well established. In speech and music perception, this phenomenon produces reliable coupling between the acoustic envelope of the sound input and the corresponding cortical responses. However, it remains unclear to what extent that neural tracking reflects low-level acoustic properties, as opposed to more abstract linguistic structures. In this talk, I will discuss a series of studies aimed at assessing the impact of higher-order linguistic information on the cortical processing of speech and music sounds. I will demonstrate methodologies for disentangling neural responses to stimulus properties at different abstraction levels, deriving multiple objective indices for probing auditory perception with experiments involving natural speech and music listening. I will then describe recent developments of these measures in the context of developmental research.