In following video, he is describing research studying every single movement of his new born child by cameras placed all over his house. To analyze the audio-video data, Roy and his MIT team developed numerous deep machine learning algorithms designed to uncover the relationships between spoken language and context. Their method led to recording and describing circumstances in learning new words.
Roy moved to company Bluefin, where he started to implement discoveries from MIT in commercial use. Using deep machine learning to ground the meaning of comments pulled out from social media, by looking at the context of words expressed by individuals, they can use the meaning of these words to connect comments back to the events, people, products, brands, and viewing contexts that caused those words to be expressed in the first place.
Their technology applies language grounding techniques to achieve successful mapping of social media commentary to mass media stimuli on TV, specifically TV shows and commercials. The data produced by this mapping of social media to TV media has previously unknown potential for brands, advertising agencies, and TV networks. They can basically see how the world is reacting to their broadcasted content in nearly real time.