Commalla: Communication for all


Collaboration with: Kristy Johnson; Advisors: Rosalind Picard, Pattie Maes


Over 1 million people in the U.S. are non- or minimally speaking with respect to verbal language (mv*) including but not limited to people with autism spectrum disorders (ASD), Down syndrome (DS), and other genetic disorders.  mv* individuals communicate richly through vocalizations that do not have typical verbal content, as well as through gestures and other modalities. Some vocalizations have self-consistent phonetic content (e.g., “ba” to mean “bathroom”) and others vary in tone, pitch, and duration depending on the individual’s intended communication and affect.  

We present, to our knowledge, the first project studying communicative intent and affect in real-world vocalizations that do not have typical verbal content for mv* individuals. Interviewed parents of mv* children cited miscommunication with people who do not know their child well as a major source of stress. Our long-term vision is to design a device that can help others better understand and communicate with mv* individuals by training machine learning models using primary caregivers’ unique knowledge of the meaning of an individual’s nonverbal communication. Our focus is currently on developing personalized models for small, noisy datasets to classify vocalizations using in the moment live labels from caregivers via the Commalla labeling app. As part of this work, we are developing scalable methods for collecting and live labeling naturalistic data, and processing methods for using the data in machine learning algorithms. We are currently piloting and refining our data collection, machine learning models, and vision with a small number of families through a highly participatory design process

Note: This project was previously referred to as "ECHOS"



Narain, J.*, Johnson, K.*, Ferguson, C., O’Brien, A., Talkar, T., Zhang Weninger, Y., Wofford, P., Quatieri, T., Maes, P., and Picard, R. “Personalized Classification of Real-World Vocalizations from Nonverbal Individuals”. Accepted. Proceedings of the 22nd International Conference on Multimodal Interactions, 2020. *Equal Contribution.

Narain, J.*, Johnson, K., O'Brien, A., Wofford, P., Maes, P., and Picard, R.  Nonverbal Vocalizations as Speech: Characterizing Natural-Environment Audio from Nonverbal Individuals with Autism.  Accepted.  Laughter  and Other Non-verbal Vocalisations Workshop 2020.  *Equal Contributation.

Johnson, K*., Narain, J.*, Ferguson C., Picard, R., and Maes, P.  The ECHOS Platform to Enhance Communication for Nonverbal Children with Autism: A Case Study.  Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Case Studies).  *Equal Contribution.

Narain, J*., Johnson, K*., Picard, R., and Maes, P.  “Zero-Shot Transfer Learning to Enhance Communication for Minimally Verbal Individuals with Autism using Naturalistic Data”.  NeurIPS 2019 Joint Workshop on AI for Social Good.  *Equal Contribution.

Johnson, K.*, Narain, J.*, Picard, R., and Maes, P.  "Augmenting Natural Communication in Nonverbal Individuals with Autism".  Abstracts of the International Society for Autism Research (INSAR) Meeting.  2020.  Accepted Technology Demonstration.   *Equal Contribution.

Narain, J.*, Johnson, K.*, Ferguson C., Picard, R., and Maes, P.  "Naturalistic Communication in Nonverbal Children with Autism".  2019 Neurodevelopmental Disorders Symposium, Boston Children's Hospital.  November 15, 2019.  *Equal Contribution.