ICS Colloquium: Peter Foltz, PhD, University of Colorado Boulder

Title: Reflections on the nature of measurement in language-based automated assessments of patients' mental state and cognitive function

Presenter: Peter Foltz, PhD, Research Professor, Institute of Cognitive Science, University of Colorado Boulder

Abstract: Advances in computational language processing methods have enabled new approaches to the measurement of mental processes. However, the field has primarily focused on model accuracy in predicting performance on a task or a diagnostic category. Instead the field should be more focused on determining which computational analyses align best with the targeted neurocognitive/psychological functions that we want to assess. In this talk, I reflect on two decades of applying language-based assessment of mental state and cognitive function by addressing the questions of what we are measuring, how it should be measured and why we are measuring the phenomena. I'll advocate for a principled framework for aligning computational models to the constructs being assessed and the tasks being used, as well as defining how those constructs relate to clinical states. I further examine the assumptions that go into the computational models and the effects that model design decisions may have on the accuracy, bias and generalizability of models for assessing clinical states. Finally, I'll describe how this principled approach can further the goal of transitioning language-based computational assessments to part of clinical practice while gaining the trust of critical stakeholders.

Bio: Dr. Peter Foltz is Research Professor at the University of Colorado’s Institute of Cognitive Science and Executive Director of the National Science Foundation AI Institute for Student-AI Teaming.  His work covers machine learning and natural language processing for educational and clinical assessments, large-scale data analytics, cognitive skills in reading and writing, team collaboration, and 21st Century skills learning,  Much of his work has focused on NLP techniques for automatically analyzing the meaning of language through writing and speaking. The approaches are used for assessing abilities, for providing feedback, and for understanding underlying cognitive mechanisms in the brain. Peter was a Founder and Chief Scientist at Knowledge Analysis Technologies, which was acquired by Pearson Education and then served as Vice President for Research at Pearson. He has served as the content lead for the framework development for Organisation of Economic Cooperation and Development’s (OECD) Programme for International Student Assessment (PISA) assessments, including the 2018 Reading Literacy assessment, the 2015 assessment of Collaborative Problem Solving, and a new assessment of reading literacy for developing countries. He has been guest editor for a number of journals including International Journal of AI in Education and Discourse Processes as well as co-editor of the recent Handbook of Automated Scoring: Theory inito Practice. His work has been covered widely in the press including The New York Times, Time Magazine, NPR, and Science.  He previously also worked at New Mexico State University, Bell Communications Research, the Learning Research and Development Center at the University of Pittsburgh, Yale University, and the Harvard Institute for International Development.