The goal of the project led by the University of Southern California is to develop an automated and efficient approach to medical chatbot evaluation with evaluation components designed to automatically generate questions that reflect a patient-community’s needs and scoring components that reflect common challenges (e.g., hallucinations, omissions, miscalibrated linguistic certainty) as well as pragmatic needs specific to the community. To support this development, USC will engage with stakeholders to understand their information needs and how they are addressed. Immediate use cases include pediatric infectious disease and cystic fibrosis.