Research

Speaker Diarization in Static and Egocentric Videos via Speech-Face Alignment

Zeyang Zhang, Xavier Yin, Zhaobo Zheng, Kumar Akash, Teruhisa Misu, Carlos Busso
ACM International Conference on Multimodal Interaction (ICMI '26), Napoli, Italy, 2026

The ubiquity of personal devices, including smart glasses, opens up opportunities for multimodal processing in highly heterogeneous and dynamic settings. We investigate multimodal solutions that effectively leverage multiple microphones and cameras to support collaborative human tasks. A fundamental task in this context is speaker diarization, which determines who is speaking at any given time. We introduce static-egocentric audio-visual diarization (SEAD), a novel approach for speaker diarization that achieves multi-channel audio diarization by fusing audio information with facial information extracted from static exo- and dynamic egocentric video recordings. SEAD incorporates face tracking, cross-video and cross-audio person matching, and audio-visual fusion, and uses a fine-tuned audio-visual active speaker detector that recovers missed speech, suppresses false alarms, and reduces speaker confusion. Evaluated on a novel multi-person, multi-perspective group collaboration dataset, SEAD improves diarization performance compared to an audio-only baseline, reducing diarization error rate (DER) from 28.65% to 15.83% and the Jaccard error rate (JER) from 29.84% to 17.05%. Our findings demonstrate the potential of combining static and dynamic video perspectives to strengthen multimodal diarization in collaborative scenarios.

STArK: Towards Synthesizing Articulatory Kinematics from Text

Xavier Yin, Carlos Busso
Interspeech 2026 (Oral), Sydney, Australia, 2026

Recent deep learning approaches in modeling articulatory kinematics have shown success in mapping speech audio to electromagnetic articulography (EMA) kinematics, but these methods are limited by the scarcity of high-quality articulatory datasets. We introduce synthesizing text to articulatory kinematics (STArK), which offers a novel approach in generating articulatory kinematics directly from text. We train STArK on the LibriTTS-R dataset with pseudo-labeled EMA features derived from a pre-trained speech-to-articulation model. STArK is the first model to demonstrate high-quality articulatory kinematics generation from text, achieves comparable performance to existing multi-speaker text-to-speech models at speech synthesis, and supports voice cloning without seeing speaker embeddings during training.

DOI Code Demo

Standard Language Ideology in AI-Generated Language

Genevieve Smith*, Eve Fleisig*, Ishita Rustagi, Xavier Yin
*Equal contribution.
2026 ACM Conference on Fairness, Accountability, and Transparency (FAccT '26), 2026

Large language models (LLMs) generate text that reinforces standard language ideology: a bias towards certain language varieties that are granted more prestige, authority, and legitimacy than others. This paper contributes a sociotechnically grounded faceted taxonomy that illustrates how generative AI systems reproduce standard language ideology and its societal implications. We introduce the concept of standard AI-generated language ideology to explain how AI systems confer legitimacy on certain language varieties while marginalizing others, structuring patterns of performance disparity, stereotyping, appropriation, and erasure. We then discuss ongoing tensions around what constitutes desirable system behavior, as well as advantages and drawbacks of generative AI tools attempting or refusing to imitate different language varieties. To address the power relations shaping generative AI, we offer recommendations that emphasize accountability, community agency, control, and ownership.

DOI

MAGIC: Multi-Agent Argumentation and Grammar Integrated Critiquer

Joaquín Jordán*, Xavier Yin*, Melissa Fabros*, Gireeja Ranade, Narges Norouzi
*Equal contribution.
16th Symposium on Educational Advances in Artificial Intelligence (EAAI-26), co-located with AAAI-26, Singapore, 2026

Automated Essay Scoring (AES) and Automatic Essay Feedback (AEF) systems aim to reduce the workload of human raters in educational assessment. However, most existing systems prioritize numerical scoring accuracy over feedback quality and are primarily evaluated on pre-secondary school level writing. This paper presents Multi-Agent Argumentation and Grammar Integrated Critiquer (MAGIC), a framework using five specialized agents to evaluate prompt adherence, persuasiveness, organization, vocabulary, and grammar for both holistic scoring and detailed feedback generation. To support evaluation at the college level, we collated a dataset of Graduate Record Examination (GRE) practice essays with expert-evaluated scores and feedback. MAGIC achieves substantial to near-perfect scoring agreement with humans on the GRE data, outperforming baseline LLM models while providing enhanced interpretability through its multi-agent approach.

DOI

Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discrimination

Eve Fleisig*, Genevieve Smith*, Madeline Bossi*, Ishita Rustagi*, Xavier Yin*, Dan Klein
*Equal contribution.
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024

We present a large-scale study of linguistic bias exhibited by ChatGPT covering ten dialects of English (Standard American English, Standard British English, and eight widely spoken non-"standard" varieties from around the world). We prompted GPT-3.5 Turbo and GPT-4 with text by native speakers of each variety and analyzed the responses via detailed linguistic feature annotation and native speaker evaluation. We find that the models default to "standard" varieties of English; based on evaluation by native speakers, we also find that model responses to non-"standard" varieties consistently exhibit a range of issues: stereotyping (19% worse than for "standard" varieties), demeaning content (25% worse), lack of comprehension (9% worse), and condescending responses (15% worse). GPT-4 improves on GPT-3.5 in terms of comprehension, warmth, and friendliness, but also exhibits a marked increase in stereotyping (+18%). The results indicate that GPT-3.5 Turbo and GPT-4 can perpetuate linguistic discrimination toward speakers of non-"standard" varieties.

DOI

LLM-PBE: Assessing Data Privacy in Large Language Models

Qinbin Li*, Junyuan Hong*, Chulin Xie*, Jeffrey Tan, Rachel Xin, Junyi Hou, Xavier Yin, Zhun Wang, Dan Hendrycks, Zhangyang Wang, Bo Li, Bingsheng He, Dawn Song
*Equal contribution.
Proceedings of the VLDB Endowment (PVLDB), Vol. 17, No. 11, pp. 3201–3214, 2024

Large Language Models (LLMs) have become integral to numerous domains, significantly advancing applications in data management, mining, and analysis. Their profound capabilities in processing and interpreting complex language data, however, bring to light pressing concerns regarding data privacy, especially the risk of unintentional training data leakage. Despite the critical nature of this issue, there has been no existing literature to offer a comprehensive assessment of data privacy risks in LLMs. Addressing this gap, our paper introduces LLM-PBE, a toolkit crafted specifically for the systematic evaluation of data privacy risks in LLMs. LLM-PBE is designed to analyze privacy across the entire lifecycle of LLMs, incorporating diverse attack and defense strategies, and handling various data types and metrics. Through detailed experimentation with multiple LLMs, LLM-PBE facilitates an in-depth exploration of data privacy concerns, shedding light on influential factors such as model size, data characteristics, and evolving temporal dimensions.

DOI Code Website

An Adapted Deep Convolutional Neural Network for Automatic Measurement of Pancreatic Fat and Pancreatic Volume in Clinical Multi-Protocol Magnetic Resonance Images: A Retrospective Study with Multi-Ethnic External Validation

John Zhiyong Yang, Jichao Zhao, Reza Nemati, Xavier Yin, Kevin Haokun He, Lindsay Plank, Rinki Murphy, Jun Lu
Biomedicines, 10(11), 2991, 2022

Pancreatic volume and fat fraction are critical prognoses for metabolic diseases like type 2 diabetes (T2D). Magnetic Resonance Imaging (MRI) is a required non-invasive quantification method for the pancreatic fat fraction. The dramatic development of deep learning has enabled the automatic measurement of MR images. Based on MRI, we developed a deep convolutional neural network (DCNN) that can accurately segment and measure pancreatic volume and fat fraction. This retrospective study involved abdominal MR images from 148 diabetic patients and 246 healthy normoglycemic participants, with 2364 recognizable pancreas images labeled and pre-treated by an upgraded superpixel algorithm for a discernible pancreatic boundary. External validation datasets included 240 MR images from 10 additional patients. The DCNN achieved the highest dice similarity coefficient (91.2%) compared with any reported models, and was the first framework to measure intra-pancreatic fat volume and fat deposition, reaching the same segmentation level as expert clinicians.

DOI

Evaluation of Modified Convolutional Neural Network for Automatic Measurement of Pancreas Volume and Pancreatic Fat Deposition

Zhiyong John Yang, Dech Dokpuang, Rinki Murphy, Reza Nemati, Xavier Yin, Kevin Haokun He, Jun Lu
ISMRM 2021 Annual Meeting, Abstract #1865, 2021

Pancreatic fat quantification carries clinical significance for type 2 diabetes risk assessment, but has traditionally required an experienced operator and been a time-consuming task. This work develops a machine learning model trained on validated quantification methods to automate pancreas volume and pancreatic fat deposition measurement from MR images, with the identification rate significantly improved through the resulting program.

Abs