Ongoing Research Directions

Human–AI Collaboration for Neurodivergent Employment (2023-present, NSF sponsored)

Problem Statement

Neurodivergent adults, particularly autistic individuals and adults with ADHD, often face significant challenges in job searching, workplace productivity, and social communication. These challenges arise not only from neurological differences but also from interactions with neurotypical individuals and environments that prioritize neurotypical ways of working and communicating. Common barriers include difficulties with planning, organization, sustained attention, workplace communication, interpreting social cues and implicit expectations, and navigating hiring practices and workplace norms that may not accommodate neurodivergent needs. Therefore, there is a critical need to design human–AI collaborative interventions that support neurodivergent individuals throughout the employment lifecycle, from job searching and preparation to workplace communication and productivity.

Vision and Approaches

This research area explores human–AI collaborative systems that support neurodivergent adults across the employment journey, while positioning AI as an adaptive collaborative agent that works with individuals and their trusted support networks. Our approach combines stakeholder-centered qualitative research, participatory design, data-driven analysis, and controlled system evaluation. In our autism employment research, we interviewed 20 autistic job seekers, supporters, and career experts, developed and evaluated four interactive design probes, and created a web-based system through which 20 autistic adults selected and annotated 200 job postings to identify difficult or ambiguous language. We also investigated how VR, human body doubles, and AI body doubles can support adults with ADHD by examining their effects on attention, motivation, accountability, safety, and task performance in complex work environments.

Impact

This research direction will deepen our understanding of the employment-related challenges faced by neurodivergent adults, inform the design of technologies tailored to their support needs, and provide practical assistance throughout the job-search process and in sustaining workplace productivity. By designing AI to complement human-in-the-loop support, this work can strengthen autonomy, confidence, and collaboration among neurodivergent workers, employers, and trusted support networks.

Collaborator(s) and Partner(s):

Elizabeth Foster (Melwood Inc.), Elizabeth Green (LinkTalent), Dave Caudel (Frist Center for Autism and Innovation at Vanderbilt University), Dasha Peppard (Center for Student Professional Development, Temple University), Andrew Hundt (Carnegie Mellon University)

Outcome(s):

[CSCW2026] Enhancing Interview Preparation for Autistic Job Seekers through Human-AI Collaboration
[NWRC2026] Understanding ADHD Productivity in Construction Work: Toward AI-enabled VR Interventions
[CHI2026] Lost in Translation: Understanding Autistic–Neurotypical Communication Style Differences in Job Postings
[CHI2026 EA] Rethinking Productivity Support for Workers with ADHD in the Construction: Preliminary Insights
[CHI2024] Collaborative Job Seeking for People with Autism: Challenges and Design Opportunities
[NWRC2024] Collaborative Design for Job-Seekers with Autism: A Conceptual Framework for Future Research

LLM-Driven Agentic System for MCI and Early-Stage Dementia Support

Developing an LLM-Driven Agentic System to Support Individuals with Mild Cognitive Impairment (MCI) or Early-Stage Dementia (2024-Present)

Problem Statement

The global population is aging, and a growing share of older adults live with mild cognitive impairment (MCI), which affects roughly 22.7% of people in the United States. Despite technological advances, individuals and their caregivers struggle to obtain personalized, context-aware guidance for unique behaviors and social environments.

Vision and Approaches

The long-term vision is to create an agentic system that supports individuals with MCI or early-stage dementia by blending state-of-the-art AI with continuous sensing and caregiver collaboration. The system combines three components: (1) LLM-Driven Conversational Intelligence—An LLM serves as the system's conversational core, drawing on physiological data, living environment, dietary habits, and social engagement to tailor advice, personalize reminders and encouragement (e.g., gentle exercise, social interactions), and maintain an empathetic dialogue; (2) Wearable-Based Behavioral Sensing—Wearables continuously monitor physiological signals and daily behaviors (heart rate, heart rate variability, stress levels, sleep quality, activity data via smartwatches), capturing a real-time picture of the user's state and triggering timely in-app responses or alerts; (3) Caregiver-AI Collaboration Loop—The system will incorporate caregiver observations and preferences into an adaptive intervention loop, providing insights and reducing monitoring burden.

Impact

By integrating personalized conversational support, continuous behavioral sensing, and caregiver collaboration, the system aims to deliver timely, context-aware interventions that enhance daily routines, promote brain-healthy behaviors, and support long-term cognitive and emotional well-being, laying the foundation for scalable, human-centered digital health solutions for aging populations.

Collaborator(s) and Partner(s):

N/A

Outcome(s):

[JApplGerontol2025] Utilizing Conversational AI Technology for Social Connectedness Among Older Adults: A Systematic Review

Multi-Robot and Multi-Video Sensemaking for Public Safety and Disaster Recovery

Multi-robot + Multi-video Sensemaking for Public Safety and Disaster Recovery (2023-present)

Problem Statement

Ground robots are increasingly available and autonomous, generating videos that can enhance situational awareness for public servants (e.g., police officers for public safety, disaster response for damaged infrastructure or people and animals in need). Robots can perform surveillance in areas where aerial images are unavailable. The challenge is that analyzing vast amounts of video from multiple robots and providing commands to them requires significant human attention. Current visual evidence collection through robots is increasing, but dedicated designs and technologies for supporting human reasoning in this context are ill-defined.

Vision and Approaches

The project aims to design, develop, deploy, and evaluate datasets, scenarios, and interactive systems that facilitate multi-robot + multi-video sensemaking for public sector workers to achieve situational awareness for both real-time decision-making and post-event investigations. Situational awareness applications include public safety (e.g., addressing assault, vandalism) and disaster recovery (e.g., clearing roads after a hurricane).

Impact

The ultimate goal is to improve the efficiency and effectiveness of group robot operations for public safety and disaster recovery, minimize officers' effort and risk, and foster seamless collaboration between humans, videos, and robots. This will help society leverage robots and videos for situational awareness and contribute to HCI, Robotics, and Computer Vision communities through a conceptual framework of multi-robot multi-videos for scenarios for social good, benchmark datasets and tasks, and advances in video understanding and interactive human-group robot control.

Collaborator(s) and Partner(s):

Steve Peterson (NIH), Michael Lighthiser (GMU)

Outcome(s):

Scalable Human-in-the-Loop and Actionable Explainable AI

Connecting AIs and Humans through Scalable Human-in-the-Loop and Actionable XAI (2021-present)

Problem Statement

An unexpected AI failure can have severe consequences, affecting human lives (safety, productivity, trust, ethics). Understanding AI's vulnerability is essential but challenging and resource-costly for Machine Learning engineers.

Vision and Approaches

The goal is to develop novel human-AI collaboration designs to help ML engineers investigate and fix AI vulnerabilities more efficiently. Two main methodological aspects: (1) Scalable Human-In-The-Loop—maintain a reasonable level of human input in the modeling process, especially with large-scale data; (2) Actionable XAI (explainable AI)—enable humans to convert learned insights into direct actions that update the AI model after assessing its decision-making using XAI techniques.

Impact

The proposed solution can enhance human-AI interaction, provide ML engineers with better capability of "communicating" with future AI models, and empower ML engineers across diverse datasets (images, videos, text) to use more reliable and controllable AI.

Collaborator(s) and Partner(s):

Liang Zhao (Emory University), Young-Ho Kim (Naver)

Outcome(s):

[ICDM2021] GNES: Learning to Explain Graph Neural Networks
[KDD2022] RES: A Robust Framework for Guiding Visual Explanation
[CSCW2022] Aligning Eyes between Humans and Deep Neural Network through Interactive Attention Alignment
[CSCW2023] Designing a Direct Feedback Loop between Humans and Convolutional Neural Networks through Local Explanations
[CSCW2024] 3DPFIX: Improving Remote Novices' 3D Printing Troubleshooting through Human-AI Collaboration

COMPLETED RESEARCH DIRECTIONS

Supporting Comic Professionals' Colorization through Human-AI Collaborative Designs (2021-2024)

Problem Statement

Comic production has rapidly transitioned from paper-based towards digital, increasing the need to generate fully colorized output. This means modern comic professionals put substantial effort into comic colorization. While several AI-driven approaches for supporting comic colorization have been proposed, professionals' colorization workflows are still manual and labor-intensive.

Vision and Approaches

This research aimed at developing interactive human-AI collaboration systems built for comic professionals' colorization. In doing so, we closely observed comic professionals' workflow and identified a set of specific stages, such as flatting, shadowing, and lighting. Based on the evidence, we have developed two systems: FlatMagic and ShadowMagic that can respectively boost comic professionals' flatting and shadowing productivity than the current state-of-the-art.

Impact

This research discovered the general workflow of comic colorization. In particular, the empirical studies found that flatting and shadowing are particularly laborious, and comic professionals may be likely to use AI-driven automation if the results are reasonably good. This research also derived the notion of Intermediate Representation; the potential threat of "merging" multiple stages of a user workflow to a single AI-driven automation. In particular, providing AI-driven automation that combines multiple stages of colorization can result in professionals giving up using AI, as they cannot have intermediate "control points" for revising the AIs' outcomes if they are not entirely compelling. The research outcomes help future researchers understand how to support comic professionals effectively. The outcomes also provide interactive design artifacts that can improve professionals' flatting and shadowing productivity.

Collaborator(s) and Partner(s):

John Joon Young Chung (MidJourney), Lee Ho (Studio Zilpung, CEO)

Outcome(s):

[CHI2022] FlatMagic: Improving Flat Colorization through AI-driven Design for Digital Comic Professionals link
[UIST2024] ShadowMagic: Designing Human-AI Collaborative Support for Comic Professionals' Shadowing link