Joyce Horn Fonteles

Multimodal AI and human–AI collaboration for understanding how people learn together.

AI researcher · software architect · educator

PhD candidate in Computer Science, Vanderbilt University · NSF EngageAI Institute

I build AI systems that help people make sense of human interaction, systems that experts can inspect, correct, and come to trust.

I'm a Computer Science PhD candidate at Vanderbilt University, advised by Prof. Gautam Biswas, and a researcher in the NSF EngageAI Institute. My work combines large language models, computer vision, and speech analysis with human-in-the-loop annotation to interpret how students learn together in real classrooms: what they say, where they look, how they move. I design the analysis pipelines, the evaluation methods that tell us when an LLM's reading of a classroom can be relied on, and the interfaces that put those results in front of researchers and teachers.

I have spent fifteen years in industry: architecting healthcare and fintech platforms, leading the engineering and database teams behind 60+ judicial systems in the Brazilian Amazon, and training the public servants who took those courts from paper to digital. That shapes how I do research. I care about data that holds up in production, and about tools that real people actually adopt.

Human-in-the-loop

Human–AI partnership

A framework for interpreting learning behaviors from classroom recordings in which AI models propose interpretations, experts verify and correct them, and authority, labor, and inspectable evidence are allocated across every stage of the analysis pipeline.

Projects
GEM-STEP: multimodal analysis of embodied science learning2022 – present

A modular framework that combines computer vision, audio processing, and LLM-based late fusion to classify self- and socially-shared regulation behaviors, refined over time with chain-of-thought prompting and active learning from human feedback.

LLMs and embodied computational thinking2024 – present

AI tools that turn students' embodied exploration into executable code through natural-language-to-code workflows, with prompt engineering and multilingual guardrails for culturally aware, pedagogically aligned interactions. Curriculum co-design and pilots in the U.S. and Israel.

Explainable AI

Interfaces people can interrogate

Review tools that keep the evidence next to every AI judgment, so researchers and teachers can validate it, override it, and learn from it, built on privacy-preserving pipelines where identifiable K–12 data is processed locally.

Projects
AI-enhanced timeline for multimodal interaction analysis2023 – present

An interface that lays gaze, movement, speech, and AI-detected behaviors side by side on a shared timeline, so learning scientists can check what the models found against the video and correct it.

LLM evaluation

Knowing when to trust an LLM's reading

Methods for evaluating LLMs and vision-language models as qualitative analysts of multimodal classroom data: where they agree with expert coders, where they diverge, and how evaluation design changes the answer.

Projects
Evaluating LLMs as analysts of classroom interaction2025 – 2026

A method for measuring how well large language models perform multimodal analysis of embodied classroom learning, compared against expert human interpretation.

Key papers
Validity & trust

Evidence that travels with every result

Validity, reliability, reproducibility, and uncertainty evidence carried through each pipeline stage, so that a questionable interpretation can be traced to the step that produced it, and experts' trust in the analysis can be measured rather than assumed.

Earlier work:Conducting particles: gesture-driven music visualization (2008 – 2021)

Real-time 3D particle systems mapped to musical structure and controlled with Leap Motion and Kinect, evaluated with musicians and conductors, plus a rhythm game for hand-therapy exercises tested with users with motor challenges.

Timeline chart loaded.
Today
At this point...
0 items active

Research

Graduate Research Assistant

Vanderbilt University · NSF EngageAI Institute

Multimodal learning analytics: deep learning and LLMs that identify and interpret student behavior from gaze, movement, speech, and gesture; interactive tools for expert validation; lead on GEM-STEP multimodal AI; co-lead of the University of Haifa collaboration.

Multimodal AIHuman-in-the-loopLLMsLearning analytics

Graduate researcher, advised by Dr. Maria Andréia Formico Rodrigues

Universidade de Fortaleza (UNIFOR)

Real-time gesture-driven systems (Leap Motion, Kinect) that visualize musical structure; usability studies with musicians, conductors, and users with motor challenges.

HCIMusic computingComputer graphics

Undergraduate researcher (Scientific Initiation)

Universidade de Fortaleza (UNIFOR) · CNPq

3D computer graphics and music computing for data visualization.

Computer graphicsMusic computing

Publications

Making Interpretations Accountable: Propagating Validation Evidence Through a Human–AI Multimodal Learning Analytics Pipeline

LAK 2027
Validity & trustHuman-in-the-loopMultimodal AI

Expanding Access to Collaborative Computational Thinking through the ReCode Loop and Role-Differentiated AI Agents

Learning and Individual Differences
AI agentsLLMsEmbodied learning

A Human–AI Partnership Framework for Multimodal Analysis in Embodied Learning Environments

AIED 2026 – International Conference on Artificial Intelligence in Education
Human-in-the-loopMultimodal AI

Do Emotion Recognition Models Generalize to Classrooms? Robustness and Fairness Analysis

CV4Edu Workshop @ CVPR 2026
Computer visionAffective computing

When Modalities Disagree: Multimodal Sentiment Conflict as an Affective Signal in Collaborative Learning

ACII 2026 Workshops and Demos
Affective computingMultimodal AICollaborative learning

A Multimodal Study of Emotion Regulation in Student Dyads During Computational Model Building

ISLS 2026
Affective computingCollaborative learning

Multimodal Engagement Analytics for Collaborative Learning: A Human-in-the-Loop Approach

EARLI SIG: AI in Learning, Teaching and Education 2026
Human-in-the-loopCollaborative learning

A Human–AI Multimodal Analysis of Collaborative Knowledge Integration through Embodiment and Reflection

British Journal of Educational Technology
Human-in-the-loopMultimodal AICollaborative learning

SyncFlow: A Scalable Platform for Multimodal Learning Analytics

US-RSE 2025
Data infrastructureMultimodal AI

Education

Ph.D. in Computer Science

Vanderbilt University

Intelligent open-ended learning environments for embodied learning, using AI and multimodal learning analytics to study students' learning behaviors.

Advisor: Prof. Gautam Biswas
Multimodal AIEmbodied learning

M.Sc. in Applied Informatics

Universidade de Fortaleza (UNIFOR)
Advisor: Prof. Maria Andréia Formico Rodrigues
Thesis: Creation and Evaluation of a Gesture-Controlled Conducting System for the Visualization of Musical Structures
HCIVisualizationMusic computing

B.Sc. in Computer Science

Universidade de Fortaleza (UNIFOR)
Advisor: Prof. Maria Andréia Formico Rodrigues
Thesis: A Particle System Controlled by Musical Computing
Computer graphicsMusic computing

Industry

Head of Project Implementation (Software Development Team Lead)

Pará State Court of Appeals

Led the development team through analysis, fixes, and modernization of dozens of court systems, and through planning, testing, and deployment of new ones. Product Owner and Scrum Master across projects.

LeadershipSoftware engineering

Head of the Database Division

Pará State Court of Appeals

Led the DBA team administering Oracle, SQL Server, PostgreSQL, and MySQL databases behind 60+ judicial systems; performance and data-integrity work.

LeadershipData infrastructure

Training Analyst

Lanlink (Pará State Court of Appeals)

Created e-learning content and taught hands-on courses that moved public servants across the Amazon region from paper-based to digital judicial processes.

Teaching & training

Lead Engineer

IMMI Financial Solutions

Led mobile (Swift, Android) and web (Node.js, React, MongoDB) development for financial platforms; agile delivery and client requirements across Brazil and the U.S.

LeadershipSoftware engineering

Technology Consultant

LEAD – Dell Technologies

Architecture guidance for a hybrid mobile app for accessible distance learning; taught Dialogflow, React Native, and Redux to development teams.

Software engineeringTeaching & trainingAccessibility

Solutions Architect

FFIT – Innovation in Health and Education

Designed and led mobile and web applications (Node.js, Ruby on Rails, Angular, Ionic, React Native) with reusable components and microservices; Product Owner and technical liaison with clients.

Software engineering

Systems Analyst

Unimed Fortaleza

Business analyst and full-stack developer for the main health-services portal (Java, JasperReports, PL/SQL); Scrum Master for the internal agile team.

Software engineering

Development Intern

Caixa Econômica Federal

Built an ASP system delivering Severance Fund (FGTS) data to clients; server maintenance and backups.

Software engineering

Honors & grants

AI4Ed Summer Fellow (second selection)

AI Institutes Virtual Organization (AIVO) · funded by Google.org

Fellowship connecting researchers across the NSF National AI Research Institutes on AI in education.

Student Horizon Travel Grant

LIVE, Vanderbilt's AI and learning sciences incubator

Competitive award for presenting research at an international conference.

Red Hat Certified Developer in Cloud-native Applications

Red Hat

AI4Ed Summer Fellow

AI Institutes Virtual Organization (AIVO) · funded by Google.org

Fellowship connecting researchers across the NSF National AI Research Institutes on AI in education; presented at the AI4Ed summit.

Faculty Seed Grant ($10,000), co-author

University of Haifa

Collaborative research on LLMs and embodied computational thinking.

Peabody Global Research Grant ($7,500), co-author

Vanderbilt Peabody College

International collaboration with the University of Haifa on AI-driven embodied learning.

Certified PBB Practitioner

Caroli.org

2nd Place, Best Full Paper (Application track)

19th Symposium on Virtual and Augmented Reality (SVR 2017)

Certified ScrumMaster®

Scrum Alliance

Scientific Initiation Scholarship

CNPq (Brazil's National Council for Scientific and Technological Development)

Undergraduate research scholarship from Brazil's National Council for Scientific and Technological Development.

Service & mentoring

Co-organizer, CV4Edu workshop: Computer Vision × Education

CVPR 2026

Bridging vision, HCI, and learning sciences around domain robustness, construct-aligned inference, and privacy-aware multimodal evaluation.

Supervising an undergraduate researcher

LLMs as late-fusion mechanisms for classification and explainable AI.

Supervising a foreign master's student

LLM-based agents for computational thinking support.

Mentoring an undergraduate intern

Transcription and speaker diarization models for classroom audio.

Reviewer

Computers & Education · CVPR 2026 · LAK 2025, 2027 · AIED 2025 · EDM 2025 · ICCE 2025 · ISLS 2024, 2025 · EAAI 2025, 2027

Mentoring two undergraduate students

Hand tracking and gesture recognition with Leap Motion; educational game development in Unreal Engine.