Abstract
Large language models are increasingly being used to reason over sensor data from the physical world and human physiology. However, multimodal sensing remains challenging because heterogeneous sensors can provide complementary but sometimes noisy, missing, or unreliable information. This paper proposes ConSensus, a training-free multi-agent collaboration framework for multimodal sensing. Instead of giving all sensor features to a single LLM, ConSensus assigns each sensor modality to a specialized modality-aware agent, allowing each agent to generate an independent prediction and rationale. These modality-level interpretations are then aggregated using a hybrid fusion strategy that combines semantic fusion, statistical fusion, and a final hybrid coordinator. The semantic fusion agent reasons over the modality-agent explanations using cross-modal evidence and domain knowledge, while the statistical fusion agent anchors its reasoning to the majority-voted prediction. The hybrid fusion agent then arbitrates between these two reasoning paths to produce the final decision. The authors evaluate ConSensus on five multimodal sensing benchmarks, including affective state recognition, sleep stage classification, kitchen activity recognition, gym exercise recognition, and daily activity recognition. Results show that ConSensus improves average accuracy over the single-agent baseline and achieves comparable performance to multi-agent debate methods while substantially reducing fusion token cost. The study highlights the value of modality-specific reasoning and structured hybrid fusion for efficient and robust LLM-based multimodal sensing.
Pegah Khorasani
Graduate Research Associate
As a doctoral candidate at Arizona State University (ASU), I am currently conducting research under the guidance of Professor Hassan Ghasemzadeh at the Embedded Machine Intelligence Lab (EMIL). My research interests encompass a range of topics, including machine learning, health monitoring system development, and mobile health.