CVPR 2026 Project Website

Breaking Sensor–Text Alignment

A cross-modal embedding poisoning attack against contrastive multimodal human activity recognition. The attack perturbs the text-side embedding at inference time, lowering correct sensor–text cosine similarity and causing semantic drift.

Malithi Mithsara W. Kankanamge · Ning Yang · Ahmed Imteaj · Abdur R. Shahid

4datasets
64Dshared embedding space
0.5similarity threshold
PGDbounded perturbation
Traditional HAR and contrastive HAR overview
Traditional HAR predicts activity labels directly. Contrastive HAR aligns sensor embeddings and text embeddings in a shared space.
Overview

From activity classification to semantic alignment

Core idea

Contrastive HAR maps wearable sensor windows and natural-language activity descriptions into the same representation space. This enables a model to match a sensor pattern with a semantic description such as “a person is climbing stairs quickly.”

The security risk is that a small text-embedding perturbation can move the semantic prototype away from the correct sensor embedding, making the model choose an incorrect interpretation.

Cross-modal embedding poisoning attack overview
Attack overview: malicious inference-time logic injects a bounded perturbation into the text embedding and disrupts contrastive alignment.
Method

Search-based embedding poisoning

The attack keeps the sensor input fixed and iteratively updates the text-side perturbation with projected gradient ascent. The goal is to increase contrastive loss and reduce correct sensor–text similarity.

Dual encoder architecture
Dual encoder: sensor MLP and DistilBERT text encoder map inputs to a 64-dimensional shared embedding space.

Attack pipeline

01
Sensor window

Accelerometer, gyroscope, or biosignal segment.

→
02
Text prototype

Semantic description for the activity or state.

→
03
Perturb

Add bounded δ to the text embedding.

  • Optimization: projected gradient ascent.
  • Budget: L∞ bound with ε sweep from −10 to 10.
  • Representative setting: α = 0.005, K = 1000.
Datasets

Motion and physiological benchmarks

WISDM

Smartphone accelerometer data for walking, jogging, stairs, sitting, and standing.

MotionSense

Smartphone accelerometer and gyroscope signals for common daily activities.

HHAR

Smartphone and smartwatch signals used to test device heterogeneity.

WESAD

Multimodal physiological signals, including ECG, EDA, respiration, temperature, and BVP.

Training setup

Approximately 60,000 sensor–text pairs trained for 30 epochs with batch size 4 and learning rate 1e−4.

Metric

Cosine similarity measures semantic closeness. Values above 0.5 are treated as matched pairs.

Interactive Demo

Move ε and watch alignment degrade

Select a dataset, then adjust the perturbation budget. The demo summarizes the paper’s reported attack trends: successful attacks reduce cosine similarity and increase contrastive loss.

WISDM

0.48estimated adversarial similarity
0.52estimated contrastive loss
Successattack status
Results

Key findings from the paper

Before attack: mean cosine similarity

DatasetPositiveNegative
WISDM0.57150.5805
MotionSense0.60090.6315
HHAR0.57690.5896
WESAD0.79560.8099

Observed vulnerability

MotionSense, HHAR, and WESAD are sensitive to very small positive perturbations, often around ε = 1. WISDM varies by activity: Downstairs is the most vulnerable, while Standing and Walking require larger perturbations.

Attack execution time is around 10 minutes for WISDM, 5 minutes for MotionSense and HHAR, and 3 minutes for WESAD.

Attack behavior grid across WISDM MotionSense HHAR
Activity-level attack behavior across WISDM, MotionSense, and HHAR.
PGD steps and perturbation norms across datasets
PGD steps and L∞/L2 norm behavior across datasets.