Mixed-Ability Human-Swarm Interaction

Weakly bounded bodies do not become embodied by becoming vague. They need grouping cues, common motion, response, and recurrent structure. A body can lose surface closure if it keeps a stable carrier such as a density core, rhythm, motion signature, or contingent response pattern.

This is why plasmatic design treats permeability as a state machine rather than a shader. The body gathers, disperses, merges, splits, protects, reveals, hides, and repairs as meaningful state changes.

Avatar research often explains embodiment through transformed self-representation. The Proteus effect is useful here because virtual bodies can influence conduct and self-perception: Stanford's VHIL summary describes studies where avatar height and attractiveness affected behavior in online and later face-to-face contexts (Yee, Bailenson, and Ducheneaut, Proteus effect).

Swarm bodies need a movement-based extension of that idea. Jeong, Kim, Xu, and Miller call this Protean kinematics: an expansion from avatar appearance toward movement-based effects and the blend of physical inputs with virtual outputs. That is exactly the design space of a plasmatic swarm body (Jeong et al., Protean Kinematics).

In this page, the avatar is not a fixed body with a different appearance. It is a dynamic body whose movement grammar carries agency. The participant's relation to the body comes from how it moves, responds, waits, resists, and returns.

A swarm body is not just "emergence." It is authored emergence. Flocking, particles, crowd solvers, and morphogenetic systems become usable when they are shaped by fields, guide curves, zones, triggers, rhythms, constraints, repair rules, and levels of detail. Mixed-ability HSI needs the same middle layer: enough local autonomy to feel alive, enough authoring structure to stay legible and negotiable (Computational morphogenesis; Reynolds; SideFX crowd docs).

Access is also the right to reshape what the interface thinks a body is. Crip Sensorama makes that especially clear for XR: default headset, controller, hand-gesture, vision, and dexterity assumptions quietly define an imagined user before accessibility is added. Mixed-ability HSI has to make that imagined user visible before it can design a fair swarm body (Jain, Crip Sensorama gesture process; Jain and Bayerlein, Hand to Mouth, 2026).

Mixed-ability collaboration cannot be reduced to one accessible controller or one adaptive interface. Ability-based design argues that systems should adapt to what people can and want to do, while interdependence work emphasizes that access is often produced between people, technologies, environments, and care relations (Wobbrock et al.; Bennett et al.).

Mixed-ability therefore includes disability, chronic illness, pain and fatigue variation, neurodivergence, sensory tolerance, mobility variation, communication preference, temporary impairment, and different willingness to disclose. The design question is not whether each person has the same channel. It is whether the group can understand, contest, rebalance, and trust how agency is distributed.

Disability and remote-VR research also warn against treating presence as a single default mode. Disclosure, sensory load, avatar presentation, recruitment, travel, fatigue, pacing, and asynchronous contribution can all change whether participation is actually available. W3C's XR Accessibility User Requirements similarly emphasize multimodal support, input/output synchronization, customization, motion-agnostic interaction, captions, alternatives, and time limits. A virtual-first HSI study should therefore support non-HMD modes from the beginning: desktop, browser, projection, tablet, captions, audio description, reduced motion, seated use, asynchronous annotation, and facilitator-mediated input (Mottelson et al.; Zhang et al.; Gualano et al.; W3C XR Accessibility User Requirements).

A mixed-ability group does not share one sensor envelope. Some participants may rely on audio, text, haptics, peripheral vision, facilitator description, memory, breath, assistive devices, or asynchronous annotation. The swarm body should make these differences negotiable without making them compulsory disclosures. Jain's umwelt work is useful here because it frames collaboration as overlapping sensory worlds and temporalities rather than as one average user model (Jain, Umwelten).

Glowacki's Isness work gives this section its clearest bridge between abstract body form, social connectedness, and physics-inspired aesthetics. Isness-D represents participants as luminous energetic essences with diffuse boundaries that can coalesce, and the 2022 study reports strong self-report outcomes around ego attenuation, communitas, and connectedness. The 2024 aesthetic paper then makes the design vocabulary more explicit: low structural specificity and low symbolic rigidity allow a representation to remain bodily and meaningful without being locked into ordinary object, body, or identity categories (Glowacki et al.; Glowacki).

This matters for the physics argument because the abstract body is not only a soft avatar. It inherits a molecular-simulation sensibility from Glowacki's earlier installation work: bodies can be modeled as fields, forces, densities, flows, coalescences, and perturbations. In public HSI language, that separates two claims that often get blurred. Physics can be a generative aesthetic substrate for shared embodiment, but it can also become a concrete implementation burden when sensors, simulations, particles, and hardware must keep the experience stable across time. The 2025 esencia paper is especially useful because it treats that lineage as an active reinterpretation problem across depth sensing, particle simulation, GPU execution, and changing software stacks (Mitchell et al.; Toledo Castro, Protopopov, and Glowacki; Essentia Foundation interview).

That same lineage is now being translated into clinical-adjacent work through aNUma's Clear Light program and the Tiny Blue Dot-funded NUMADELIC project. The current public evidence should be read carefully: a published observational cohort of 15 people facing life-threatening illness reports feasibility and improvements on self-report measures, while also stressing the limits of having no randomization or control group. The stronger efficacy question is routed through a public OSF registration for a randomized controlled trial, and IRL/CiTIUS describe the wider 2024-2028 project as combining lab studies, design iteration, physiological measurement, and an RCT. Numadelic Labs Collective adds a current nonprofit translation surface for the same lineage: its public materials describe AI-augmented group therapy in VR, with group, embodied, and immersive elements, energetic essences, arm-guided energy between bodies, and a science page that lists the peer-reviewed numadelic research base. That site is useful as organizational and design-context evidence, not as independent efficacy proof (Kettner et al.; OSF RCT registration; CiTIUS NUMADELIC project; aNUma Clear Light; Numadelic Labs; Numadelic Labs science page).

John Desnoyers-Stewart's particle-body line is the clearest existing precedent for this middle layer. Transcending Projection and Transcending the Virtual Mirror Stage show how point-cloud and particle bodies can become ownable through mirrors and shared public space. Body RemiXer then moves from ownership into intercorporeality through particle auras, touch exchange, and abstract body swapping. Star-Stuff frames participants as constellation bodies in a shared cosmic encounter, while ETC turns aura avatars into a telepresent social-touch system using pseudohaptics and cross-modal cues (Desnoyers-Stewart; Desnoyers-Stewart, Smith, and Riecke; Desnoyers-Stewart et al. 2020; Desnoyers-Stewart 2022; Desnoyers-Stewart et al. 2023).

Bernal and Maes add a second useful precedent with Emotional Beasts. Their abstract avatars used color, glow, aura-like changes, fur, and particle expression to make internal state visible without returning to a realistic body. The later Bernal/OpenBCI line matters because it connects emotionally expressive avatars to sensor-rich HMD work: OpenBCI describes Emotional Beasts and PhysioHMD as part of Bernal's effort to integrate biosensors into head-mounted displays, while Galea carried that direction into a mixed-reality platform with EEG, EOG, EMG, EDA, and PPG sensing (Bernal and Maes, Emotional Beasts; OpenBCI Bernal interview, 2020; Bernal, Developing Galea, 2021; MIT PhysioHMD; OpenBCI Galea release, 2020).

The Galea development account makes that access lineage more concrete: Bernal describes working with Christian Bayerlein to connect EMG to muscles across Christian's body and use those signals as additional switches or actuators for controlling other devices. Christian's later XR/HCI work with Puneet Jain, including the Crip Sensorama gesture process, the Christian's Coffee installation, and the Hand to Mouth paper, reframes access around mouth gestures, crip-hacking, preference-tuned mappings, and disability-led interaction rather than around retrofitting hand-centric interfaces. For mixed-ability HSI, the lesson is not that more sensors are automatically better (Bernal, Developing Galea, 2021; Jain, Crip Sensorama gesture process; Jain, Crip Sensorama: Christian's Coffee; Jain and Bayerlein, Hand to Mouth, 2026).

"Embodied" or "controllerless" interaction is not automatically accessible. Mid-air gestures, breath control, gaze dwell, and mouth gestures can all create fatigue, exposure, social awkwardness, or calibration burden. Access depends on whether the channel is voluntary, sustainable, replaceable, private when needed, and allowed to fail safely (Jain, VR navigation study).

The most useful explicit source here is Kim, Drew, Domova, and Follmer's user-defined swarm-control study. It shows that people do not simply choose "gesture" as a generic input. Their interaction vocabulary changes with swarm size, unit proximity, tabletop versus mobile context, one-hand versus two-hand use, touch, verbal commands, and whether the interface can infer the intended state of the swarm (Kim et al.).

That source should anchor the page, but the map has to be broader. HSI literature already distinguishes behavior selection, parameter setting, environmental influence, leader influence, and sub-swarm selection; Kolling's foraging-swarm study contrasts intermittent selection with environmental beacon control; gesture work distinguishes free-form and shape-constrained control, deictic, representational, manipulation, pose, motion, and hybrid gestures; mixed-granularity AR combines environment-oriented and unit-oriented control (Kolling et al., HSI survey; Kolling et al., two HSI types; Alonso-Mora et al., gesture taxonomy; Patel, Xu, and Pinciroli, mixed-granularity HSI).

Mixed-ability HSI therefore needs a stack, not an input menu. A mouth gesture, EMG switch, breath signal, hand pose, touchscreen text entry, voice command, or environment marker can all be valid, but none is generic. The same family of signals can differ by anatomy, fatigue, equipment, history, comfort, cultural meaning, and preference. Each becomes a different social proposal when it binds to a whole swarm, a sub-swarm, one particle, a field, a rule, a color state, a safety veto, or a shared goal. MOSAIX is important because it moves this question into public multi-human-swarm interaction: people contribute ideas through tangible tiles, the swarm clusters them semantically, and the installation becomes a social sorting medium rather than a single-operator controller (Alhafnawi et al., MOSAIX).

Existing HSI asks how a human can influence a swarm. Multi-human HSI asks how several people can influence a swarm. Mixed-ability HSI asks how several differently situated people can share, inspect, contest, and repair influence through a swarm body. MOSAIX is a useful contrast case because it makes public multi-human-swarm interaction concrete, while this page asks for shared embodiment, access negotiation, visibility choice, authorship, and repair.

The working input families are direct manipulation, spatial gesture, speech and text, physiological and muscular sensing, device/controller pose, environmental annotation, and semantic contribution. None of those families determines the design by itself. The decisive question is whether the channel changes appearance, selection, a rule, an attractor, a form seed, a game role, or a safety and repair affordance.

Some mappings also need to adapt over time. Dwell thresholds, gesture confidence, pause duration, sensitivity, and repair prompts may need to change across repeated use. Jain's adaptive gaze keyboard work is useful here because it treats timing as user-specific and dynamically adjustable rather than as one fixed setting. In a mixed-ability swarm body, adaptive rules should remain inspectable, reversible, and overridable, with a clear distinction between participant-controlled adaptation and system-controlled adaptation (Jain, adaptive virtual keyboard; Jeevithashree et al., adaptive virtual keyboard).

A useful adjacent vocabulary comes from crowd animation and VFX. Those systems often separate the behavior layer from the art-direction layer and the rig or production layer: seeking, avoiding, flocking, and following are tuned alongside fields, sketches, guide curves, zones, triggers, states, clips, skeletons, foot-planting, levels of detail, and caches. For mixed-ability HSI, the point is not that animated crowds are the same as deployed swarms. The point is that collaborators may approach the swarm through motion design, game AI, avatar rigs, simulation fields, or physical systems, so the stack has to name what a signal controls: behavior, field, rig state, appearance, goal, role, or safety state (Reynolds, steering behaviors; Patil et al.; Colas et al.; Prazak et al.; SideFX crowd docs).

This part is implementation-facing. Its purpose is to show how the mapping stack could be made inspectable in software.

The same signal can then be routed into several paradigms without pretending they are the same. A breath estimate might soften boundary opacity in Optics, bias local attraction in Matter, trigger a Manifold pause command, or become only a private self-monitoring cue. A mouth gesture might select a sub-swarm, change a role, or veto a game phase. A controller pose might seed an SDF form. A room surface, beacon, AruCo marker, semantic cluster, or obstacle might act as an environmental stimulus. A VR swarm-hand technique such as Swarm Manipulation shows how hand tracking can control selection, rotation, resizing, and particle distribution, while interpreter-based formation work shows how a wearable gesture can become a high-level command that is translated into low-level formation dynamics (Li et al., Swarm Manipulation; Suresh and Martinez, formation interpreters).

Openness is not a publishing preference here; it is part of the method. If the stack is closed, a participant cannot tell whether fatigue came from the input channel, the binding, the dynamic rule, the renderer, latency, calibration drift, or another collaborator's hidden control. An explicit open stack makes it possible to swap sensors and platforms, compare versions across media, log what happened, preserve consent and authorship, and carry only the surviving dynamics into later translation. The public Morphospace page therefore serves as a status boundary: current contracts and validation proofs are separated from planned HSI authoring, future robotics translation, and farther biotech speculation.

Physiological channels need an extra guardrail. EEG, EMG, EOG, EDA, PPG, ECG, breath, and related signals must be opt-in, inspectable, replaceable, and non-diagnostic. They should not be treated as truth about emotion, pain, intention, fatigue, or consent. A participant should be able to route a biosignal into public expression, private self-monitoring, or nowhere at all, and the system should document interpretation, privacy, and retention choices as part of the mapping (Chiossi et al., PhysioCHI: Towards Best Practices for Integrating Physiological Signals in HCI).

Access practices

existing devices, gestures, rhythms, rest strategies, communication habits, assisted routines, and off-limits signals

Inputs

gesture, touch, switches, controllers, voice, text, breath, biosignals, EMG, gaze, pose, timing, rule edits, and environmental markers

Swarm variables

density, rhythm, coupling, boundary softness, luminosity, color, speed, local attraction, SDF seed shape, and environment response

Access variables

visibility, disclosure, pacing, rest, turn-taking, veto, asynchronous contribution

Repair variables

undo, pause, rebalance, remap, explain, replay, consent change, authorship note

Calibration burden

setup time, posture, camera angle, occlusion, false positives, rejected gestures, assistance, and recalibration

Outcomes should not stop at task score. The important measures include shared agency, legibility, fatigue, trust, role satisfaction, repair success, disclosure comfort, authorship, and whether participants can describe how the swarm body represented their contribution.

Logs should be treated as situated records, not as neutral truth. A system may overcount fast signals, undercount quiet forms of participation, repeat uncertain classifications, or miss invisible labor such as waiting, masking fatigue, or deciding not to act. The study should pair system logs with participant interpretation and allow participants to annotate, contest, or delete records (Jain, Epistemological Intervention).

Authorship should be planned before data collection. If participants invent mappings, name swarm states, define repair gestures, reject sensor categories, or contribute access vocabulary, the study should specify how those contributions are credited, anonymized, withheld, licensed, or carried into later prototypes. Authorship is part of the mapping contract, not only a post-study acknowledgement.

A visual swarm body can explore mappings cheaply and reversibly. That reversibility is not a weakness; it is what makes access negotiation safe enough to study. Physical swarms add material consequence: contact, obstruction, impact, actuator limits, localization error, maintenance, and partial failure.

Contemporary swarm robotics still faces substantial deployment barriers, so the physical horizon should be treated as a stress test for selected mappings rather than as the default endpoint of the project (Kegeleirs and Birattari).

Morphogenetic and bioelectric work is useful here as a vocabulary for many-part coherence: fields, thresholds, state memory, perturbation, homeostasis, repair, and return. This does not mean the swarm body is biological, therapeutic, diagnostic, or prosthetic.

DiffeoMorph belongs in this horizon as a far-future computational-morphogenesis reference: it suggests learned many-agent target-forming dynamics, not a current access method and not evidence for bioelectric physiology (Pahng et al., DiffeoMorph; hormoz-lab/diffeomorph).

RoomShift makes this shift concrete because it uses robot assistants to move real furniture for room-scale haptics, turning the design into questions of lift height, object weight, under-table clearance, optical tracking, path planning, collision avoidance, and whether the object can safely support a person's body once placed. Roombots similarly frames adaptive furniture as an active modular robotics problem, not only a shape-design problem (Suzuki et al., RoomShift; Sprowitz et al., Roombots).

Swarm robotics is no longer only a metaphor or an animation vocabulary, but its real-world use is still limited. The current field has strong laboratory demonstrations, increasingly capable sensors and computation, better simulators, mixed-reality interfaces, ROS-based integration, hardware-in-the-loop testing, and growing interest in applied domains. It still struggles with platform limits, simplified simulations, robot-to-robot communication assumptions, localization, security, maintenance, and the deployment gap between one physical platform and another (Kegeleirs and Birattari; Zheng, Jarecki, and Lee).

The related assembly field is not usually named "human swarm assembly robotics" as a single stable discipline. It overlaps with collective robotic construction, programmable self-assembly, modular self-reconfigurable robotics, swarm user interfaces, and tangible robotic swarms. The shared idea is that a person should not usually pilot every unit. The human should specify goals, constraints, fields, morphologies, roles, or safety conditions, while the swarm or modular body resolves lower-level coordination (Petersen et al., collective robotic construction review; Werfel, Petersen, and Nagpal, TERMES; Rubenstein, Cornejo, and Nagpal, Kilobot self-assembly).

For mixed-ability HSI, assembly is important because it changes what "control" means. A participant might guide a desired furniture configuration, ask a modular body to extend reach, mark a construction zone, set an attractor field, request repair, or veto an unsafe intermediate state. Recent user-guided modular-robot work makes this especially concrete: the user can manipulate a morphology-matched interface, but an optimization layer blocks commands that would violate torque, collision, balance, or environmental constraints (Bolotnikova et al., user-guided modular robots).

This is also why virtual simulations have limited translation capacity. They are useful for exploring mappings, roles, affective tone, and the social feel of shared control, but they can hide the hardest parts of real deployment. Kegeleirs and Birattari describe how swarm robotics still faces platform limitations, abstraction of real tasks such as object manipulation, a sim-to-real reality gap, and a wider deployment gap where a controller that works in one physical platform may fail on another. For mixed-ability HSI, that warning should become a design principle: if the goal is connectedness or game-based collaboration, visual physics can be tuned for legibility and felt agency; if the goal is physical accessibility, furniture, mobility, or assistive room support, physical envelopes, load cases, safety, localization, maintenance, and failure recovery have to enter the project early (Kegeleirs and Birattari).

The bridge is therefore not a straight export from visual design into robots. It is a staged constraint test. The swarm-body phase asks which dynamic cues make shared agency readable and inhabitable. The material phase asks which of those cues still work when the swarm must move mass through a room, stay safe around people, tolerate sensor uncertainty, and remain useful after errors.