Images & videos minimised for storage, originals shown at live presentation
VISUAL
HARMONICS
Visual Harmonics

Physical Computing as a Means of Making Real-Time Visuals Accessible to Creative Practitioners

Carissa Rosita Jayanto  ·  BADC6E

Table of Contents

01

Context

Background & Rationale

02

Precedents

Influences & References

03

Design Statement

04

Semester 1

Experiments & Interviews

05

Semester 2

Prototypes & Testing

06

Next Steps

Future Direction

CONTEXT
Context

Background & Rationale

Musicians and performers engage with sound through deeply embodied, physical gestures, however the tools available for live visual performance (such as TouchDesigner or Resolume) often rely on interfaces that disconnect the performer from their craft.

This project investigates how physical computing interfaces can bridge that gap, with a particular focus on traditional Malay instruments like Kompang, Rebana, and Gedombak for exploring the relationship between tangible, rhythmic input and generative visual output.

Influences & References
Affordances
Embodied Knowledge
Audiovisual Art
Contemporary · Examples
Sound Art
Live Visual Performance
Arduino · p5.js
Serial Communication

Visual Harmonics

Physical Computing
+ Generative Visuals

Theory &
Design

Norman · Ishii
Maeda

Artistic
Practice

Interviews
Observations

Technology
& Making

O'Sullivan & Igoe
Processing / p5.js
DESIGN
STATEMENT
Design Statement

Design Statement

Traditional performers often have vivid visual concepts for their live performances but face significant barriers when attempting to create digital visuals. Current tools demand technical expertise that feels disconnected from embodied performance practice, forcing practitioners to rely on external technicians rather than controlling their own visual expression.

Visual Harmonics looks at what happens when performers can shape visuals the way they already shape sound. The project develops sensor-based prototypes that take cues from traditional instruments like Kompang and Rebana, so performers can control digital visuals through striking, texture and gesture. Instead of asking practitioners to pick up complex software, the work brings technology closer to how they already move and play.

Semester 1 RECAP

Experiments & Explorations

EXP 01–03

Sensor Explorations

Initial experiments with piezo sensors, force-sensitive resistors, and capacitive touch to detect percussion gestures. Mapping analog signals to digital triggers.

Sensor experiments
EXP 04–06

Visual System Development

Exploring generative visual systems across p5.js, Blender, and TouchDesigner, developing particle systems, waveform visualisations, and reactive geometries driven by sensor data.

Visual development
EXP 07–09

Hardware Integration

Connecting Arduino-based circuits to p5.js via serial communication. Joystick controllers and piezo-embedded surfaces as input devices for visual manipulation.

Hardware integration
Prototype 01
01
Prototype 01
Prototype 01

Haptic bars, gambang reference, texture-based navigation

Prototype 02
02
Prototype 02
Prototype 02

Hexagonal grid, elastic fabric surface, modular layer system

Prototype 03
03
Prototype 03
Prototype 03

Handheld joystick, continuous modulation, built for portability

Semester 2, Term 1

User Testing

User Testing
01

Prototype Testing Sessions

Testing with practitioners and also general participants to evaluate intuitiveness and responsiveness of each prototype.

02

SUS Questionnaire

System Usability Scale evaluation across all three prototypes to quantify user experience.

03

Findings

Key insights on sensor sensitivity, visual mapping clarity, and areas for hardware refinement.

NEXT
STEPS
Next Steps

Future Direction

01

Refine sensor calibration and improve visual responsiveness based on practitioner feedback, focusing on reducing latency and making gestural mappings more expressive.

02

Test how the tool could be integrated into an actual performance setting, exploring contextual applications such as building a diorama or installation that uses the system as part of a larger environment.

03

Produce a practical manual for the tools, documenting how to set up, calibrate, and use each prototype so other practitioners can pick it up independently.

THANKIES
YALL
Thank You

Visual Harmonics

Carissa Rosita Jayanto  ·  BADC6E