DM 4144 Interface Design Advanced Studio

When talking about computer interfaces, we almost always think of sitting in front of a monitor, mouse, and keyboard and manipulating visual elements on the screen. This is an unnatural asymmetric interaction, with the human communicating using physical input, and the computer communicating using a visual response. This kind of interaction completely avoids sound, and greatly limits the tactile and visual interaction.

This course aims to break out of the WIMP (Windows Icons Mouse Pointer) world of interfaces and move into utilizing more of the realm of human senses. We start by looking at how humans interact, from speech to physical behavior, and then apply this thinking to creating engaging interfaces that use audio, video, 3D graphics, game controllers, sensors, and networking.

week 1 the six senses

• sight, hearing, taste, smell, feeling
• proprioception/kinesthesia (nope, no ESP)
• the human brain processes 160 GB/sec. in visual data alone

Reading: When will computer hardware match the human brain?

week 2 fundamentals of human interaction

• giving out vs. giving off
• 2 minute penalty for interruptions in concentrated work

week 3 tracking auditory events

• processing audio input in real-time
• dealing with noise
• realtime synthesis and effects

week 4 controlling video

• live camera input
• using movie files
• video effects and filters
• matrix operations

Assignment due: create a program that responds to auditory or visual stimulus

week 5 tracking visual events

• video tracking: grid, blob, color, movement
• environmental limitations for video tracking
• filtering out unwanted stimulus, targeting zones

Example: "The Manual Input Sessions:http://tmema.org/mis/ is a performance environment which tracks shapes made with the hands to generate audio events.

week 6 3D graphics

• core OpenGL? concepts
• displaying data in 3D
• Using 3D objects to represent interface ideas

Example: VIFE is a DJ/VJ environment where the performer manipulates 3D objects to control the music. The 3D objects in turn display their settings visually.

RBL soft

week 7 controllers

• tablets, joysticks, gamepads, mice, keyboards
• force feedback devices
• HID API (Human Interface Device)

Example: StickMusic

Assignment due: create a program which a user can control auditory and visual output in real-time using controllers, auditory input, and/or visual input.

week 8 sensors

• arduino with Pd firmware
• processing sensor data
• dealing with noise

Example: Audio Visual Aero Device is a immersive environment controlled by the position and movement of the user.

week 9 output to the physical world

• arduino with Pd firmware
• LEDs?
• relays to control bigger things
• motors, lights, servos, etc.

Example: YoYo Berimbau is a musical robot which responds sound, especially rhythmic events.

week 10 Mapping

• getting beyond direct mapping
• non-obvious reactions make for a more playful interface
• initial interaction should draw the user in
• learn from existing interfaces: musical instruments

Example: Bob Huott's Ski is custom built musical instrument with a novel interface.

Assignment due: analyze the mapping of an existing program, propose a more engaging mapping, and implement it. You can use your previous homeworks, projects, or some of the examples provided.

week 11 networking

• TCP, UDP, OSC, Service Discovery (DNS-SD)
• interacting across distances
• collaborative enviroments

Example: netpd allows electronic musicians to jam with each other across any network.

week 12 user testing

• do your ideas match up with your users' experience?
• work out final sticking points
• trial runs

week 13 final presentations

• each student will present their final project to the class for commentary and criticism from the class.

Readings

These are the core readings for the class, which are all required.

• Pd Manual http://crca.ucsd.edu/~msp/Pd_documentation/
• Snow's Two Cultures Revisited: Perspective from Design in Human-Computer Interfaces, by Bill Buxton http://www.billbuxton.com/2cultures.html
• Composing Interactive Music, by Todd Winkler ( MIT Press, ISBN 0-262-23193-x)
• Physical Computing http://itp.nyu.edu/~dbo3/physical/physical.html
• Computer Vision for Artists and Designers http://www.flong.com/writings/texts/essay_cvad.html

Optional Readings

This is a collection of readings on related topics for anything that you migth want to delve deeper into.

Interaction

• Buxton, W. (1990). The Natural Language of Interaction: A Perspective on Non-Verbal Dialogues. http://www.billbuxton.com/natural.html
• Buxton, W. (1990). A Three-State Model of Graphical Input. http://www.billbuxton.com/3state.html

Audio

Video

OpenGL?

Networking