Digital Musical Instruments

Digital Musical Instruments

A nice compilation of  new interfaces for real-time electronic music performance.

Bill Buxton Shows Us His Favorite Tech

Bill Buxton Shows Us His Favorite Tech | Show Us Your Tech | Channel 9.

Really nice interview with Bill Buxton, during CHI2011 conference, where he exhibited his impressive collection of gadgets that range from watches, to all sorts of joysticks and hand held devices.

By the way, Microsoft Research and Bill Buxton are compiling this impressive and valuable collection on a website that is a mandatory study reference for anyone studying/working on HCI, Interfaces and Interaction. The site includes detailed photos, descriptions and comments by Bill himself and videos of the working user interfaces and how some of the devices operate are being uploaded into the website.

So, before you dare to present any “novel” interface/interaction design/HCI solution to the world, just follow Bill’s advice and check first if someone else already proposed that same approach some years ago (and save yourself from some embarrassment ).

Kinect Audio: it’s more than just a 3D camera

Kinect Audio: Preparedness Pays Off – Microsoft Research.

Really nice article that let us know the sound and speech features included in the Kinect (which so far have been under the radar for most FOSS driver hackers/developers/users), and which will definitely pave the way for truly multimodal NUIs.

Laser Harp Project

This Laser Harp 2010 project is the inspiration for a project at FEUP I’m currently co-advising (more news on that soon).

Some pointers related to this:

• Pixart/Wiimote sensor library for Arduino
• Pixart Sensor and Arduino

What the Kinect sensor actually does…

What the Kinect sensor actually does…

Interesting post on how the MS Kinect may actually work.

Some (unofficial and still to be confirmed) specs summarized from the post linked above (and comments):

• the Kinect appears to be a 640×480 30fps video camera that knows the *depth* of every single pixel in the frame. It does this by projecting a pattern of dots with a near infrared laser over the scene and using a detector that establishes the parallax shift of the dot pattern for each pixel in the detector (parallax seems to be more robust than intensity – some sources said that materials ( hair in particular ) caused large fluctuations in intensity, so it doesn’t seem like it would be a useful channel to probe for depth data).
• The depth buffer is only 320×480 (unconfirmed). It seems that the hardware will happily give a 640×480 version (this is Xbox360 API memory, so upscalingmay actually occur on the XBox360) but the hardware itself only gets enough data to fill 320×480.
• Alongside this there is a regular RGB video camera that detects a standard video frame. This RGBZ (or ‘D’) data is then packaged up and sent to the host over USB.
• It seems that the Kinect framerate (for RGB image and depth buffer) is 30Hz.
• The Kinect does not identify shapes within the field of view and does not attempt to map skeletal outlines of those shapes recognised. For that, you would need to take each one of the 640×480 frames and copy them into a framebuffer so they can be processed by a vision library like OpenCV. Typical operations would be to threshold the depth image to get the “closest” pixels – then perform a blob analysis ROI to group these pixels into identifiable features and then track those blobs over their lifetime.
• The Kinect uses a pattern of laser dots to detect depth, as can be seen in this video (and another one, and another one, and another one ) and in these images. It seems to exist a 3×3 checker board effect in that dot pattern (no clue why yet… any suggestions?).

So, processing all this data seems to be quite heavy (mainly if you try to do it in an embedded board like the guy from the post above). Using a full-fledged PC/Mac using openCV and/or OpenCL in a multicore machine will get you the required juice for advanced image processing.

Finally, some quite interesting resources for Kinect related stuff:

• Kinect Hacks
• Freenect – Kinect Projects
• Open Kinect drivers
• OpenNI Kinect Drivers
• PrimeSense NITE Middleware
• Kinect Teardown

Huge multitouch wall project

Reality Touch Theatre < CIT < University of Groningen.

Really impressive project at the University of Groningen in the Netherlands. Some videos here and  here.

“This is the curved (!) screen in our reality center of the University of Groningen. We just finished building our own touch detection for it.

We used six Optitrack v120 slim camera’s which have a good sensitivity for infrared light. We used 16 cheap infrared emitters (the kind used for security systems) with a total of 1000 LED’s.
The touch detection software runs on three old computers each with two camera’s connected. One extra computer combines the output from the detection computers and send event data to our main visualization system.
This way we have (even using the old computers) enough processing power to be able to run the detection software at 60Hz and with a latency between 30 ms and 50 ms. It can detect without any problem 100 different touches at any time (more is possible, but it becomes slower)

We used a modified version of Community Core Vision (CCV) 1.4 (nuigroup.com) (modified so it can do two camera’s on one computer).
The communication protocol is preferable TUIO (tuio.org) and we did install Multi-touch Vista (multitouchvista.codeplex.com), which translates TUIO events to WM_TOUCH events for windows 7.
The demos you see in the video are from Multitouch for Java(tm) MT4J (mt4j.org/mediawiki/index.php/Main_Page). The part where the wizards are throwing fireballs at each other is using msafluid (project home is at msavisuals.com/msafluid ).

The curved screen itself is consist of a 3 mm dark acrylic layer, coated with a diffuser on the front. Illumination is from behind using six full hd Barco projectors.

The cameras and the ir-leds are also located behind the screen.“

Pat Metheny plays with Robots

YouTube – Pat Metheny – The Orchestrion EPK.

Btw, here is an interesting interview with Eric Singer (the guy behind probably most of those robots):

http://www.youtube.com/watch?feature=player_embedded&v=ehSJVqyr8FY

And a movie about the work of Trimpin:

http://www.trimpinmovie.com/

And last, but not the least, the site of Ajay Kapur (a good friend, who will soon pay us a visit here in Porto! ):

http://www.ajaykapur.com/

All big names in music robots!

MS Kinect, OpenNI and OSC – tohm judson

My PhD student André Baltazar just pointed me to these quite well compiled instructions on how to make MS Kinect work in OSX (and Ubuntu and Windows):

OpenNI to Max/MSP via OSC – tohm judson.

I will now install everything in my own MacBookPro laptop and then go out and buy another Kinect for the lab

Some neat ideas to do with this:

• take the OSC messages produced from the skeleton code and input them into some algorithmic music system (in Chuck, Pd, Max/MSP, SuperCollider, etc), that generates sound/music based on body analysis;
• Take the RGB body image captured by the Kinect and over-impose virtual elements that interact with the computed skeleton (augmented reality scenario);
• Show this to our 3D modeling and animation colleagues and try to send the detected body movements into a 3D scene with a 3D modeled character (and in this case, we should try to adapt the skeletonization code to also provide depth coordinates);
• …

 

Human Computer Interfaces (HCI) and Designing Interactive Systems (DIS): some pointers

I’ve recently been quite interested in HCI and Interactive Design topics, and here are some pointers I’ve gathered in order to get a bit more into this field. Suggestions and additional pointers are most welcome!

iTune University courses related to HCI and Interactive Systems:

• Stanford University Human-Computer Interaction Seminar (Winter 2010)
• RWTH AACHEN – Designing Interactive Systems I (WS 2010/2011)
• RWTH AACHEN – Designing Interactive Systems II (Summer 2010)
• RWTH AACHEN - Current Topics in Media Computing and Human Computer Interaction (Summer 2010)

Some R&D Groups working in this field (quite incomplete list!):

• Stanford HCI Group
• RWTH AACHEN Media Computing Group (with HCI and DIS related courses pages here, here and here)
• Carnegie Mellon Human Interaction Institute
• University of Madeira – Math and Engineering Department (they have a new Masters Program on HCI, in close collaboration with Carnegie Mellon Univ.)
• School of Arts of the Portuguese Catholic University/CITAR (where I teach – we have an Interactive Art research group at CITAR and a PhD program, which includes HCI topics)
• …

Conferences on HCI related topics:

• CHI (2010, 2011)
• UIST
• DIS
• NIME
• ACM MM
• Ubicomp
• TEI
• TABLETOP
• …

Journals on HCI related topics:

• TOCHI
• Interactions
• …

And a site with some pointers to more conferences, journals, events related to Design, HCI, Interaction and so on.

Some more links to stuff I need to review/research/organize:

• BOOK: Human-Computer Intreraction, Dix, Finland, Abowd, Beale
• A study on HCI by Alan Dix et al. (the same authors of the book above)
• A book chapter on HCI
• BOOK: Designing the User Interface: Strategies for Effective Human-Computer Interaction, Ben Shneiderman
• Human–Computer Interaction: A Journal of Theoretical, Empirical, and Methodological Issues of User Science and of System Design (and some PDF here)
• HCI bibliography site
• Wikipedia HCI article

Bifocal Display by Robert Spence and Mark Apperley

Really neat ideas developed in the early 80s, whch with today’s multitouch devices are brought once again to the discussion of the day.

Bifocal Display by Robert Spence and Mark Apperley.

I wonder (and will do some research on this) on Bifocal display work being done in the field of sound (well, Cherry proposed something related in his cocktail party effect theory)…