This fingertip haptic display with integrated force, tactile, and thermal feedback in a miniature form-factor can be worn easily and used with augmented reality applications without affecting existing tracking technologies. It can be used to alter the haptic properties of real objects by rendering projected visual and haptic feedback. The system consists of a custom force-display mechanism (Gravity Grabber) to render vertical forces, shearing forces, high-frequency tactile vibrations, and a Peltier module for thermal display. The integrated haptic display module weighs less than 50 grams, can be easily interfaced to a PC with just one micro USB cable, and works independent from additional hardware. The Altered Touch display could be expanded to design a haptic glove that can interact with both virtual and augmented worlds.

atmoSphere uses spatial audio and haptic feedback provide immersive music experiences. Through its combination of spatialized music and a sphere-shaped device that provides haptic feedback, users imagine large sound environments and feel haptic sensations in their hands.

Cardiolens is an augmented reality application that enables real-time hands-free measurement and visualization of blood flow and vital signs. It provides a new way to understand physiology and has applications in healthcare and affective computing.

Imaging photoplethysmography with a webcam enables remote measurement of blood perfusion and vital signs. However, the perfusion signal is not perceptible by the unaided eye. In Cardiolens, a commercially available augmented reality headset is modified to both measure and visualize physiological signals. A front-facing camera captures ambient light reflected from the subject's face and analyzes the light to compute blood-volume pulse and vital signs.

Cardiolens has numerous applications. In surgery, it could be used to identify blood flow to various organs simply by looking at the body part. In public speaking, it could be used to measure the cognitive load of the audience by sensing changes in heart-rate variability (sympathetic nervous system response).

This do-it-yourself add-on integrates position tracking to provide six degrees of freedom to most smartphone-based mobile VR systems (for example, Google DayDream, Samsung Gear VR, etc.) and enable a wide variety of activities that require hand manipulation. It enables distribution of therapeutic content that involves embodiment and hand-eye coordination to mobile VR users, which is a larger population than desktop VR users. Attendees can build their own customized controllers and participate in a mindfulness exercise: Stone Balancing.

Since visual and auditory navigation of human walking requires interpretation of navigation information, user attention is consumed, and risk is increased. HangerON uses the Hanger Reflex, an illusory phenomenon caused by haptic stimulus. Mounted on the user's waist, it causes a rotating left-right movement to manipulate walking direction. The demo includes three applications of the method: normal walking navigation that automatically achieves a destination, remote control of one user by another user, and user-controlled walking.

Hapbeat delivers strong force feedback with high temporal resolution in a compact and easy-to-wear device. With two coreless motors and a string wound around the user’s body, it provides both compressive force and acoustic vibration. The user can distinguish the strength of hitting forces and the material of the hitting object (whether a sword is iron or bamboo, for example).

The device transmits forces to a wider area of the body than conventional vibrators, and while vibrators have a limited linear stroke, motor rotation is not restricted. Conventional vibrators also have limited bandwidth for high frequencies because they move their heavy bodies to transmit vibration. In contrast, Hapbeat moves only light coreless rotors and a string. For sound listening, these features are turned into sensations of strong low-frequency air movements from drums or cannons and high-fidelity vibrations of acoustic instruments.

Most VR experiences require users to wear headset , but they occlude the face and block eye-gaze. This project virtually “removes” the headset and reveals the face underneath it, creating a realistic see-through effect. Using a combination of 3D vision, machine learning, and graphics techniques, the system synthesizes a realistic, personalized 3D model of the user's face to reproduce the user's appearance, eye gaze, and blinks. With person-detection and pose-estimation technology, the system can simultaneously detect and track body and face keypoints of multiple people in the scene.

Some of the fundamental limitations of existing near-eye displays (NEDs) for augmented reality (AR) are limited field of view (FOV), low angular resolution, and fixed focal state. Optimizing a design to overcome one of these limitations typically leads to a trade-off in the others.

This project introduces an all-in-one solution: deformable beamsplitters. It is a new new hybrid hardware design created by combining hyperbolic half-silvered mirrors and deformable membrane mirrors (DMMs). The key components are two see-through DMMs controlled by airtight cavities to change the effective focal power and present a virtual image at a target depth plane. The benefits of the membranes include a wide field of view (100 diagonal) and fast depth switching (from 20 cm to infinity within 20 ms). The display promises to address vergence-accommodation conflict caused by lack of accommodative cues and offers wide-field-of-view capabilities.

MetaLimbs proposes a novel approach to body-schema alternation and artificial-limb interaction. It adds two robotic arms to the user's body and maps the global motion of legs and feet relative to the torso. It also maps local motion of the toes. Then it maps these data to arm and hand motion, and to fingers gripping the artificial limbs, adds force feedback to the feet, and maps the feedback to the manipulator's touch sensors.

Sensing objects hidden behind an occluder is a fascinating emerging area of research that is expected to play a significant role in numerous fields such as automobile safety, remote observation, or endoscopy. In the past, this problem has consistently been solved with expensive time-of-flight technology, which often requires a long reconstruction time. This system is the first that relies only on off-the-shelf intensity cameras and lasers. It includes a novel analysis-by-synthesis scheme that utilizes the output of a specialized renderer as input to an optimizer to perform the reconstruction. Viewers can freely move the occluded object around in the hidden scene, while the camera setup on the other side of the wall reconstructs the object position and orientation in real time.

Real Baby - Real Family is an entertainment VR baby-avatar-generation system that includes visual, audio, and haptic feedback; a physical baby form; and a virtual baby whose face is generated by combining two photos of the players. I project simulates a complete baby-nursery experience.

Commercial virtual reality devices such as the Oculus Rift and HTC Vive have demonstrated plausibly realistic audio/visual sensory input, but somatosensory feedback has been more limited in scope. Most successful attempts to provide feedback to the human kinesthetic system have been addressed by vibration-based devices, like linear resonant actuators and eccentric rotating mass actuators, aimed at stimulating receptors near the surface of the skin. This method is widely accepted as a proxy in lieu of resistance to muscle tension, but it is not considered a path to realistic input.

Advances in materials science and soft robotics present a plausible alternative. This research explores their use for kinesthetic feedback in virtual reality. The materials used to build these devices (silicon, for example) have mechanical properties similar to those of human tissue and are well-suited for applications that require direct and prolonged contact with human skin. The devices are also well-suited for commercial applications due to their low material cost and compatibility with scalable manufacturing techniques.

Two key demonstrations highlight the use of fluidic elastomer actuators to provide haptic feedback. They allow users to progress through a series of brief experiences where the hand-held controller adjusts its form and behavior to simulate virtual objects including a goo gun, a pistol, and a mallet.

Holograms are emerging in our living environment with higher and higher levels of quality. The current solutions from Microsoft, Daqri, or Meta enable users to place virtual elements in a static location beside real objects, which gives the virtual elements a strong feeling of presence. Among the technical advances, new display systems play a large role, but the evolution of inside-out tracking is probably the more impressive step forward. Still there is considerable scope for progress in severawl areas, from hardware solutions to interaction metaphors: How can we manipulate this virtual object that seems so real?

This demonstration adds a method for touching the objects, based on a touch development kit from Ultrahaptics, the only mid-air tactile feedback technology. It provides a touch feeling without any mechanical equipment in the visualization area (which would be inconsistent with the hologram concept). Touch Hologram in Mid-Air is a singular and surprising experience that illuminates a broad field of future research. It is unique in giving physical presence to intangible objects.

Augmented reality (AR) is gaining momentum, with multiple commercial see-through near-eye displays arriving such as the Meta 2 and the Microsoft Hololens. These devices are a big step toward Ivan Sutherland’s original vision of an "ultimate display", but many limitations remain. Existing displays struggle to achieve a wide field of view (FOV) with compact optics. Such displays also suffer from the vergence-accommodation conflict: they present synthetic images at a constant optical distance, requiring a fixed focus from the eye's accommodation mechanism, while the vergence of the two eyes working in concert places the synthetic object(s) at a distance other than the accommodation distance.

This project employs a novel wide-FOV optical design that can adjust the focus depth dynamically, tracking the user's binocular gaze and matching the focus to the vergence. Thus the display is always "in focus" for whatever object the user is looking at, solving the vergence-accommodation conflict. Objects at a different distance, which should not be in focus, are rendered with a sophisticated simulated defocus blur that accounts for the internal optics of the eye.