Aim:To create a Sixth Sense device which works of the
principles of gesture recognition and image processing to capture, zoom(in and
out) and toggle pictures with ease just by the help of colored caps worn on the
fingertips of the user.
Motivation: When we were thinking of projects for the summers to
execute, we thought of making a hand, which would do exactly what our hand is
which could have been something brilliant in robotics. Then we came across a
TED talk by Pranav Mistry where he was talking about how we can connect the
physical world with the digital world using Sixth Sense. It looked simply
amazing how he was able to just show his fingers in front of his camera (phone)
and the camera would click pictures. This just enthralled us and was just
something we had to make considering it’s sheer brilliance.
Theory: ‘SixthSense’ is a wearable gestural interface that
augments the physical world around us with digital information and lets us use
natural hand gestures to interact with that information. Every one of us is
aware of the five basic senses – seeing, feeling, smelling, tasting and
hearing. These senses have evolved through millions of years. Whenever we
encounter a new object/experience our natural senses tries to analyse that experience
and the information that is obtained is used to modify our interaction with the
environment. But in this new age of technologies the most important information
that helps one to make right decision is something that cannot be perceived and
analysed by our natural senses. That information is the data in the digital
form, and it is available to everyone through sources like internet. The sixth
sense technology concept is an effort to connect this data in the digital world
in to the real world. Although miniaturized versions of computers help us to
connect to the digital world even while we are travelling there aren’t any
device as of now which gives a direct link between the digital world and our
physical interaction with the real world. Usually the information’s are stored
traditionally on a paper or a digital storage device. Sixth sense technology
helps to bridge this gap between tangible and non-tangible world. The sixth
sense technology was developed by Pranav Mistry, a PhD student in the Fluid Interfaces
Group at the MIT Media Lab. According to him the sixth sense technology has a
Web 4.0 view of human and machine interactions.The SixthSense technology
contains a pocket projector, a mirror and a camera contained in a pendant-like,
wearable device. Both the projector the camera and sensors are connected to a
coding device (in our case-a laptop) in the user’s pocket. The projector
projects visual information enabling surfaces, walls and physical objects
around us to be used as interfaces; while the camera recognizes and tracks
users' hand gestures and physical objects using computer-vision based
techniques. The software program processes the video stream data captured by
the camera and tracks the locations of the colored markers (visual tracking fiducials)
at the tips of the user’s fingers. The movements and arrangements of these
fiducials are interpreted into gestures that act as interaction instructions
for the projected application interfaces. The Sixth Sense prototype is used to
implement several applications that have shown the usefulness, viability and
flexibility of the system.

The Sixth Sense: SixthSense is a wearable gestural interface that
augments the physical world around us with digital information and lets us use
natural hand gestures to interact with that information.
Components of the device:Camera It
captures the image of the object in view and track the user’s hand gesture. The
camera recognizes individuals, images, pictures, gestures that user makes with
his hand. The camera then sends this data to a smart phone for processing.
Basically the camera forms a digital eye, which connects to the world of
digital information.
Colored
Marker There are color markers placed at the tip of users finger. Marking the
user’s fingers with red, yellow green and blue colored tape helps the webcam to
recognize the hand gestures. The movements and arrangement of these markers are
interpreted into gestures that act as a interaction instruction for the
projected application interfaces.
Microsoft-enabled
Laptop The SixthSense device consists of a laptop which process the data send
by the camera. The laptop interprets the hand gestures with help of the colored
markers placed at the fingertips. Dev-C++ was the programmer used for all codes
made. MinGW was the compiler used. OpenCV libraries were used which contained
all functions sufficient to make our project.
Projector
The information that is interpreted through the smart phone can be projected
into any surface. The projector projects the visual information enabling
surfaces and physical objects to be used as interfaces. The projector itself
consists of a battery which have 3 hours of battery life .A tiny LED projector
displays the data sent from the smart phone on any surface in view- object,
wall or person.
Technologies that are related to Sixth Sense Devices
Augmented Reality
The augmented Reality is a visualization
technology that allows the user to experience the virtual experience added over
real world in real time. With the help of advanced AR technology the
information about the surrounding real world of the user becomes interactive
and digitally usable. Artificial information about the environment and the
objects in it can be stored and retrieved as an information layer on top of the
real world view. When we compare the spectrum between virtual reality, which
creates immersive, computer-generated environments, and the real world,
augmented reality is closer to the real world. Augmented reality adds graphics,
sounds, haptic feedback and smell to the natural world as it exists. Both video
games and cell phones are driving the development of augmented reality. The
augmented systems will also superimpose graphics for every perspective
available and try adjusting to every movement of the user's head and eyes. The
three basic components of an augmented reality system are the head-mounted
display, tracking system and mobile computer for the hardware. The main goal of
this new technology is to merge these three components into a highly portable unit
much like a combination of a high tech Walkman and an ordinary pair or
eyeglasses. The head-mounted display used in augmented reality systems will
enable the user to view superimposed graphics and text created by the system.
Another component of an augmented reality system is its tracking and
orientation system. This system pinpoints the user's location in reference to
his surroundings and additionally tracks the user's eye and head movements.
Augmented reality systems will need highly mobile computers. As of now many
computers aren’t there to satisfy to provide this option.

Gesture Recognition: It is a technology which is aimed at interpreting
human gestures with the help of mathematical algorithms. Gesture recognition
technique basically focuses on the emotion recognition from the face and hand
gesture recognition. Gender recognition technique enables humans to interact
with computers in a more direct way without using any external interfacing
devices. It can provide a much better alternative to text user interfaces and
graphical user interface which requires the need of a keyboard or mouse to
interact with the computer. Interfaces which solely depends on the gestures
requires precise hand pose tracking. In the early versions of gesture
recognition process special type of hand gloves which provide information about
hand position orientation and flux of the fingers. In the SixthSense devices
colored bands are used for this purpose. Once hand pose has been captured the
gestures can be recognised using different techniques. Neural network
approaches or statistical templates are the commonly used techniques used for
the recognition purposes. This technique has a high accuracy usually showing
accuracy of more than 95%. Time dependent neural network will also be used for
real time recognition of the gestures.

Computer Vision:Computer
vision is the technology in which machines are able to interpret/extract
necessary information from an image. Computer vision technology includes
various fields like image processing, image analysis and machine vision. It
includes certain aspect of artificial intelligence techniques like pattern
recognition. The machines which implement computer vision techniques require
image sensors which detect electromagnetic radiation which are usually in the
form of ultraviolet rays or light rays. The computer vision find itself
applicable in varies field of interest. One such field is bio medical image
processing. It’s also used in autonomous vehicle like SUV’s. The computer
vision technique basically includes four processes.
1.
Recognition: One of the main task of computer vision technique is to determine
whether the particular object contain the useful data or not.
2. Motion Analysis: Motion analysis includes
several tasks related to estimation of motion where an image sequence is
processed continuously to detect the velocity at each point of the image or in
the 3D scene.
3. Scene
Reconstruction: Computer vision technique employs several methods to recreate a
3D image from the available images of a scene.
4. Image Restoration: The main of aim of this
step is to remove noise from an given image. The simplest method involves using
simple filters like low pass or median filters. In order to get better quality
images more complex methods like Inpainting are used.
ADVANTAGES OF SIXTH SENSE DEVICES
Portable
One of the
main advantages of the sixth sense devices is its small size and portability.
It can be easily carried around without any difficulty. The prototype of the
sixth sense is deigned in such a way that it gives more importance to the
portability factor. All the devices are light in weight and the smart phone can
easily fit into the users pocket.
Cost effective
The cost
incurred for the construction of the sixth sense proto type is quiet low. It was
made from parts collected together from common devices. And a typical sixth
sense device cost up to $300. The sixth sense devices have not been made in
large scale for commercial purpose. Once that happens it’s almost certain that
the device will cost much lower than the current price.
Connectedness
between real world and digital world
Forming a connection between the real world
and the digital world was the main aim of the sixth sense technology.
Data access directly from the machines in real
time
With help
of a sixth sense device the user can easily access data from any machine at
real time speed. The user doesn’t require any machine-human interface to access
the data. The data access through recognition of hand gestures is much easier
and user friendlier compared to the text user interface or graphical user
interface which requires keyboard or mouse.
Open source software
The
software that is used to interpret and analysis the data collected by the
device is made open source. This enables other developers to contribute to the
development of the system.
APPLICATION:
The sixth
sense technology finds a lot of application in the modern world. The sixth
sense devices bridge the gap by bringing the digital world into the real world
and in that process allowing the users to interact with the information without
the help of any machine interfaces. Prototypes of the sixth sense device have
demonstrated viability, usefulness and flexibility of this new technology.
According to the words of its developers extend the of use of this new device
is only limited by the imagination of human beings. Some practical applications
of the sixth sense technology is given below
Map
Application With the
help of a map application the user can call upon any map of his/her choice and
navigate through them by projecting the map on to any surface .By using the
thumb and index fingers movements the user can zoom in, zoom out or pan the
selected map.
Taking
Pictures Another application of sixth
sense devices is the implementation of a gestural camera. This
camera takes the photo of the location user is looking at by detecting the
framing gesture. After taking the desired number of photos we can project them
onto any surfaces and then use gestures to sort through those photos and
organize and resize them. The user taking making a pose gesture. The camera
takes the photo of the location
Drawing
Application The drawing
application allows the user you to draw on any surface by tracking the
fingertip movements of the user’s index finger. The pictures that are drawn by
the user can be stored and replaced on any other surface. The user can also
shuffle through various pictures and drawing by using the hand gesture
movements
Making
Calls We can make calls with help
of sixth sense device .The sixth sense device is used to project the keypad
into your palm and the using that virtual keypad we can make calls to anyone.
Interacting
with physical objects The SixthSense system also helps to interact with
physical objects we use in a better way. It augments physical objects by
projecting more information about these objects projected on them. For example,
a gesture of drawing a circle on the user’s wrist projects an analog watch on
the users hand. Similarly a newspaper can show live video news or dynamic
information can be provided on a regular piece of paper Newspaper showing live
feeds an analog watch projected into the wrist
Getting Information Sixth sense devices can be used for getting various
information relating to our everyday life by getting in contact with objects
Product Information: Sixth sense
technology uses marker technology or image recognition techniques to recognize
the objects we pick in our hand and then provide information relating to that
product.
Book Information: By holding and shuffling through the book pages, the
sixth sense provides Amazon ratings on that book, other reviews and other
relevant things related to the book.
Flight Updates: With the
help of the sixth sense technology it is no longer required to log into any
sites for checking the status of the flights. The system will recognize your
boarding pass and let you know whether the flight is on time or not.
information About People: With help of face recognition techniques the sixth sense devices are
able to provide information about the people when we meet them. The sensor
detects the face and checks the data base for the relevant information. The
system will then project the relevant information about a person like what they
do, where they work, Boarding pass showing current status.
SOURCES: INTERNET,
BOOKS
PROJECTS
ASSIGNMENTS