A Usb-camera Based Pointing Device - Research Paper Example

Table of Contents Chapter 2 Introduction 2 1 Overview 2 chapter 2 2 2 Background theory 2 2 touchscreen 3 2 1 Types of Touch Screen 3 2.2 Previous work on the topic 5 Chapter 3 9 3 Design 9 3.1 Camera-Screen calibration 9 3.2 The Main system 10 3.3 Cursor movement 10 4 Implentation/test 10 5 discussion 15 6 conclusion 15 Chapter 1 1 Introduction Pointing devices are described as interface such as the touchpad and graphic tablets are essential parts of computer peripherals used by general computer users and graphic designers to input data and perform designing projects. These are several known pointing devices some of which are mouse, graphics tablets and touchpad just to mention but a few. Mouse, are widely known pointing device that operates by discovering two-dimensional motions relative to its supporting surface. While graphics tablet are computer-inputting device that grants the user the ability to utilize their hands to draw on the tablet while the images drawn is subsequently captured and translated on the computer screen. Apparently these devices are new in the market and available systems that perform related task are usually hardwired and expensive making them inaccessible to generally users. Therefore, the objective of this project is to build a system that's affordable, convenient and readily available to people from various background. In general people prefer to have intuitive systems when performing graphical task and this could be an inexpensive way of achieving this because a USB cam typical cost about '25. 1.1 Overview Therefore the main objective of this project will attempt to create an alternative to the conventional pointing device by creating software, which sets a USB cam as a focusing device taking video images and processing them in real time and visually tracking a stylus looking object. The stylus movement will typical translate into the movement of the pointer (cursor) on the display which will allow for a smooth control of the graphical user interface. The project will firstly employ a camera to screen calibration system, which will enable the software to determine the connection between different locations on screen and several locations in the view of the USB cam. This calibration process will ideally run when the software starts up, which might displays a distinctive pattern at different locations, which will be detected and located by the user using the software. In addition, a stylus-tracking system, which could works by subtracting, from each video frame, a running average of the frames, so that the stylus looking object will stand out, and then perform image processing to it. chapter 2 2 Background theory Generic touch screen devices detect the location and presence of a touch in areas within and around certain displays. The essential role of this device is to reduce dependence on conventional mouse and enable the user interact with the object displayed directly. However, due to the fact these devices employ specifically designed hardware making them prohibitively and inaccessible to those that truly requires them. As earlier stated, mouse, as widely known is a type of pointing device that operates by discovering two-dimensional motions relative to its supporting surface. The motion of a mouse characteristically transform into the motion of a pointer on a display, making a fine control of a user interface realizable. Graphics tablet on the other hand, are computer-inputting device that grants the user the ability to utilize their hands to draw on the tablet while the images drawn is subsequently captured and translated on the computer screen. This could be expressed by a flat surface in which a user can hand draw images or graphics using a stylus. 2.1 touchscreen A touchscreen is more or less a display which senses the presence and location of a touch within such displayed area. The term by and large refers to touch or contact made to the display of the device by the use of ones finger or hand.Touchscreens are designed with an ability to sense other passive objects, such as a stylus. Nevertheless, if the object sensed is active, as with a light pen, the term touchscreen will generally be of no effect. Therefore ability to interact directly with a display typically indicates the presence of a touchscreen 2.1.1 Types of Touch Screen The following are a number of types of touchscreen; Surface acoustic wave Optical imaging touch screen devices DViT (Digital Vision Touch) Technology Interactive Touch Screen Windows Surface acoustic wave (SAW) technology Require an ultrasonic wave that passes through the touchscreen panel. As soon as the panel is touched, a portion of the wave is then absorbed. The change that occurs in the ultrasonic waves documents the position of the touch event transferring this information to the controller for processing. Surface wave touchscreen panels if placed with outside elements can most likely be contaminated. Hence, contaminants on the surface can in like manner interfere with the functionality of the touchscreen. Optical imaging touch screen devices Another type of touchscreen is a relatively-modern development in touchscreen technology, two or more image sensors are placed around the edges primarily in the corners of the screen. The Infrared backlights are placed in the camera's field of view on the other sides of the screen. A touch shows up in form of a shadow and each pair of cameras can then be triangulated to pinpoint the touch or so much as measure the size of the touching object. This technology's acceptance is growing by the day, due to its scalability, flexibility, and affordability, especially for larger units. DViT (Digital Vision Touch) Technology This newer developed device by SMART Technologies Inc. Allows not only clear-cut, instinctive touch control but likewise excellent image quality. DViT at present is familiar with touch gesture also. With both the Rear-Projection SMART Board interactive whiteboards and the SMART Board flat-panel display superimposes. This type of technology allows users to touch the interactive whiteboard's surface in order to control computer applications or write on the screen. Fingers are detected with the use of Digital cameras and software or pen-tool contact on the screen, later translated to the computer as mouse or pen-tool activity. Unlike resistive technology, DViT technology does not necessitate technical components in the screen base on the fact that the digital cameras are in the bezel (frame), not in the screen's surface. See figure 1. Figure 1: Smart Board Interactive Touch Screen Windows With this kind of technology the lightweight Touch Screen Foil can be applied directly to a window or a glass sheet and then a rear projection screen or LCD can be mounted behind the Touch Screen Foil to create a through-window/glass touch experience. Visual Planet Ltd fabricates and is the global distributor of an Interactive Touch Screen Foil known as the "ViP Interactive Foil. This ViP Interactive foil can be mounted using either a permanent or a removable fixing method allowing the touch unit to be moved to another location if required. The Touch Screen Foil is ideal for upgrading existing through window rear projection or LCD displays to create a through window touch experience. See figure 2. Figure 2: Interactive Window 2.2 Previous work on the topic Previous works about this topic has proved that several students and professionals have tried out the idea of a cheaper version of a commercial touch screen system. A Microsoft professional by the name Michael Schwarz created a finger tracking software, which basically tracks the finger, translates the finger movement to the cursor movement. He adopted a method by which a blank A4 paper was placed on a surface and points the web cam facing down toward the blank paper. He then adjust the web cam with the system and places his finger between the paper and the camera moving his finger which then translates, into the cursor moving but he was only able to get a vertical movement. Figure 3: Picture of Kmando Another example was software called Kmando that was created by a group of students and associate in the Microsystem & Machine Vision Laboratory at Sheffield Hallam University, UK. This software was created for camera-projector interaction. The software utilized a webcam that is arranged against a standard projector screen. The webcam is afterward used to determine the position of physical pointer for example, a pen, which is eventually used to virtually move the pointer. Point-and-click functionality was also been implemented. See figure 3. Equally of importance, studies relating to vision-based new input devices have been recognized. In systems labelled Visual Panel an ordinary piece of paper can be used as virtual mouse, keyboard and a 3D controller. User can make use of their fingers or other pointing devices to enter text and also can perform some action like clicking, dragging etc. In an alternative system called Finger-Mouse; a camera mounted above the keyboard allows the user's finger to be tracked and the mouse cursor is moved accordingly. A laser pointer drawing the characters on a flat surface and a head mounted camera captures the end user actions. The beam of the laser pointer is detected in each camera image and the corresponding characters are recognized from these traces of the laser beam. The Wellner's concept of Digital Desk is considered as one of the first examples of vision-based HCI system. This concept uses a couple of camera and a video projector to create an augmented environment. Users are able to interact with papers on the desk by using their bare fingers or by the use of any pointing devices making it possible for them to store up all the data in digital form. Although requiring a highly restrained atmosphere, it appears that visual input offers natural mains of communicating with computers. Quite a number of different projection systems have recently been developed following Digital Desk. Crowley et al. developed a system called Magic board in which an ordinary white board is turned out to be a digital one. Therefore making possible for all markings on the board to be kept in digital form and some high level operations such as copying, moving, zooming etc. are achievable by projecting these changes on the board via a projector. Figure 5: Example of a Digital Desk As a replacement for using certain devices such as electronic tablets or digitizers to obtain handwritten data, pen trajectories can as well be tracked by means of inspecting the visual data. These systems are composed of only a camera and a pointing device such as a pen. Munich and Perona work; a vision-based method is proposed for obtaining on-line handwritten data. Afterwards, this system is extended by a word recognition module. Alternatively, vision based system for recognizing isolated characters is introduced. A stylus or a laser pointer draws these characters on a flat surface and a head mounted camera captures the user's actions. The beam of the laser pointer is detected in each camera image and the corresponding characters are recognized from these traces of the laser beam. Additionally, visual inputs are very appropriate in the systems developed for people with disabilities. For instance, in American Sign Language is recognized through the use of video input sequence in real-time. IBM likewise developed vision-based Touch-Free for disabled people making it feasible for a switch to be activated by a choice of large or small body gestures. Infrared cameras can in addition be used to track head gestures and the gazing of the eyes. An additional case in point, Zhai et, al. developed an eye tracker system called MAGIC; of which some products are available in the market. Prior works have been reported to demonstrate the use of human fingers as pointing device. Although the configurations of these systems differ with the use of a wearable computer and also using camera-projector pairs, they are all assuming a plane-to-plane image-screen mapping. In view of the fact that a large number of computer screens are not flat, this hypothesis does not apply in this case. Many applications can be discover which proves this type of vision-based interface is desired. For instance, in a smart room, the user wants to handle a remote as well as large display or play a game, but they are in a comfortable couch instead of sitting directly in front of the computer, and therefore the mouse and keyboard may not be easily accessible. What could they perform their task in such situation' They may pick up an arbitrary paper at hand and move their fingers or pen on the paper to drive a cursor or type some text. Without a doubt, such an interaction will be made possible by having a camera gaze at the user and therefore analyzing the movement of the paper and the user alike. Other instances show several people are discussing in a meeting room using a large display. They may be required to draw some pictures to illustrate their ideas. However, it became unrealistic to facilitate every user a mouse and a keyboard. What could be done in such circumstance' Once more, they may select any paper and use their fingers to draw their ideas that will be then shown in the large display. By this means, a more immersive discussion can be achieved, especially, in a large lecture room where the lecturer sometimes requires students to something jolted on a small white board. However, the audience in a distance from the lecturer may not be able to see clearly what's been written. Due to the constraints of the bandwidth, it would not be feasible to broadcast the video of the writing. In this situation, a vision-based system is needed to analysis what the lecturer writes and retrieves it in remote display. Consequently, in scenarios such as the smart rooms, immersive discussions and teleconferencing, conventional mice and keyboard turns out to be unsuitable consequently motivating the development of a vision based gesture interface. Chapter 3 3 Design This section of the report will attempt to expand on the design of the system, how it functions and what the end result is meant establish. The setup of the system basically consist of a camera positioned in front of computer screen and the position has to be either at an angle from any direction and placed in an area that will not interfere with the end user. Normally the camera will should not be too far from the screen. The system is divided into three different sections, which performs at different stages in a continuous state. The sections are as followed: Camera-screen calibration The main system i. Video input ii. Colour segmentation iii. Locating stylus tip iv. Cursor positioning Cursor movement 3.1 Camera-Screen calibration The relationship between objects into the real world and an image taken by an optical system, it can be describes with the simplest mathematical pinhole model. Almost all, optical systems like webcams and video cameras, are well modelled with the pinhole model. Thus, the camera to screen calibration is performed at the start of the system, the camera-screen calibration system is used to determine the mapping between the image coordinates and the screen coordinates. It takes into accounts defect likes distortion of the lens, what is usually modelled with radial and tangential coefficients. This mapping is then used in the main system to set the matching screen coordinates for the tip point once its image coordinates are determined. The system might employ some sort of pattern on the screen to be used as calibration points. The patterns could be square boxes or circles and this will ideally be displayed on the screen and user will either be required to select a point on the pattern or a system that calculates the centre of the pattern will be used. Calculating the centre of a circle will be easier so once this is determined by either been selected or calculated then the camera to screen coordinates will be determined forming so sort of grid system. 3.2 The Main system The main system is the central part of the system. 1) The system is fed by video stream (sequential images), at each frame the system has the purpose to locate the tip point of the stylus and map its image coordinates into the screen coordinates. The task of tip point location contains two processes: a) segment the stylus from the background, and b) find the tip point of the stylus. The segmentation process consist of several stages: The tip point of the stylus is found: 3.3 Cursor movement The resultant (x, y) coordinates of the stylus tip determined by the find stylus tip function is mapped to the coordinates on the screen, which is determined by the camera-screen calibration initially. We propose a pair of model that relate the mapping between image coordinates and pixel screen position. Xpixel = A11 + A12 Ximage + A13 Yimage Ypixel = A21 + A22 Ximage + A23 Yimage or Xpixel = A11 + A12 Ximage + A13 Yimage + A14 Ximage2 + A15 Yimage2 Ypixel = A21 + A22 Ximage + A23 Yimage + A24 Ximage2 + A25 Yimage2 In practical terms, we know (Xpixel,Ypixel) and (Ximage,Yimage), so we need to calculate the coefficients A11, A12, A13, A14, A15 and A21, A22, A23, A24, A25. This can be done using least square fit method. 4 Implentation/test Camera to screen calibration: In order to calibrate camera and screen, we have taken an image with a regular pattern on the screen, as it is shows in the figure 6. Figure 6: It has taken a screenshot using a webcam with 640 x 512 pixels. The chessboard pattern can be used to calibrate the camera to screen. We propose two laws mapping between the image coordinates and the screen coordinates, given by: Xpixel = A11 + A12 Ximage + A13 Yimage Ypixel = A21 + A22 Ximage + A23 Yimage Xpixel = A11 + A12 Ximage + A13 Yimage + A14 Ximage2 + A15 Yimage2 Ypixel = A21 + A22 Ximage + A23 Yimage + A24 Ximage2 + A25 Yimage2 This mapping is used extensively in the main system to find the tip point coordinate (Xpixel,Ypixel) as a function of screen coordinates (Ximage,Yimage). The figure 7 shows the detected points that help us to find the mapping. The patterns square boxes displayed on the screen and user will either be required to select a point on the pattern or a system that calculates the centre of the pattern will be used. Figure 7: It shows the corners detected. These screen coordinates (Ximage,Yimage) are related to the image coordinates (Xpixel,Ypixel). The segmentation: In order to build the stylus model, we have used some images (see figure 8) of similar objects under different environment conditions. We can add many more images to our stylus library. Figure 8: It is shows several styluses for our library. The sample images are converted from RGB to YCbCr space, We build the Joint Gaussian Distribution Model for this library and we plot the 2D Histogram of Cr Cb values of sample stylus images. Then the sample means for both Cb and Cr components and the covariance matrix are computed in order to fit the data into Gaussian Model. See Figure 9. Figure 9: (left) It shows the 2d histogram Cr Cb values of sample stylus images. (right) It shows the fitted joint gaussian distribution model for our own stylus library. The clusters of pixels with similar colours in the colour space are then grouped together. The reason for segmentation is to limit the discrepancies that might occur under different light intensities. Finally, the stylus segment image is shows in figure 10. Figure 10: (Left) Original frame, it shows the screen and stylus. (Right) It shows the segment image. The background is gone. Note the oblique angle between the plane of the screen and the optical axis of the webcam. The tip point of the stylus is found as describe below: Given a masked image, we calculate the aspect ratio of the object using the stylus function. On the other hand, we calculate centroid on the masked image alone. See figure 11: Figure 11: Masked image. It shows the centroid of the masked image. The stylus function is used to determine longest thinnest object by finding the regional properties of the filled area of the segmented image from the colour segmentation function, the major axis, which is the length of the object in the segmented image, and the minor axis, which is the width of the object in the segmented image, the result is then calculated by finding the square of the filled area (a2) and dividing it by the cube of minor axis (ma3). Figure 12: Block diagram of find stylus tip point. The best possible values, which will be the resultant object is then determined by a calculation that if the value of the minor axis is greater than a certain predetermined level and the calculated result is also greater than a predetermined level then the resultant object is the longest thinnest object the system could find. Once the ellipse is around the cluster of pixels, the orientation of that object is found using a built in matlab function and the major axis of the object is also determined and the centre of mass of the object is also located using the built in centriod function. When the centre of mass (object) of both object are found, another mathematical calculation is used to determine in what direction the tip of the stylus tip is Figure 13: It shows the tip points that we get after whole process, (Ximage,Yimage). Finally, the cursor position on the screen is given by the mapping between image coordinates and screen coordinates. 5 discussion USB-Camera pointing devices. Nowadays, real-time processes are possible using fast clock devices. This project has great practical application when is used with a webcam 320x256 pixels. It has on average, 30 frame-per-second video, which is acceptable. The camera-calibration phase is well-studied and implemented on Matlab and c++/c languages. Any, optical system defect is compensate with second or third order terms in the mapping relationship between real world coordinates and pixel image coordinates. Furthermore, segmentation codes are well-studied too A suitable colour transformation can be used to make the problem invariant under varieties of colour. It can be treated with Bayesian theory, maximum entropy method or neuronal network algorithm. Given the binary image of a stylus object, we can extract its geometrical properties. Once we know the position of the stylus object on the image, we are able to know the cursor position on the screen given by the mapping relationship. 6 conclusion The camera calibration state is fundamental to know the relationship between world coordinate and pixel screen position. A very good calibration (mapping relationship), it can eliminate optical and/or mechanical defects in an optical system. The segmentation phase is dependent on the sample image used to extract the desired object from our own image. Cluster analysis of the segment image is fundamental to know the orientation and position of the tip point's stylus. Works Cited Bouguet , Jean-Yves. Camera Calibration Toolbox for Matlab. 2008. Jie Yang and Alex Waibel, A Real-Time Face Tracker, CMU CS Technical Report. Hartley, Zisserman. Multiple View Geometry in computer vision, Cambridge University Press. 2003. T.A. Clarke and J.G. Fryer. The Development of Camera Calibration Methods and Models. Photogrammetric Record, 1998. Read More