Projects
Activity Recognition
Research Areas: Pattern Recognition, Ubiquitous Computing
Project Description: Traditional approaches to human activity recognition relying on vision as the primary sensory medium have met with little success. The emergence of the ubiquitous and pervasive paradigm of computing has ushered in new low bandwidth wearable, unobtrusive, inexpensive and easily deployable sensors like accelerometers, gyroscopes and RFID tags for the purpose of human activity recognition.
The low level processing of the accelerometer data stream is an interesting multimedia as well as a pattern recognition problem. We are investigating the data stream to identify features that are discriminative, to aid in distinguishing between the different action patterns. In addition to the standard statistical and spectral features based on the Fourier transform, we are studying the properties of the acceleration data stream with respect to other signal processing techniques like Wigner distribution, wavelet transform to identify features with good discriminative capability. We believe that an ideal time frequency representation of the signal will be crucial in distinguishing between the different action patterns.
We are looking to developing a multimodal sensory framework, where the vision sensors are supplemented with other low bandwidth sensors like accelerometers and RFID tags. We propose to use this framework for detecting the objects the user is interacting with along with the associated motion patterns. Bayesian networks that fuse the results obtained from the different data streams is an option that we are considering to develop the multi-modal sensory framework. We also plan to develop activity models for inferring the activities based on the information about the objects the user is interacting with along with motion patterns.
Project Contact: Dirk Joel Luchini Colbry
Project Members: Narayanan Chatapuram Krishnan, Sethuraman Panchanathan, Dirk Joel Luchini Colbry
Funding Sources: National Science Foundation, CUbiC Technology and Research Initiative Fund
Cardiac Decision Support
Research Areas: Computer-aided Diagnosis, Pattern Recognition, Machine Learning
Project Description: CUbiC@ASU is engaged in an active collaboration with a high-volume cardiology practice catering to patients across Arizona - Advanced Cardiac Specialists - to design and develop computational tools and frameworks for cardiac decision support. While there are publicly available heart risk score calculators, we plan to develop a framework that allows for risk propagation, as a patient goes through different diagnostic tests in the clinical pathway. We are currently working on risk stratification for patients who have undergone a Percutaneous Coronary Intervention (PCI) procedure.
This project surfaces several research challenges in multimedia, pattern recognition, clinical machine learning and natural language processing. The fields of pattern recognition and machine learning are founded on the definition of distance metrics between data points that are analyzed. A fundamental problem that we are tackling at CUbiC is the design of inter-patient distance metrics that combine patient data with related ontologies to provide consistency and semantics in the metrics. Also, a very important requirement in applying pattern recognition algorithms to the field of medical diagnosis, more than any other field, is a measure of confidence or belief of an algorithm in its result. We are working on theoretical frameworks to compute confidence measure values that be calibrated for these scenarios. We believe that this design provides a co-aptive system that collaborates with physicians, rather than replacing their skill and expertise.
According to the World health Organization (WHO) reports in 2005, 30% of the deaths in the world result from cardiovascular diseases, and one person dies from a cardiovascular disease every 34 seconds in the United States alone. Although the causes are well-known, the physician-to-patient ratio is low, and the load on cardiologists is only increasing by the day. This has created a need for computational approaches that can provide decision support to aid cardiovascular patients.
Project Contact: Ramkiran Gouripeddi
Project Members: Vineeth Balasubramanian, Sethuraman Panchanathan, Ramkiran Gouripeddi
Funding Sources:
FacePix(100)
Research Areas: Face recognition
Project Description: FacePix is a face image database created at the Center for Cognitive Ubiquitous Computing (CUbiC) at Arizona State University, and made available free of charge to the worldwide research community. In the first version of the FacePix database, called FacePix(30), there are 3 sets of face images for each of these 30 people (each set consisting of a spectrum of 181 images) where each image corresponds to a rotational interval of 1 degree, across a spectrum of 180 degrees, as follows: The first set contains 181 color face images that collectively represent a spectrum of pose angles. These images were captured with a moving video camera, using two stationary diffuse light sources that simulate ambient light. The face images in this set contain very little shadowing. Pose angle variations vary across a range from +90 degrees to –90 degrees, where +90 degrees represents a left profile view, 0 degrees represents a frontal view, and –90 degrees represents a right profile view. (An example of this first set is shown below.) The second set contains 181 color frontal face images that collectively represent a spectrum of illumination angles. These images were captured with a stationary video camera that captured a sequence of frontal views, while a spotlight rotated around the front of the person. The spotlight illumination angles vary across a range from +90 degrees to –90 degrees, where +90 degrees represents a spotlight position that illuminates the right profile of the face, 0 degrees represents a spotlight position that illuminates the front of the face, and –90 degrees represents a spotlight position that illuminates the left profile of the face. This spotlight illumination was combined with two stationary diffuse light sources that simulate ambient light. As a result, the face images in this set contain soft shadowing. The third set is similar to the second set, except that the spotlight illumination was not supplemented with any ambient light. As a result, the face images in this set contain harsh (dark) shadowing.
More Information: http://www.facepix.org/
Project Contact: John A. Black, Jr
Project Members: Sethuraman Panchanathan, Michael J. Astrauskas, John A. Black, Jr,
Funding Sources: National Science Foundation
iCare Reader
Research Areas: Assistive Usability
Project Description:The inspiration for the iCare Reader was provided by focus groups of people who are blind, who asked CUbiC researchers for a reading machine that could taken anywhere, and that was easier to use than the traditional flatbed scanning technology solutions that are available for accessing printed text.
The reader project has been progressing in three phases. The first phase involved the construction of a table-top iCARE Reader, the second phase a portable iCARE Reader, and the third phase a wearable iCARE Reader. The first phase prototypes have now been completed, and are deployed in educational institutions. The second phase prototypes have been constructed, evaluated, and we are now looking for an industry partner to produce the portable as a commercially available assistive technology. The final phase is currently our active area of research. We face the challenge of porting the easy to use iCARE Reader to a hand-held mobile platform, with a lower resolution. Such a system would mean that a non-technical person would be able to use the Reader in the manner that the focus groups intended (i.e. read their mail, recipes, user manuals, and the plethora of print material that is available).
It is our belief that if assistive devices such as the iCARE Readers were manufactured and made available to people who are blind, and prospective employers, they would provide a very effective tool to promote job readiness, successful job placement, and sustainable employment.
Project Contact: Dirk Joel Luchini Colbry
Project Members: Sethuraman Panchanathan, Vineeth Balasubramanian, Lakshmi Gade, Sreekar Krishna, Colin Juillard, Daniel Merril, Dirk Joel Luchini Colbry
Funding Sources: iLearn: IT-enabled Intelligent and Ubiquitous Access to Educational Opportunities for Blind Students NSF ITR-0326544
http://icare.eas.asu.edu/oasisInteraction Assistant
Research Areas: Dimensionality reduction, Expression recognition, Face recognition, Face tracking
Project Description: Interaction Assistant, part of the iCARE projects, is intended to deliver information to a user who is blind so that they can improve their social interactions with sighted counterparts. The device consists of a camera mounted on a pair of sunglasses which the user wears. The video from the camera is analyzed to extract information that can help the user better understand other people in their surroundings. An important goal of the system is face recognition and facial expression analysis.The work also concentrates on developing various delivery mechanisms that the user can resort to for the extracted information. The important research and development components under the interaction assistant project include:
- Reliable face recognition under varying pose and illumination conditions.
- Accurate pose angle estimation.
- Tracking of detected faces.
- Developing innovative information delivery mechanisms (using audio and haptic cues).
- Incorporation of 3D face analysis engine into the existing system.
- Reliable facial feature extraction for expression analysis.
Project Contact: Dirk Joel Luchini Colbry
Project Members: Sethuraman Panchanathan, Vineeth Balasubramanian, Lakshmi Gade, Sreekar Krishna, Colin Juillard, Daniel Merrill, Dirk Joel Luchini Colbry
Funding Sources:National Science Foundation, CUbiC Technology and Research Initiative Fund
Interactive Shopping Environment (Interactive Retail Space)
Research Areas: Ambient intelligence, Aware environments, Assistive shopping environment, RFID reader
Project Description: The goal of the project is to help people who are blind or visually impaired to shop independently. Present day shopping environments are centered around the needs of the consumer and they aim to facilitate convenient access to different product lines varying from groceries to books to everyday electronics. Supermarkets and hypermarkets are examples of trends that have evolved providing a rich shopping experience. Unfortunately, such systems cater to sighted customers and inadequately address the necessities of a person who is blind or visually impaired. In this work we are developing a system that caters to two important aspects of helping shoppers who are visually impaired: Providing indoor navigation within a shopping space and making available all product information.
The navigational aspects of the project are addressed by incorporating combined dead reckoning systems (linear and angular displacement measurement systems) and proximity sensors while product data acquisition is addressed by using RFID technology.
Project Contact: Terri Hedgpeth
Project Members: Sethuraman Panchanathan, Vineeth Balasubramanian, Narayanan Chatapuram Krishnan, Mohammad Alzubaidi, Sreekar Krishna, Terri Hedgpeth
Funding Sources: National Science Foundation - HCC: The iCare Ambient Interactive Shopping Environment IIS-739744
OASIS - Organizing, Annotating, and Serving Information to Students
Research Areas: Accessibility, Data and information manipulation and management, Usability
Project Description: In CUbiC under the iCare project at Arizona State University, in partnership with Arizona’s Rehabilitation Services Administration, we are developing a digital library system for delivering textbooks and other course materials to post secondary education students, who are blind or visually impaired, throughout the state of Arizona. This system is called: OASIS (Organizing, Annotating, and Serving Information to Students). The ultimate goal of OASIS is to enable these students to effectively use relevant content from multiple sources such as books, notes, journal articles, and web sources as they study and do research for their courses.
OASIS provides a fully accessible digital library to support these students in successfully accessing not only their own textbooks and course materials, but books and other educational materials as well; and offers an innovative set of tools to facilitate learning such as automatic summarizations, user generated virtual books, automated content recommendations during searches, and an easy to use upload tool.
Project Contact: Terri Hedgpeth
Project Members: Sethuraman Panchanathan, Terri Hedgpeth
Funding Sources: Department of Economic Security - Rehabilitation Services Administration
Prosopagnosia and Face Recognition
Research Areas: Face Recognition
Project Description: Prosopagnosia is a condition where individuals with otherwise normal vision are able to see faces, but cannot recognize faces. A major study in Germany (Kennerknecht 2006) reported that a congenital (inherited) form of prosopagnosia occurred at a rate of 2.5% in a sample of 689 subjects. This study implies that the number of people who have problems with face recognition is much larger than previously assumed and implies that many people with prosopagnosia have "found personal solutions" to overcome their deficits. By understanding how people with and without prosopagnosia complete face recognition tasks, we can determine the set of typical and atypical visual features that are robust for face recognition.
This project seeks to identify a set of visual features that will robustly recognize individuals in real-world settings (where lighting, pose and expression may vary) by working with Prosopagnosiacs (people who have prosopagnosia, or “face blindness”). These features in addition to resulting in the design of assistive devices for individuals with atypical vision will also enhance the human recognition performance in other applications such as homeland security, authentication, and combating identity theft. We will investigate the set of visual features employed by individuals with typical vision and individuals with prosopagnosia (face blindness).
Project Contact: Dirk Joel Luchini Colbry
Project Members: John A. Black, Jr, Sethuraman Panchanathan, Dirk Joel Luchini Colbry
Funding Sources: CUbiC Technology and Research Initiative Fund
Rate the Surgeons
Research Areas: Collaborative framework, Haptics
Project Description: Development and refinement of surgical skills are critical and time consuming components of the training curriculum of novice surgeons. Traditional methods for surgical training and evaluation are known to be long-drawn processes in which interns and junior residents perform surgical procedures under the supervision of a senior surgeon, commonly known as the apprenticeship method. This method is used for training as well as subjective evaluation of surgical interns. The apprenticeship method is designed to provide focused one-to-one training that is critical to learning life saving procedures. In recent years, the preferred mode of surgical access has shifted from open surgery to minimally invasive surgery (MIS). Advances in computer science and applications such as sensing, modeling and visualization, artificial intelligence, control systems, data acquisition and analysis have made robotic surgery an achievable goal and a logical next step in the evolution of surgical procedures. While open surgery allowed surgeons to make sizeable incisions and manually handle surgical tasks such as suturing and sewing, MIS and robotic surgery distances the patient from the surgeon thereby limiting manual handling. With increasing distance between the patient and surgeon, in some cases by thousands of miles, surgical instruments, such as probes, have become more sophisticated and complex. Senior surgical educators are faced with the daunting task of teaching residents how to properly conduct procedures and simultaneously judge their proficiency. One of the major problems with training surgical residents is obtaining an objective surgical proficiency evaluation. Now, more than ever, surgeons are faced with novel complex procedures, increased focus on patient safety, and reducing costs. This complexity leads to an increased need for improved training and evaluation methods for novice surgeons.
Project Contact: Kanav Kahol
Project Members: Sethuraman Panchanathan, Daniel Villanueva, Kanav Kahol
Funding Sources: Banner Health
http://www.ratethesurgeons.com
Remote Visual Explorer
Research Areas: Eye Tracking, Assistive Technology
Project Description: The Remote Visual Explorer is designed to be an intuitive and natural extension of the human vision system. By integrating an eye-tracking monitor and a camera on a pant-tilt mechanism it allows an individual to explore a remote location. The user simply sits in front of the monitor, which is showing a full-screen image of what the camera sees, and looks around, causing the pan-tilt mechanism to center on what the user is focusing on. Various methods of zooming exist, including focus duration, voice control, keyboard control, distance between eyes and monitor, and retina size. The user can also save and restore points of interest.
Project Contact: John A. Black, Jr
Project Members: Sethuraman Panchanathan, John A. Black, Jr
Supervised Non-invasive Mixed-signal EMG Decomposition
Research Areas: Supervised Searning
Project Description: RSI (Repetitive Stress Injuries) in the hands, wrists, elbows, arms, shoulders, back, and neck are caused by a number of factors, including repetitive tasks, awkward and fixed positions, forced movements, and insufficient rest between these. Injury can occur in muscles, tendons, and nerves, with symptoms ranging from irritation to debilitating pain. Fatigue has been linked to RSI, leading to analysis of muscle motion and fatigue. However, to precisely determine which muscle are used in hand and lower movements, current technology requires the use of invasive probes injected through the skin into the muscles.
The aim of this project is to use a Cyberglove and the EMG (electromyogram) to establish if hand posture data can be used to supervise machine learning in order to decompose a mixed signal coming from the surface electrodes placed on a hand and lower arm, thereby allowing for non-invasive monitoring of muscle actuation and fatigue in situ and in motion, with the goal of finding natural hand positions for each participating individual.
Project Contact: Michael J. Astrauskas
Project Members: Sethuraman Panchanathan, Stuart Braiman, Michael J. Astrauskas
Tactics
Research Areas: Image Processing
Project Description: While technologies such as screen-readers and refreshable brail displays exist to allow individuals who are blind access to textual inform, there is still a lack of devices that can automatically convey pictorial content to an individual who is blind. A rough estimate of computer science course curricula at ASU indicates that between 50% and 65% of the material is presented diagrammatically making this information inaccessible to students who are blind or visually impaired.
The goal the Tactics project is to make printed images more accessible to individuals who are blind. The Tactics system is a valuable tool for students and professionals who are blind, because it provides them with the means for independently converting images they find in PowerPoint presentations or on the web into an easy to use tactile Braille image they can touch. What makes this project different from other research is the accessibility of the software. From the installer program to the printing process, we have developed the Tactics system to be fully compatible with existing screen reader software and tactile printers. The Tactics system automatically converts a user-identified image into a format suitable for embossing by adjusting the image to an appropriate resolution and emphasizing the shape of objects in the targeted image. Next, the converted images are sent to a Braille printer/embosser, which embosses the converted image(s) onto Braille paper, making them immediately accessible to the user.
Project Contact: Dirk Joel Luchini Colbry
Project Members: Baoxin Li, Sethuraman Panchanathan, Mohammad Abdulhadi Moh'd Alzubaidi, Daniel Merrill, Brian C Yang, Xiaolong (Jeff) Zhang, Dirk Joel Luchini Colbry
Funding Sources: ASU Foundation Women & Philanthropy
The Note Taker
Research Areas: Accessibility, Computer vision, Educational development
Project Description: Students with mild to severe (but not complete) visual loss, routinely encounter problems of accessibility. When taking notes in an academic setting, most low-vision students must use optics capable of significant zoom. Invariably, however, these preclude them from convenient access to their notepads. Time is lost between zooming, finding and focusing on a spot on the board, looking down and recording whatever is remembered, and then finding and refocusing on the same information. Monoculars and head-mounted cameras—the staple assistive technologies—all suffer from this problem. Note-taking services are also typically provided for these students across the country; however, there are significant disadvantages to these services which make the student less self-reliant and less engaged in the learning process. Something better is needed, something that allows low-vision students to work independently of a service and to view both the board and notes near-simultaneously without a change in context.
We are developing a portable note-taking device that is neither handheld nor head-mounted, and that doesn’t require any additional classroom infrastructure. This note-taking device involves a tablet PC, zoom camera, and electronic pan/tilt mechanism. From the tablet PC, a user will take notes with digital ink while simultaneously viewing the board through a live camera feed. The camera’s position is adjusted by commands issued to the pan/tilt from within the same interface. This simple prototype is being used by students with low-vision. The current interface narrows the gap between low-vision and fully-sighted students in classroom setting, however we intend to further close this gap by developing computer vision algorithms that will automate much of the process.
Project Contact: Dirk Joel Luchini Colbry
Project Members: Sethuraman Panchanathan, David Hayden, Dirk Joel Luchini Colbry
Funding Sources: CUbiC Technology and Research Initiative Fund
Virtual Environment for Orthopedic Drilling Skill Training
Research Areas: Haptic rendering
Project Description: Surgical training typically involves training by performing procedures on synthetic bone models. The bone models used in training are prone to significant wear and tear. This coupled with the lack of cheap yet high quality models affect the quality of the training itself. The need for low cost, extensible and reusable tools for surgical training makes virtual reality (VR) based surgical training environments an appealing complement to current training modules.
Drilling is an important skill in orthopedic surgery that requires significant training and evaluation. While many physical simulators provide surgeons to practice drilling, these simulators are limited by their lack of dynamism. At CUbiC we are developing a virtual environment that provides multi-modal feedback to promote training of the drilling skill in particular. Metrics have been defined to track the user's learning and evaluate performance.
Project Contact: Kanav Kahol
Project Members: Sethuraman Panchanathan, Mithra Vankipuram, Kanav Kahol
Funding Sources: Banner Health
Visio-Haptic Guidance and it's Application to Reaching
Research Areas: Haptics, Vision
Project Description: For an individual who is blind, the ability to reach for an object is a slow and often frustrating activity. The ability to locate an object within the grasp of the individual would be of great assistance. This would allow for the individual who is blind to reach for the object in a more straightforward manner, rather than a slow sweeping motion across the field of reach of the individual. A hand held device using haptic feedback is proposed to allow the individual who is blind to locate objects within the grasp of the individual. Once the individual has a mental picture of the location of the object, the user can more directly reach for the object.
Visio Haptic Guidance Device is comprised of a single hand graspable device capable of collecting video. The device will collect video which will be analyzed to locate the objects in the view of the camera. The analysis will provide haptic cues to the user indicate the position and number of the object in the view of the camera through the use of buzzers in a haptic glove of other haptic interface device. Using this haptic information, the user will determine the position of the object and more directly reach for and grab the device. The user will be able to sense the position of the device by aiming the camera at the object and triangulating its position where after the user will more directly reach for the object. The triangulation of the object will be accomplished by moving the camera in a user determined number.
Project Contact: Michael Rush
Project Members: Sethuraman Panchanathan, Michael Rush
Visio-Haptic Perception
Research Areas: Haptics
Project Description:Humans have the uncanny ability to estimate how an object feels in terms of its shape, size, texture, material, etc., entirely from its visual image. From a biological standpoint, algorithms that estimate haptic features from images mimic the human ability to transfer knowledge from one perceptual modality (vision) to another perceptual modality (touch). These algorithms are known as visio-haptic transfer algorithms, which can estimate haptic information from visual data at either a physical level or a perceptual level. Physical visio-haptic transfer algorithms perform 3D reconstruction to create virtual, tangible models of environments or objects, which may be haptically explored by users. On the other hand, perceptual visio-haptic transfer algorithms classify haptic features, e.g., texture, shape, etc., of objects or surfaces into pre-determined perceptual categories, e.g., a surface’s texture may be classified as smooth, rough, bumpy, etc. Hence, perceptual visio-haptic transfer algorithms provide information about objects or surfaces at a conceptual level to enable real-time user perception.
The aim of our research is to design a general framework for visio-haptic transfer, and design, develop and test physical and perceptual visio-haptic transfer algorithms. Our application area of interest is assistive technology for individuals who are blind or visually impaired, with the aim of developing a wearable assistive device for remote object perception for individuals who are blind.
Project Contact: Sethuraman Panchanathan
Project Members: Troy McDaniel, Sethuraman Panchanathan
Funding Sources: CUbiC Technology and Research Initiative Fund