Visual Attention Research
composed by Zenon Pylyshyn (Center for Cognitive Science and Department of Psychology)
Acton, B. (1993). A network model of indexing and attention. M.A.Sc. Dissertation, Dept of Electrical Engineering, University of Western Ontario.
Acton B. & Eagleson, R. (1993). A neural network model of spatial indexing. Investigative Ophthalmology and Visual Science, 34, 413 (abstract).
Avant, L.L. (1965). Vision in the Ganzfeld. Psychological Bulletin, 64, 246-258.
Ballard, D.H., Hayhoe, M.M., & Pook, P.K. (1995). Deictic codes for the embodiment of cognition. Behavioral and Brian Sciences, (in press).
Ballard, D.H., & Rao, R.P.N. (1994). Seeing behind occlusions. In Proceedings of the Third European Conference on Computer Vision (ECCV), Stockholm, Sweden, p. 274-285.
Bridgeman, B., Van der Heijden, A.H.C. & Velichkovsky, B.M. (1994). A theory of visual stability across saccadic eye movements. Behavioral and Brain Sciences, 17(2), 247-292.
Burkell, J. (1988) Is colour change a primitive visual feature? MA Thesis, Dept of Psychology, University of Western Ontario.
Burkell, J.A., and Pylyshyn, Z.W. (1995). Searching through selected subsets of visual displays: A test of the FINST indexing Hypothesis. Submitted to Cognitive Psychology.
Cavanagh, P. (1990) Pursuing moving objects with attention. Proc 12th Annual Meeting of the Cognitive Science Society, Boston (P1046-1047). Hillsdale, NJ Erlbaum.
Dawson, M.R.W., and Pylyshyn, Z.W. (1989). Natural constraints in apparent motion. In Z.W. Pylyshyn (Ed.), Computational Processes in Human Vision: An interdisciplinary perspective. Norwood: Ablex Publishers.
Eagleson, R. (1991). Group-Theoretic Motion Analysis for the Coordination of Robotic Movement. In Active Perception and Robot Vision. A. Sood, editor. Springer NATO ASI Series, August 1991.
Eagleson, R., and Pylyshyn, Z.W. (1988). A computational model of a 2D (`FINST') tracking mechanism using spatio-temporal operators and a predictive filter. University of Western Ontario, Centre for Cognitive Science, Technical Report No. 38.
Eagleson, R, & Pylyshyn, Z.W. (1991). The Role of Indexing and Tracking in Visual Motion Perception. Conference on Spatial Vision in Humans and Robots, York University, June 19-22, 1991.
Engelken Edward J, Stevens Kennith W. (1989). Saccadic eye movements in response to visual auditory and bisensory stimuli. Aviation Space and Environmental Medicine V60(8) 762-768.
Enns, J., & Rensink, R. (1991). Preattentive recovery of three dimensional orientation from line drawings. Psychological Review, 198, 335 - 351.
Eriksen, C., & St. James, J. (1986). Visual attention within and around the field of focal attention: A zoom lens model. Perception and Psychophysics, 40(4), 225-240.
Fisher, B.D., and Pylyshyn, Z.W. (1994). The Cognitive architecture of bimodal event perception: a commentary and addendum to Radeau (1994). Current Psychology of Cognition, 13(1), 92-96.
Fodor, J., and Pylyshyn, Z.W. (1988). Connectionism and cognitive architecture: A critical analysis. Cognition, 28, 3-71.
Goodale, M.A. (1988). Modularity in visuomotor control: From input to output. In Z.W. Pylyshyn (Ed.), Computational Processes in Human Vision: An interdisciplinary perspective. Norwood: N.J., Ablex.
Goodale, M.A. & Milner, A.D. (1992). Separate visual pathways for perception and action. Trends in Neuroscience, 15, 20-25.
Hikosaka, O., Miyauchi, S. & Shimojo, S. (1993a). Focal visual attention produces illusory temporal order and motion sensation. Vision Research, 33. 1219-1240.
Hikosaka, O., Miyauchi, S. & Shimojo, S. (1993b). Voluntary and stimulus-induced attention detected as motion sensation. Perception, 22, 517-526.
Howard, I.P. (1982). Human Visual Orientation. London: Wiley.
Intraub, H. Mangels, J., and Bender, R. (1992). Looking at pictures but remembering scenes. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18, 180-191.
Hughes, H.C., & Reuter-Lorenz P.A. (1994). Visual-auditory interactions in sensorimotor processing: Saccades versus manual responses. Journal of Experimental Psychology: Human Perception and Performance, 20(1) 131-153
Intriligator, J. & Cavanagh, P. (1992). An object-specific spatial attentional facilitation that does not travel to adjacent spatial locations. Investigative Ophthalmology and Visual Science, 33, 2849 (abstract).
Irwin, D.E. (1993). Perceiving an integrated visual world. In D.E. Meyer & S. Kornblum (Eds.), Attention and Performance XIV, (pp 121-143). Cambridge, MA: MIT Press.
Irwin, D.E. (1995). Properties of Transsaccadic Memory: Implications for Scene Perception. Talk presented at Cambridge Basic Research. June 26, 1995 (abstract).
Irwin, D.E., McConkie, G.W., Carlson-Radvansky, L.A., & Currie, C. (1994), A localist evaluation solution for visual stability across saccades. Behavioral and Brain Sciences, 17, 265-266.
Jolicoeur, P. (1988). Curve tracing operations and the perception of spatial relations. In Z.W. Pylyshyn (Ed.), Computational Processes in Human Vision: An Interdisciplinary Perspective. Norwood, N.J.: Ablex Publishing.
Jonides, J. (1981). Voluntary versus automatic control over the mind's eye movement. In J.B. Long and A.D. Baddely (Eds.), Attention and Performance IX (pp. 187-203). Hillsdale, N.J.: Erlbaum.
Perrott, David R.; Saberi, Kourosh; Brown, Kathleen; Strybel, Thomas Z. (1990) Auditory psychomotor coordination and visual search performance. Perception and Psychophysics, 48(3), 214-226
Pollatsek, A., & Rayner, K. (1992). What is integrated across fixations? In K. Rayner (Ed.), Eye Movements and visual cognition: Scene perception and reading. New York: Springer- Verlag.
Pylyshyn, Z. W. (1998). Visual indexes in spatial vision and imagery. Visual Attention. R. D. Wright. New York, Oxford University Press: 215-231.
Pylyshyn, Z. W. (1999). “Is vision continuous with cognition? The case for cognitive impenetrability of visual perception.” Behavioral and Brain Sciences 22(3): 341-423.
Pylyshyn, Z. W. (2000). “Situating vision in the world.” Trends in Cognitive Sciences 4(5): 197-207.
Pylyshyn, Z. W. (in press). “Visual indexes, preconceptual objects, and situated vision.” Cognition.
Pylyshyn, Z. W. (in press). Is the imagery debate over? If so what was it about? In Cognition: a critical look. (Ed) E. Dupoux. Cambridge, MA, MIT Press.
Pylyshyn, Z. W. (under review). “Mental Imagery: In search of a theory.” Behavioral and Brain Sciences.
Pylyshyn, Z. W. and J. Cohen (1999). Imagined extrapolation of uniform motion is not continuous. Annual Conference of the Association for Research in Vision and Ophthalmology, Ft. Lauderdale, FL, Investigative Opthalmology and Visual Science.
Pylyshyn, Z.W. (1994). Some primitive mechanisms of spatial attention. Cognition, 50, 363-384.
Pylyshyn, Z.W. (1991). The role of cognitive architecture in theories of cognition. In K. VanLehn (Ed.), Architectures for Intelligence. Hillsdale: Lawrence Erlbaum Associates Inc.
Pylyshyn, Z.W. (1990). Computing in Cognitive Science. In M.I. Posner (Ed.), Foundations of Cognitive Science, Cambridge, MA.: MIT Press.
Pylyshyn, Z.W. (1989). The role of location indexes in spatial perception: A sketch of the FINST spatial-index model. Cognition, 32, 65-97.
Pylyshyn, Z.W. (1988). "Here" and "There" in the visual field. In Z.W. Pylyshyn (Ed.), Computational Processes in Human Vision: An Interdisciplinary Perspective. Norwood, N.J.: Ablex Publishing.
Pylyshyn, Z.W. (1981a). The imagery debate: Analogue media versus tacit knowledge. Psychological Review, 88, 16-45.
Pylyshyn, Z.W. (1981b). Psychological explanation and knowledge- dependent processes. Cognition, 10, 267-274.
Pylyshyn, Z., Burkell, J., Fisher, B., Sears, C., Schmidt, W. & Trick, L. (1994). Multiple parallel access in visual attention. Canadian Journal of Experimental Psychology, 48(2), 260-283.
Pylyshyn, Z.W., & Storm, R.W. (1988). Tracking multiple independent targets: Evidence for a parallel tracking mechanism. Spatial Vision. 3(3), 1-19.
Radeau, M. (1994). “Auditory-visual spatial interaction and modularity.” Cahiers de Psychologie Cognitive / Current Psychology of Cognition 13(1): 3-51.
Rensink, R. A. (2000). “The dynamic representation of scenes.” Visual Cognition 7: 17-42.
Rensink, R. A., J. K. O'Regan, et al. (1997). “To see or not to see: The need for attention to perceive changes in scenes.” Psychological Science 8(5): 368-373.
Rensink, R. A., J. K. O'Regan, et al. (2000). “On the failure to detect changes in scenes across brief interruptions.” Visual Cognition 7: 127-145.
Rock, I. (1983). The Logic of Perception. Cambridge, Mass., MIT Press, a Bradford Book.
Rock, I., Ed. (1997). Indirect Perception. Cambridge, MA, MIT Press.
Scholl, B. J. (in press). “Objects and attention: The state of the art.” Cognition.
Scholl, B. J. and A. M. Leslie (in press). Explaining infant's object concept: Beyond the perception/cognition dichotomy. Rutgers University Lectures in Cognitive Science. E. Lepore and Z. W. Pylyshyn. Oxford, Blackwell.
Scholl, B. J. and Z. W. Pylyshyn (1999). “Tracking multiple items through occlusion: Clues to visual objecthood.” Cognitive Psychology 38(2): 259-290.
Scholl, B. J., Z. W. Pylyshyn, et al. (submitted). “What is a visual object: Evidence from multiple-object tracking.” Cognition this issue.
Scholl, B. J., Z. W. Pylyshyn, et al. (1999). When are featural and spatiotemporal properties encoded as a result of attention allocation? (abstract). Investigative Ophthalmology and Visual Science, 39(4), S872, Ft. Lauderdale, FL, Investigative Ophthalmology and Visual Science.
Scholl, B. J., Z. W. Pylyshyn, et al. (1999). “When are featural and spatiotemporal properties encoded as a result of attentional allocation?” Investigative Ophthalmology & Visual Science 40(4): 4195.
Scholl, B. J., Z. W. Pylyshyn, et al. (submitted). “The relationship between property-encoding and object-based attention: Evidence from multiple-object tracking.” .
Sears, C. R. and Z. W. Pylyshyn (2000). “Multiple object tracking and attentional processes.” Canadian Journal of Experimental Psychology 54(1): 1-14.
Rock, I. (1983). The logic of perception. Cambridge, MA: MIT Press.
Schmidt, W. C., Fisher, B. D., & Pylyshyn, Z. W. (1995). Multiple onset stimuli elicit illusory line motion. Submitted to the Journal of Experimental Psychology: Human Perception and Performance.
Schmidt, W. C., & Pylyshyn, Z. W. (1993). An investigation of the role of auditory information in the smooth occulomotor system. Centre for Cognitive Science, University of Western Ontario, Technical Report: Cogmem 68.
Schmidt, W. C. (1995). Inhibition of return is not detected using illusory line motion. Investigative Ophthalmology and Visual Science, 36(4), S373.
Sears, C. (1991). Spatial Indexing and Information Processing at Multiple Locations in the Visual Field. MA Dissertation, Dept of Psychology, University of Western Ontario, London, Canada.
Sears, C. (1995). Inhibition of return of visual attention and visual indexing. PhD Dissertation, Department of Psychology, University of Western Ontario.
Sears, C., Schmidt, W.C. & Pylyshyn, Z.W. (1993). Negative priming in a localization task: Inhibition of distractor objects or distractor locations? Paper presented at the annual meeting of the Brain, Behavior and Cognition Society, Toronto, Ontario.
Sears, C. & Pylyshyn, Z.W. (1992). Multiple object tracking and visual sensitivity. Paper presented at the annual meeting of the Canadian Society for Brain, Behaviour, and Cognitive Science, Laval, Quebec. (Also University of Western Ontario Technical Report Series, Cogmem #67, 1993)
Sears, C.R.& Pylyshyn, Z.W. (1995). Multiple object tracking and attentional processing. Accepted for publication in the Canadian Journal of Experimental Psychology.
Simon, T. & Vaishnavi, S. (1995). In afterimages, five is too many to count: Implications for the role of object individuation in visual enumeration. Submitted to Perception and Psychophysics.
Simons, D.J. (1995). In Sight, Out of Mind: Noticing Changes to Scenes. Talk presented at Cambridge Basic Research. June 26, 1995 (abstract).
Treisman, A. (1986). Features and objects in visual processing. Scientific American, 255(5), 114-125.
Treisman, A. (1988). Features and objects. Quarterly Journal of Experimental Psychology, 40A, 201-237.
Trick, L. (1991). Three theories of enumeration that won't work and why, and then one that will: Subitizing, counting and spatial attention. In Nature and Origins of Mathematical Abilities, J. Campbell (ed). Elsevier Press.
Trick, L.M., and Pylyshyn, Z.W. (1993). What enumeration studies can show us about spatial attention: Evidence for limited capacity preattentive processing. Journal of Experimental Psychology: Human Perception and Performance, 19(2), 331-351.
Trick, L. M., and Pylyshyn, Z.W. (1994a). Why are small and large numbers enumerated differently? A limited capacity preattentive stage in vision. Psychological Review, 101(1), 1-23.
Trick, L., & Pylyshyn, Z. (1994b). Cueing and counting: Does the position of the attentional focus affect enumeration? Visual Cognition, 1(1), 67-100.
Ullman, S. (1984). Visual routines. Cognition, 18, 97-159.
Wright, R.D. (1994). Shifts of visual attention to multiple simultaneous location cues. Canadian Journal of Experimental Psychology, 48, 205-217.
Wright, R.D., and Dawson, M.R.W. (1987). Determinants of the speed of perceiving inside/outside spatial relations. Paper presented at the annual meeting of the Canadian Psychological Association. (Abstract published in Canadian Psychology, 28(2a), 679)
Wright, R.D., and Pylyshyn, Z.W. (1988). Effects of figural size on the perception of inside/outside spatial relations. Paper presented at the annual meeting of the Canadian Psychological Association. (Abstract published in Canadian Psychology,
Wynn, K. (1992). Addition and subtraction by human infants. Nature, 358, 749-750.
Yantis, S., & Jonides, J. (1984). Abrupt visual onsets and selective attention: Evidence from visual search. Journal of Experimental Psychology: Human Perception and Performance, 10, 601-621.
Yantis, S. (1992) Multielement visual tracking: Attention and perceptual organization. Cognitive Psychology, 24, 295-340.
Research Intern's Manual
The Visual Attention Laboratory (VAL) conducts experimental and theoretical investigations in order to better understand some of the bottlenecks in human visual information processing, especially as these pertain to people’s ability to visually attend to several things at once. The theoretical perspective behind this work is called the Visual Indexing Theory. This theory has broad application to many different phenomena involving visual perception and mental imagery.
This internship program is conducted under the supervision of Professor Zenon Pylyshyn.
General topic of the research
The experiments currently being carried out in this laboratory investigate the nature of visual attention and study people’s ability to split their visual attention among several objects or locations. One of the primary techniques used in this laboratory is called Multiple Object Tracking or MOT, a procedure which requires subjects to track several objects (the Targets) displayed on a screen which move randomly and independently among a set of identical moving Nontargets that must be ignored. Using MOT, we have shown that people can normally keep track of about four or five moving objects, even when they are mixed in with four other identical moving objects. This technique has proven useful for exploring a range of questions concerning human visual information processing. Over 20 papers have been published using variants of this method. In the present series of studies, we will examine certain factors that affect this ability. In particular, we will test a number of ideas concerning what limits the number of objects that can be tracked (to about 4 or 5).
Objectives of the Internship Program
The research internship program was designed to familiarize students with the steps involved in developing a research project, including:
- Understanding the initial motivating ideas behind the research.
- Suggesting a rough design for an experiment.
- Taking part in the preparation of stimulus materials.
- Participating in the design and execution of pilot studies.
- Participating in the preparation of a draft design of a full study.
- Participating in the process of tuning various parameters of the experiment by trying it out on themselves and other interns.
- Participate in the execution of the experiment.
- Participate in the analysis of data, which may lead to further studies.
To familiarize students with methods used to study human information processing – including the use of animated sequences, masking, or priming, and the measurement of reaction time, error rates, and other measures of skilled performance. The use of appropriate control conditions and baselines measures.
To familiarize students with the problems of discovering patterns in the data. This will involve learning about various methods of data summarization and statistical analysis tools. The importance of interaction effects and methods of stage analysis in testing theories. Guarding against speed-accuracy tradeoffs, response biases.
To provide practical experience in carrying out research projects, analyzing data, and writing up and presenting results in meetings (including experience in using specialized tools at each stage of this process).
Steps towards meeting these objectives
- In order to benefit fully from the training opportunities that you will receive here in the lab, we require that all Rutgers University students make a commitment of one full academic year.
- Research Interns (RI) will be trained in the use of laboratory techniques and methods. These methods and techniques will be used for studies that are currently fully designed and running, and for future studies that RIs are encouraged to develop on their own.
- We also provide short courses on Excel, SPSS, PowerPoint, VisionShell, and the use of library resources.
- Interns are expected to attend regular lab meetings, and to participate fully in the discussion, and to present at least one research article.
Tour of the Visual Attention Lab
The Visual Attention Laboratory is located in the annex section of the Psychology building, room A132 on the Busch campus. Room A132 is where we have many informal laboratory meetings, prepare experiments, read articles, meet subjects, etc.
Some notes about the Lab
- Where do things happen? – Experiments are run in room A132A, A136, and A130. Weekly meetings are held in the RUCCS playroom (A139) or the smaller LVR meeting room (A114). A map is reproduced in Appendix A and is available on line at: http://zeus.rutgers.edu/~feher/lvrmap1/lvr_ruccs.html
- Work Stations – There are currently 10 computers for running experiments, analyzing data, writing reports, and exchanging emails. Four of these will be assigned to Research Interns. See Appendix B for a full description of equipment available for RIs.
- Reprint Box – Reprints of articles relevant to research that is going on in the lab will be placed in the reprint box.
- Cork Board – The corkboard contains all RI schedules, schedules for general psychology classes where subjects can be solicited (with permission from the professors in charge); subject sign-up sheets, directions for current experiments, and other pertinent information.
- Laboratory web site is at: http://ruccs.rutgers.edu/val-home-page/
- Recruiting and running subjects
- Putting up signs, visiting classes with sign up forms (after obtaining permission from the professors).
- Working with subjects
- Informed consent documentation
- Talking with the subjects, answering questions, and formal feedback.
- Heading subject payments when appropriate
Attending lab meetings
- General lab meetings are held weekly. Individual meetings between people involved in particular projects and the PI are held weekly on a different day from the general lab meetings. All members of the laboratory, including staff and interns, are expected to attend all schedule meetings.
- Presentations at general lab meetings. Each lab member is expected to present a summary of a relevant research article at least once per semester. The presentation will summarize the main points of the article and its relevance to the work the student is conducting in the laboratory. Other lab members will also give periodic reports on the research projects on which they are working. Presentations will be done in PowerPoint
Readings and library research
- Principal and secondary readings will be provided at the start of the Internship. The principal readings are general articles on the theme of the laboratory’s work and students are expected to have read them within a few weeks of starting their internship. Secondary readings will be placed in your mailbox as other lab members find articles that are relevant to the lab’s research interests.
- Along with the resources of the Rutgers library, the secondary readings provide references for the student’s presentation and paper. Students are expected to become proficient in the use of the various library resources, including on-line materials.
Acquiring research related skills
- Students are expected to acquire certain research-related skills and in some cases will be aided in this process by lectures given by staff and by the PI.
- These include an appreciating the ethical issues in human research. Anyone who supervises the running of an experiment on human subjects is requiredto pass the Human Subject Certification Program, an online course required of individuals running experiments involving human subjects. This is a requirement imposed by NIH and by the Rutgers Internal Review Board.
- Students will become familiar with a number of tools used in data collection and analysis, sufficient for using these tools (Though not necessarily for programming new experiments or analyses from scratch).
- These may include:
- The use of computer based experiment- running software, such as VisionShell, E-Prime, or Presentation. Optionally, students may also become familiar with the use of the ISCAN eye movement tracking equipment.
- First-level knowledge of the use of tools for data-summarization, graphing, analysis, and presentation. These may include Excel, SPSS, PowerPoint, and other graphics software.
Other Things that you might be asked to do
From time to time, as time permits, interns may also be asked to carry out some additional work for the laboratory, such as:
- Help maintain our database for research articles, including making Xerox copies of articles
- Organizing a VAL library
- Helping to maintain our web page
Of course, interns are free to come into the lab as often as they wish.
Gathering data on human subjects is a prime function of the laboratory. If you have made prior arrangements to run subjects, it is your responsibility to try to arrange for other members of the lab to take over for you in your absence.
- If you plan to get to lab early in the morning, staying late at night, or come in on the weekends then you will need outside door keys, and lab keys. If this applies to you please discuss this with Amir ASAP.
- Please keep the doors to the lab locked at all times when there is nobody in – this is an essential security measure (computers have been known to disappear in the past).
- If you are locked out of VAL and you cannot find an RA, a Post-Doc, or Zenon, please ask either Sue Consentino in A133 or Jo'Ann Meli in A129 to let you into the lab.
Evaluation and Credit
Evaluation of student’s work as an intern
- Everyone intern is expected to take an active part in the regular lab meetings and to present his or her ideas on ongoing research projects.
- Interns will present an article at lab meetings.
- Submit a final report that would form the basis for a paper or a poster or talk submitted to a national conference, such as the Cognitive Science Society.
- Evaluation via the Research Assistant Evaluation form (See Appendix C).
Getting Academic Credit for working in the lab
- Research in Psychology course credit – get forms from Zenon or Sue Cosentino.
- Honors Thesis in Psychology
- Research in Cognitive Science course credit.
- Minor in Cognitive Science research requirement.
- Grading for Research in the Lab will be based on attendance, performance of assigned duties, participation in meetings, the Research Intern Evaluation form and the final report. The grading will be submitted as required by the university and will appear on letters of reference.
- To obtain credit for a full Internship a student must spend approximately 120 hours in the laboratory over the summer and/or school term. Shorter periods can also be accommodated but, depending on departmental requirements, may not earn a research 3 course credit, unless additional work is done.
Equipment available to RI’s
The lab currently has 11 computers:
- Dell PC computers, (at least 833 Mhz processor, 256 of RAM, 30 GB hard drive, and a 64 MP video card), with Windows 2000 OS, with Office 2000
- 3 older PCs (at least 233 Mhz processor, 16 MB RAM, and 2 GB hard drive), with either Windows 98 or Windows 2000, with Office 98 or Office 2000 respectively.
- 2 G4 Macs with 466 Mhz processor, 32 MB of RAM, 4 GB hard drive, and a 2MB video card, with Mac OS 9.0.4, with Office 2000.
- 1 Power PC Mac and 1 Quadra Mac.
- ISCAN Eye-tracking equipment, both head-mounted and table-mounted. All eye-tracking equipment is located in A130 and used jointly with the human-computer interaction laboratory (“The Village”). We are currently working on creating software to integrate the eye-tracking equipment with the experiment-running that we are using in the lab.
- All of the programming with the Macs is done with VisionShell PPC 1.0, and the C programming is done with Code Warrior 6.0.
- On the PC’s we are using E-Prime, Presentation, or Matlab with Psychophysical Toolbox
First level data analysis, i.e., averages, t-tests, standard deviations are done with Excel. Full analysis of variance is done with SPSS. Interns need to know how to use Excel to summarize data and to construct charts and graphs.
We are using Dreamweaver 4
Telelearning and Teleconferencing
Zenon Pylyshyn (Center for Cognitive Science and Department of Psychology)
Applying research on spatial organization to problems of cooperative work, telelearning and communication of knowledge.
Background: Psychological use of physical and mental space
A major cognitive framework for individuating, visualizing, and keeping track of different items of knowledge (such as who said what in a conference or what items of data go with what) is the use of real 3D spatial locations. We use space both literally (as in the desktop or office model of data organization) and also figuratively. Examples of the latter includes such techniques as mentally locating different facts and premises in certain imagined spatial loci -- a technique widely used in mnemonic aids, and the use of spatial location in reasoning where so-called "spatial paralogic" provides an important scheme for keeping track of different components of a problem. The use of distinct spatial loci in reasoning and visualizing can be enhanced and its effectiveness in communication increased if distinct spatial locations can be shared. This, of course, happens routinely when people use gestures, pointing, and carving shapes in the air when they converse. Sharing a common workspace and conceptual space is now becoming technologically feasible through the use of interactive multimedia workstations, in which sound and 3D visual locations can be communicated and spatial indicators such as pointing in space can form part of the human-computer interaction.
There is already considerable psychophysical evidence concerning the interaction of sound and vision in localization and identification as well as the interaction of these modalities with motor movements. However this evidence has not been brought to bear in the design of multimedia (or telepresence) workstations because such interactions between users and computers have heretofore not been technically feasible.
Although in many places high bandwidth links make it possible to use high-immersion displays, the introduction of space-based communication-enhancing information will have to be incremental and deal initially with what can be done with current low bandwidth multimodal communications networks to enhance communication in education, training, and cooperative work. For example, partly because of bandwidth limitations and the delays inherent in packet-switching technology, current teleconferencing and cooperative work systems designed to operate over digital networks (such as those supplied commercially) are extremely primitive and provide only very impoverished information about location of speakers and their gestures. This sort of information, however, could usefully be augmented by recognizing and highlighting which person is speaking (using sound-localization and speaker identification techniques already available) and even by automatically zooming in and providing additional bandwidth in the region of the speaker.
Initially the proposed research will deal with such questions as what spatial information (e.g. gestures, location of speakers, location of objects referred to, etc) is most important for communications and how to optimally encode such information for initially low-bandwidth channels. The recognition of gestures and their encoding and transmission represents a longer-term research program. However, the work that has already been carried out in the design of certain computerized choreography tools such as the LifeForms system at Simon Fraser University is an excellent step towards this goal since it provides both a compact model of the human form (for use in automatic recognition body orientation and for parameterizing this information for compact transmission) and also a compact representation of human movements and gestures. Used in conjunction with motion-capture techniques it could provide compact transmission of both gestures and directional information within a shared 3D space.
Benefits of this line of research
In addition to providing design principles for workstations and cooperative work/learning environments, this line of exploration feeds into the following application areas:
1) Communication and learning aids for the disabled
We know that the use of space and spatial metaphors in communication is universal (Lakoff & Johnston). It is also well-known that spatial location plays a major role in communication for the deaf, as is shown by the way sign languages deal with pronouns and anaphora. It is also a major framework by which blind people not only navigate but also remember and model their environment. Although congenitally blind persons can't be said to have "visual images" in the usual sense, they do have a very well developed representation of space that serves a very similar function. Indeed it is well-documented (Golden-Meadow & Gleitman) that children blind at birth use spatial terms exactly the way sighted children do, though without ever having experienced the referents visually.
2) Communication, training, education and work in distant places
The ability to show and manipulate objects, to use communicative gestures as well as words, to refer and point to places in a room where ideas and blackboards are located, and in general to utilize the spatial framework of a common room, is important to intimate working relations. But it is even more important when a common language and linguistic culture do not naturally exist. It is also important for establishing the sense of personal interaction that is eroded by long distance unimodal communication. Studies of such communication media as email and comparisons of voice and visual contact show that simply increasing bandwidth is not always the answer to greater apparent "intimacy" and that different media are best suited for different kinds of interactions. However for the kind of learning experiences that, say, children respond best to -- the most intimate and close connections between teacher and pupil is essential.
3) Closely related to telelearning is cooperative work-at-a-distance
Although teleconferencing has been a moderately active subject of research, the logical extension of this idea to task-oriented workshops which use multi-media and shared workspaces is in its infancy. Cooperative interactive work is becoming more and more important, especially where the workforce is widely dispersed. The more complex the combination of skills required for some piece of work, such as designing some artifact, the more important does the technology of multiple-authoring and multiple-person designing become. Design of such artifacts as novel computers requires the cooperative effort of many fields of expertise, as does the production of a multi-authored document. When people are present in the same room and work on a design together they not only talk vigorously, but they sketch (perhaps using a computer console and electronic pointing device), walk about the room, jot down notes on different boards, point to previous sketches, and may even physically examine a partially completed or roughly approximated three-dimensional object. None of these are available in tele-operation mode. However some of these interactions can in principle now be realized in high-immersion interactive workstations with a virtual shared environment. It is possible to provide several screens situated around a room, to encode and transmit gestures and sounds so as to preserve their 3-D locations and directions. The use of spatial location in this way would not only enhance communication, but would serve to distinguish and index different types of information. Participants could refer to different aspects of their discussion the way blind people sometimes do; by associating different ideas or aspects of the joint work with places in a common workspace.
Undergraduate Research Internship Program
General topic of the research
This research studies the nature of visual attention and assesses people's ability to split their visual attention and to track multiple independently moving objects, displayed on a screen.In this laboratory we have shown that people can normally track 4 moving objects even when they are mixed in with 4 other identical moving objects that they are to ignore.The basicMultiple Object Tracking(MOT) technique has proved useful for exploring a range of questions concerning human visual information processing.Over 20 papers have been published using this paradigm.In the present series of studies we will examine certain factors that affect this ability.In particular we will test a number of ideas concerning what limits the number of objects that can be tracked (to about 4 or 5).
Objective of the Internship
- To familiarize students with the steps involved in developing a research project, from initial motivating ideas to a rough design, design of materials, pilot studies, draft final design, tuning of parameters, and execution of the experiment.
- To familiarize students with methods used to study human information processing – including the use of reaction time and error measures.
- To familiarize students with the problems of discovering patterns in the data.This will involve learning about the various data summarization and statistical analysis tools of human research.
- To provide practical experience in carrying out research projects, analyzing data, and writing up and presenting results in meetings (including experience in using specialized tools at each stage of this process).
Steps toward achieving these objectives
The Internships involve training in the use of laboratory techniques and methods as well as practice in the use of these techniques in the course of helping to run an already designed study as well as to design and execute at least one original experiment using the computational tools already developed in this laboratory.
Evaluation of student’s work as an intern
Every intern is expected to take an active part in the regular lab meetings and to present his or her ideas on ongoing research projects. Interns are also required to submit a final report that would form the basis for a paper or a poster or talk submitted to a national conference, such as the cognitive Science Society.Evaluation of this work will be included in any letters of reference requested of the laboratory head.
- Recruiting and running subjects
- Putting up signs, visiting classes with sign up forms.
- Dealing with subjects
- Informed consent forms
- Talking with the subjects and answering questions.Formal feedback.
- Handling subject payments when appropriate
- Attending Lab meetings
- General lab meetings are held weekly.Individual meetings between people involved in particular projects and the principle investigator are held weekly on a different day from the general lab meetings.All members of the laboratory, including staff and interns, are expected to attend all scheduled meetings.
- Presentations at general lab meetings.Each lab member is expected to present a summary of a relevant research article at least once per semester.The presentation will summarize the main points of the article and its relevance to the work the student is conducting in the laboratory.Lab members are also expected to give periodic brief reports on the project on which they are working.
- Readings and library research
A set of principle and secondary readings are provided at the start of the Internship.The principle readings are general articles on the theme of the laboratory’s work and students are expected to have read them within a few weeks of starting their internship.Along with the resources of the Rutgers library, the secondary readings provide references for the student’s presentation and paper.Students are expected to become proficient in the use of the various library resources, including on-line materials.
- Acquiring research skills
Students are expected to acquire certain research-related skills and in some cases will be aided in this process by lectures given by staff and by the PI.These include:
- Appreciating the ethical issues in human research.All students are required to pass theHuman Subjects Certification Program, an on-line course required of individuals running experiments involving human subjects.This is a requirement imposed by NIH and by the Rutgers Internal Review Board.
- Students will become familiar with a number of computer tools used in data collection, sufficient for using these tools, though not necessarily for programming new original experiments.These will include:
- The use of computer based experiment-running software, such as VisionShell or rudiments of MatLab sufficient to appreciate how to explain requirements to our programmers and to make simple modifications to existing programs.
- Students may also need to become familiar with the use of the ISCAN eye movement tracking equipment.
- First-level knowledge of the use of tools for data-summarization, graphing, analysis, and presentation.These may including Excel, SPSS, PowerPoint and graphics software.
- Acquiring experience in analyzing data and presenting results
On arrival, each intern will be provided with a key, ID,photocopy account number, and a copy of the current Intern's Manual. The current manual is available at Current Intern's Manual.
VSS 2011 - Vision Sciences Society 2011 Conference Abstracts
Aks, Alley, Rathakrishnan, Kourtev, Haladjian, & Pylyshyn (2011). When vision loses its "grip" on tracked objects: Lessons from studying gaze-to-item dynamics. Vision Sciences Society 2011, Naples, FL.
We use a unique gaze-to-item analysis to study when vision "loses its grip" on tracked objects. Important insights can be gained by looking at spontaneous tracking failures and those that occur during uninterrupted vs. interrupted tracking (such as when we blink our eyes or objects overlap each other). We generate an explicit trace of eye-movement paths and each of the eight item positions recorded over the course of each of (138 - 5 sec) multiple object tracking (MOT) trials. Temporal profiles of scan- and item-paths, help identify sources of tracking failures obscured by the aggregated accuracy measures typically recorded at the end of each trial. We show tracking failures from object crowding, and subsequent gaze-switching from targets to non-target items. We also show how spontaneous switching across tracked objects is common, and does not impair tracking accuracy (See Elfanagely et al, VSS 2011). Finally, we show when object tracking is disrupted briefly (<1 sec), our gaze continues to remain close to those items tracked just prior to their disappearance (See Alley et al, VSS 2011). Because we have tested conditions where gaze and attention are correlated, scan path patterns are easily understood in terms of gaze and attentional indexing as two systems coordinating to effectively track objects.
Alley, Rathakrishnan, Harman, Kourtev, Kugal, Haladjian, Aks, & Pylyshyn (2011). Tracking objects and tracking our eyes during disrupted viewing. Vision Sciences Society 2011, Naples, FL.
We are studying how people track objects, and how eye-movements and attention contribute to this ability. We extend Keane and Pylyshyn (2006) and Aks, Pylyshyn, Haladjian et al., (2010), research on multiple object tracking (MOT) during disrupted viewing to learn whether the visual system encodes the position of tracked objects. Observers blinked their eyes when a brief tone was presented midway into each trial where they were tracking 4 of 8 identical items. Eye-blinks triggered item disappearance and the onset of a mask that blocked the display of items (for up to 1 second). During their disappearance, objects either continued moving, or halted until their reappearance. Better tracking occurred when items halted (or were displaced further back along their quasi-random motion trajectory) suggesting that the visual system refers back to past position samples to guide where tracked items are likely to reappear. In the current study, we explore the role of eye-movements in MOT. Our gaze-to-item analysis, described in Aks et al., VSS 2011, shows parallels between eye-movements and MOT performance. Gaze tends to remain near targets that were tracked just before the blink when objects disappeared. This gaze- to-item linkage was reliable across "halt" trials, highly idiosyncratic on "move" trials, and intermittent during the uninterrupted part of the tracking task. Switching gaze across targets, accounting for the intermittency, was surprisingly common and often spontaneous (see Elfanagely et al., VSS 2011). These results suggest that different eye-movement strategies can be used to maintain mental links to tracked objects.
Elfanagely, Haladjian, Aks, Kourtev, & Pylyshyn (2011). Eye-movement dynamics of object-tracking. Vision Sciences Society 2011, Naples, FL.
Tracking requires maintaining a link to individual objects as they move around. There is no need to maintain a record of object position over time; all that is needed is maintaining a connection, or index, to target items as they move (Pylyshyn, 2004) . Yet, how well we maintain links is undoubtedly reflected in tracking behaviors. Both the time course and pattern of eye-scanning used in multiple object tracking (MOT) may help us understand how humans track objects. By analyzing MOT dynamics, we explore why better tracking occurs when objects halt during their disappearance (Keane & Pylyshyn, 2006), and how the visual system maintains a memory of prior object-positions. We use the MOT task described in (Alley et al. 2011), and "gaze-to-item" analysis measuring relative distance between eye-positions and each of 8 changing item positions (4 are tracked targets). We also use Recurrence Quantification Analysis (RQA) to determine whether recurring eye-movement patterns play a role (Webber & Zbilut; 1994). How smooth and repetitive are gaze paths? Fehd & Seiffert (2008) report that gaze follows the center of a group of targets, and that this "centroid" strategy reflects tracking a global object formed by grouping. This leads to a prediction that such a "center-looking strategy" should be smooth since the centroid moves with the average instantaneous position of independently moving objects . However, among gaze dynamic patterns that we found, one surprising result is the pervasiveness of switching gaze across items. Such frequent switching occurs spontaneously, and under crowding conditions, and is consistent with the alternative indexing account that individuated objects are tracked separately. By focusing only on aggregated positions, we may be masking important dynamics. Perhaps most significant are recursive scan paths of which switching behavior is a critical component. This may reflect iterative coding for sequences of prior object positions.
Haladjian, Griffith, & Pylyshyn (2011). The attentional blink impairs localization but not enumeration performance in an "enumerating-by-pointing" task. Vision Sciences Society 2011, Naples, FL.
Earlier we reported (Haladjian & Pylyshyn, 2010) that observers are able to rapidly and accurately enumerate up to six items when using an "enumerating-by-pointing" method (compared with the typical subitizing limit of four). We have been exploring possible reasons for this increase. The present study examines the role of increased encoding time (without increasing actual viewing time) by testing whether two presentations of the stimulus separated by a variable interval improves enumeration performance. Additionally, this allowed us to test if the second presentation of the stimulus was sensitive to the attentional blink. Participants were shown masked displays that contained 2-9 randomly-placed black discs (~1° diameter) on a gray background. The stimulus was presented once for 100-ms or presented twice for 50-ms (each) with a delay of 200-, 400-, or 600-ms (ISI) between the mask offset and the second presentation onset. Participants then marked the locations of each disc using a computer mouse.
Trials with two separate 50-ms presentations showed better enumeration performance than trials with a single 100-ms presentation for numerosities >4; the delay conditions did not significantly differ from each other (except in 5-item displays). For localization performance, two-presentation trials produced more accurate responses than single-presentation trials for numerosities <7. Here, location accuracy was significantly better in the 600-ms delay condition for displays with 5-8 items. This suggests an additive benefit when presenting the second display outside of the attentional blink in trials where observers needed to enumerate >4 items. These results (that the attentional blink affects localization more than enumeration) suggest that attention is more critical for the encoding of location information than for enumerating small sets. These results also point to the possibility that the increased coding time associated with the mouse pointing (when marking object locations) may play some role in the increased subitizing limit. .
Harman, Haladjian, & Pylyshyn (2011). Eye movements during an enumerating-by-pointing task enhance spatial compression. Vision Sciences Society 2011, Naples, FL.
Observers can accurately enumerate and localize sets containing up to six randomly-placed dots when using an "enumerating-by-pointing" method (Haladjian & Pylyshyn, 2010). Analyses of localization errors suggest a form of compression, where location responses are closer to the centroid of the set of dots than their actual locations on the stimulus screen. We address the following questions in the current study: Is this compression stronger around the centroid of the dots or the point of central fixation? Is the frequency of fixations correlated with response accuracy? Is compression of pointing responses linked to eye-movements? We used an EyeLink 1000 eye-tracker to examine the role of eye-movements in our enumerating-by-pointing task. Participants were shown a display with 1-10 randomly-placed black dots (~1° diameter). This gaze-contingent display appeared immediately after participants fixated the center of the screen for one second. After a full-screen mask, participants used a mouse to place markers on a blank screen indicating the perceived locations of the dots. Analyses were performed on enumeration accuracy and localization errors (distance between dots and nearest response marker). Results show strong compression around the centroid of dots, and some compression around fixations (i.e., localization errors are smaller and less variable around the centroid). Stronger compression (on 2/3 of the cases) required at least one fixation to the centroid. More fixations, as well as dots, also strengthened centroid compression. Increased fixation frequency, however, did not improve localization or enumeration performance. Overall, these results suggest that compression is centered on the centroid of a set of stimuli, and eye-movements play a role in perceived shrinkage of the display configuration, but not judgments associated with counting.