Cognitive load modulates attentional capture by color singletons during effortful visual search☆

Cognitive load modulates attentional capture by color singletons during effortful visual search☆: "Publication year: 2010
Source: Acta Psychologica, In Press, Corrected Proof, Available online 31 May 2010
Bryan R., Burnham
Color singletons that are irrelevant to locating a visual target do not typically capture attention if visual search is effortful. In contrast, when search is efficient color singletons are often found to capture attention. Such distraction by a color singleton can be modulated by single-task vs. dual-task manipulations when visual search is efficient. This is due, presumably, to the increased cognitive load in the dual-task condition, which interferes with top-down attentional control. This study investigated whether capture by a color singleton is also modulated by single-task vs. dual-task manipulations when visual search was effortful. The results of three experiments revealed..."

import .edf files in batch into DataViewer

Dear SR Researchers,

I love SR Research Product, and benifits a lot from your product. But how I wish I could import the .edf files in batch, because it take me over one hour to import the expeirment data into dataviewer. It is time cost and boring to do this manual work. To save my time, and many other researchers like me,I wrote a simple software toimport .edf in a batch into DataViewer. This will save me lots of time. You can download the software at the following link.
http://cid-bbc15003189d7799.skydrive.live.com/browse.aspx/Public/BatchImporter

Hope you like the tool.


As a fan of SR Research products, I love their eye tracker very much. But how I wish I could import the .edf files in batch, because it take me over one hour to import the expeirment data into dataviewer. It is time cost and boring to do this manual work. To save my time, and many other researchers like me,I wrote this simple software. I have tested the program with DataViewer 1.8.1. I guess it will work with any version of DataViewer.

------------------------------

---------------------
Function:
---------------------------------------------------
- To import the .edf files into DataViewer in batch.
- to Save your time

---------------------------------------------------
How to use?
---------------------------------------------------
1. set up config.ini
put the dataviewer directory in the second line in the config.ini.
The default directory is "c:\Program Files\SR Research\DataViewer\DataViewerW.exe"

2. set edf data location
put the directory of the edf files in edfLocation.txt
Each directory takes one line. It does not require the directory to be the folder just containing .edf. It can be any folders as long as itself or its subfolders contain the .edf file.

3. Click BatchImporter.exe to run
Then the software will ask DataViewer to import all the edf files you set in edfLocation.txt


---------------------------------------------------
License
---------------------------------------------------

I wrote this tool to help myself and my coworkers. I am not affiliated with SR Research. Use the tool at your own risk, although I do not believe there will be any risk.It is a free software.

Eprime Learning materials

for those friends who are interested in e-prime programming. 

Getting Started Guide.pdf

GettingStartedGuide

Reference Guide.pdf

ReferenceGuide

Users Guide.pdf

UsersGuide

and also a useful site @ CMU

http://step.psy.cmu.edu/

SurfLogger: A Logging Browser and Data Processing Method

SurfLogger

SurfLogger , developed by He, Jibo at University of Illinois, is an automated data logging tool for web-based studies. It is written in python, free, open-source, cross-platform, and easy to modify. This page is devoted to information and resources about SurfLogger. Features: SurfLogger is a useful tool for collecting data for web-based researches. With its great features of automated data logging, free, open-source, cross-platform, and no dependence on other browsers, SurfLogger can free many researchers from the financial and time cost in data collecting. SurfLogger is expected to contribute more to the increasing interest in web-based researches.  Technical Specifics: SurfLogger is written in Python, a scripting language, and the GUI (Graphical User Interface) is created with wxPython, which is a Python bundle of wxWidget. SurfLogger can record a variety of user actions with the web pages and the browsers. SurfLogger produces two files, logfile.txt and urlfile.txt. Logfile.txt stores action IDs (natural numbers assigned to each action, used to track the record to the responding actions), the time for each actions, interaction with the browsers (such as, clicking on the Back, Forward,Home, etc. buttons), and mouse coordination when clicking. The time record could be used to compute the time of completion for each task. The number of button press on the browsers could be used as a measure of effort in carrying out the task. SurfLogger also captures the images of each screen when the web page refreshes. Marking the mouse coordination on the screen captures could tell us what links the users clicked at. Urlfile.txt stores action IDs and URLs (Uniform Resource Locator). Action IDs are used to synchronize the record in logfile.txt and urlfile.txt. URL record is stored in a separate file because the abundant information it can provides. I will give an example about how to extract information from urlfile.txt in case study section of this paper. SurfLogger also calls external software to record the whole process of user actions. Currently, I used Michael Urman’s Screen Recorder named cankiri as my external software for recording, because it is also written in python and shares the same spirit of   open source.  With video record, the researchers could know more about users’ actions. If quality of recording is emphasized, SurfLogger could easily switch to call other recording software, and only one line of the code has to change to refer to the path of the external software. Case Study To demonstrate how SurfLogger could benefit web-based research, I will briefly explain the usability analysis of IGroup as a case study (Wang, Jing, He, and Yang, 2007). IGroup is an image search engine, presenting the results in semantic clusters. To test whether IGroup can increase search efficiency compared to MSN, we developed the predecessor of SurfLogger, which functioned similarly like SurfLogger, but less flexible. We developed a measure of Search Effort to compare IGroup and MSN objectively. Search Effort was defined as the number of query input, and number of links and cluster names clicked by the users. Query input, links and cluster names clicked were extracted from URLs recorded by our automated logging tool. A sample URL recorded in this study was listed as follows: Wednesday, August 30, 2006 3:06:54 PM http://msra-vss50-b/igroup2/search.aspx?q=Disney#g,14,1,-1 The characters in bold, “Disney”, “14”, and “1” were the input query, ID of cluster name, and result page. The information could be extracted from the URL by simple text processing.  For code of data reduction, URL extraction and source code of SurfLogger please refer to my project page of SurfLogger. Resources: 1. Paper Potential users could get more information and description about SurfLogger from my draft paper [click to download] . This paper will be presented at SCiP (Society of Computer in Psychology)' 2008 at Chicago. 2. Codes nSource code qSurfLogger.pyw and run.py qDependencies: nPsychoPy http://www.psychopy.org/  nwxPython http://www.wxpython.org nOr download the dependency bundle at here nExecutable program qSurfLogger.exe 3. Data Processing Methods: To be added soon.  :-P Rights You are free to use this tool for non-commercial purpose.  Use by commercial companies or organizations must get permission from the author, He, Jibo.  Download nSource code: qSurfLogger.pyw and run.py qDependencies: nPsychoPy http://www.psychopy.org/  nwxPython http://www.wxpython.org nExecutable program qSurfLogger.exe q Contact Feel free to contact me for help and information. If you would like to help me with the development, do please let me know.  Department of Psychology, Beckman Institute for Advanced Science and Technology University of Illinois, Urbana Champaign, 603 East Daniel St., Champaign, IL 61820 Tel: 217-244-4461(office) 217-244-6763(lab) Email: hejibo@gmail.com  MSN: hejibopku@hotmail.com

####Matlab Programming Notes####

####Matlab Programming Notes####

 

by He, Jibo for Psych 593

Course Link:

http://www.psych.uiuc.edu/~alleras/classwelcome.htm

exam:

http://www.psych.uiuc.edu/~alleras/exam.htm

Resources for psychtoolbox

Psychtoolbox Wiki:

http://psychtoolbox.org/wikka.php?wakka=HomePage

 

Psychtoolbox 2.0

http://psychtoolbox.org/PTB-2/

 

Eyelink toolbox

http://www.psychtoolbox.org/wikka.php?wakka=EyelinkToolboxCredits

 

PTB tutorial @ Silver lab - University of California, Berkeley

http://argentum.ucbso.berkeley.edu/PTBtutorial

 

Matlab for the Behavioral Sciences: How to program your own experiment.

http://faculty.washington.edu/ionefine/MatlabCourseNotes08.html

 

An Introduction to Matlab

 http://www.maths.dundee.ac.uk/~ftp/na-reports/MatlabNotes.pdf

Matlab style guidelines

http://www.datatool.com/downloads/matlab_style_guidelines.pdf

 

Help for Psychtoolbox

• For beginners: Psychtoolbox Tutorial and Ione Fine's Psychtoolbox Tutorial.
  
• The Psychtoolbox includes built-in help available from the MATLAB command line.
  
  
  
  • Enter "help Psychtoolbox" at the MATLAB prompt for a list of Psychtoolbox function categories.
    
  • To display a list of Psychtoolbox functions within a category ask for help with the category, e.g. "help PsychBasic."
    
  • To get help for a Psychtoolbox function ask for help on the function, e.g. "help GetChar."
    
  • Psychtoolbox functions such as Screen,which accept a subcommand as an argument, include built-in help for there subcommands.
    
    
    
    • For a list of all subcommands issue the function call with no arguments, e.g. enter "Screen" at the command line.
      
    • To display documentation for a subcommand invoke the subcommand with a trailing question mark, e.g. Screen('OpenWindow?')
  
  
• Read our answers to frequently asked questions.
  
• Your friends and colleagues might help. Check out the Psychtoolbox forum.

Help for MATLAB

• Typing "doc" in MATLAB will activate their browser-based help system, which is quite handy.
  
• You can search the Mathworks web site.
  
• There's an active MATLAB newsgroup (mostly Windows and unix users).
  
• Typing "help" and "help help" at the MATLAB command line will list help topics and explain MATLAB help.

 

---

--2008/8/28--

--Lecture 1--

 

1. Monitor and graphics card (VRAM)

 

•2. Dynamic Random Access Memory (D)RAM

 

–# L1 cache - 10 nanoseconds, 4 kilobytes to 16 kilobytes in size

–# L2 cache - around 20 to 30 nanoseconds, 128 kilobytes to 512 kilobytes in size) (some L3)

–# Main memory - Memory access of type RAM (around 60 nanoseconds, 32 megabytes to Gigabytes in size)

3.  CRT Monitors and VGA:

 Red, Green, and Blue  Phosphor(磷光体;磷光剂) dots

 Analog ergo the video/graphics adapter

 

Dot pitch (distance of two adjacent same color ) and resolution

 The gun shots light on the monitor line by line.

 

4. LCD Monitors

•Digital

 

Notes: We can not make sure the exact time for stimuli on display and energy of the stimuli for LCD monitors.

 

5. Matlab

 

•MATrix LABoratory:
–High-level language •slow to interpret by machine •easy to understand by humans

• –I will teach you “syntax” •Algorithm based

• –Our vocabulary will be MATRICES.
 MATLAB is optimized for matrix-based calculations

 

 

 6.A few important commands

 

•SET PATH!

- Window or in command line:

 –Setting the path in your code: –

>p=path;

–>path(p,‘o:\MatlabCourse’);

 

•pwd: Where are you?

 

•cd: Change Directory

 

•dir: lists contents of folder

 

•mkdir: makes a new directory •

 

•who: lists of variables in workspace

 

•whos: who +  size of variables

 

•clear X: erases X

 

•clear all: erases all.

 

•help topic

•lookfor string  

 

7. Matrices

 

 First mentioned dimension is ALWAYS the number of rows
Second dimension = number of columns

 

•8. Name your matrices always lowercase  

 

---

--2008/9/4--

--Lecture 2--

 

 1. Attention: , and ; differs!!

>> e=['world','hello']

e =

worldhello

>> e=['world';'hello']

e =

world
hello

 

2.  mod(x,y): remainder of the division of x by y:

•mod(10,3) = 1

•mod(9,3) = 0

 • •SUPER USEFUL COMMAND (counterbalancing conditions)

 

floor.

Rounds towards minus infinity. •floor(3.4) = 3

ceil:

rounds towards plus infinity

round

rounds to nearest integer

fix

rounds towards zero

CLEAR ALL : clear the workspace

 

 3. Script file

> edit moo.m

 

•FOR EXAMPLE: UofI.m

 

•FOR EXAMPLE: UofI.m ØWRITE A SCRIPT TO CALCULATE THE FIRST 10 numbers in the sequence:
u(n) = u(n-1) x (n+1)
with u(1) = 1

 

---solution 1: u.m-----------

function

[u]=u(n)

u(1)=1;

for

i=2:n

u(i)=u(i-1)*(i+1);

end

 

---solution 2:uofi.m----

 

u=zeros(10,1);

u(1)=1;

for

n=2:10

u(n,1)=u(n-1,1).*(n+1);

end

u 

 

 It would be even nicer, if we asked the user the first value of the series. –

–Use ‘input’ – –Syntax: 
VAR = input(‘your text here’); – (EVALUATED INPUT)
–For entering strings of characters
VAR = input(‘Your text here’,’s’); – (NOT EVALUATED INPUT)

 

4. logic expression

 

LOGIC  p.8

•AND: &

•OR: |

•Complement: ~

•Exclusive OR:       xor (when a is true or b is true, but not true at the same time)

5. loop control

SWITCH:
syntax

 

•More generally, if testing the value inside variable 'n'

•switch n

• case n1

• …

• case n2

• …

• otherwise

• …

•end

 

 

HOMEWORK

 

---

--2008/9/11--

--Lecture 3--

 

disp('hello world')

% display a string or expression, equals print or printf in python or C

 

>> disp('type ''q''')        % double single quotes to input ' in the string
type 'q'

%Write a program that asks user for PIN number, until user gets it right.

correctPin = 1234;

true =1;

counter=0;

 

while true

 

guessPin=input('Please input your PIN,type q to quit:\n')

counter=counter+1;

 

if guessPin==correctPin

disp('you input the correct PIN\n')

break;

elseif guessPin=='q'

disp('you quited successfully\n')

break;

else

disp('you input the wrong PIN\n')

end

if counter==3

disp('you failed for three times!\n')

break

end

end

 

 

 

find

–FINDS a specific value in an array (or matrix) and returns the

–SYNTAX:
indeces = find(expression);

–returns indeces for which expression is true.

 

Lookfor:

to look for help with keywords

 

---

--2008/9/17--

--Lecture 4--

 

Functions

•- can take input (passing variables)

•- can return outputs.

-variables are internal.

-NAME OF M-FILE AND FUNCTION HAVE TO BE THE SAME.

------example of function:-----------

function output = nameoffunction(passvar1,passvar2…);

Persistent Variables

•If you want a function to keep accessing a variable, every time you run it, declare the variable as persistent.

•In other words, Matlab will NOT erase it upon exiting the function, so it will be able to access it again.

 

GLOBAL variables:

• -> can be used ANYWHERE…

• modified anywhere too!

•

•PERSISTENT variables:

• -> allow you to have some values persist in memory from one function use to the next.

• -> UNTOUCHABLES!! outside of function.

•

• -> beware of what's lurking under the surface.

 

For further reading…

•> anonymous functions (do not require an m-file)

• > f = @(arglist)expression

• > sqr = @(x) x.^2;

•> subfunctions are created within a function

•> private functions: only visible to their parent directory scripts.

•> nested functions: share variables.

 

SAVING AND READING MATRICES

•save NAME

–saves ALL the variables in your workplace, regardless of differences in format, on file NAME.mat

–Type:

–> clear all %CHECK WORKSPACE

– %CHECK CURRENT DIR.

–> load Worlds

save NAME var1 var2

–specifies which variables to save in file NAME.mat

•if you want to ADD stuff to a current mat file:

–save NAME var1 -append

wilcard *

•save avariables a* %saves all variables starting with a in file avariables.mat

 

reading text is different than reading numbers, but you can transform one into the other:

–num2str: transforms numbers into strings

–int2str: transforms integers into strings

–mat2str: transforms 2D matrices into strings

–char: convert to character array.

 

–print to screen: sprintf

–print to file: fprintf

–read from string: sscanf

–read from file: fscanf

 

---------file operation---------

 fid = fopen(filename)

–fclose(fid) %0 means success.

•%don’t suppress output.

firstline = fgetl(fid)

•%don’t suppress output.

–nextline = fgets(fid)

% fgetl reads a line but does NOT copy the end of line character to the string. fgets does.

feof(fid) : 0 while not end of file , 1 once it is found.

fprintf(fileID, PRINTING FORMAT, variable).

•Most common conversion characters.

–%c single character

–%d decimal notation

–%s string of characters

–INSIDE PRINTING area:

•\n new line

•\t horizontal tab

 

Rather than printing the text to the screen, let’s transfer it to another file.

–newfid = fopen(‘newgoldi.txt’,’w’);

–fprintf(newfid, ‘%s \n’, newline);

–

–‘w’ means write to this new file.

–‘r’ means open to read.

–‘rt’ reads as text.

–‘a’ means append (add at the end).

 

FIND SPACE CHARACTERS IN FIRST LINE.

–> spaces = find(firstline == ' ')

 

strcmp: compare whether two strings are the same

 

---

--2008/9/25--

--Lecture 5--

Intermixing text and variables

 

•fprintf('This is trial %2d.\n', trial);

•for count=1:10

–fprintf('This is trial %2d, and condition %d\n.',trial(count), condition(count));

–end;

 

 

Exercise --result display:

•Create a three column matrix with:

•first column: numbers from 1-10.

•second column: alternating 0-1.

•third column: random number between 150 and 1000.

•WRITE TO screen:

–think trial number, condition, RT.

 

Answer--

data = zeros(10,3);

data(:,1)=1:10;

data(:,2)=mod(data(:,1),2);

data(:,3)=rand(1,10)*850 +150;

 

%writes data column-wise.

%Treats matrix as comma-delimited list.

%CONTINUES EXECUTION until all the specified variables HAVE BEEN PRINTED.

%what we want is:
%data': the transposition of data

 

fprintf('%2d %d %3.1f\n',data');

 

Last issue.

•How do you print a ' or % or \ with fprintf?
ex: it's a beautiful day!
ex: I'm 100% certain 2\4=2.

•

•Answer: you double the escape character to make it printable (page17)

•> fprintf('I''m 100%% certain 2\\4=2.')

 

 

###########IMAGE###########

IMAGES (1) p. 22

•Let's play with Matlab's demo:

•Type:

•> clear all

•> load durer %check workspace.

•> image(X) %what happened?

•> colormap(map) %ruminate on this…

•> axis equal

•> axis image %same as equal, but image fits tightly

•> axis off %turns off tick marks

 

Other types of images

•You can load TIFF, JPEG, BMP… with

•imread

•[X,map] = imread(filename,ext);

 

 

Write images to files

•Let's make and save a random b/w mask image.

• imwrite(matrix,'nameoffile','extension')
%imwrite(matrix,colormap,'nof','ext') for indexed images

mask = rand(400,400);

imwrite(mask,'mask','bmp');

clear all

[x,map]=imread('mask','bmp');

image(x);

colormap(map);

 

 

•Let's make and save a random color-noise mask image -unindexed

•> mask2 = rand(400,400,3); %why 3?

•> imwrite(mask2,'mask2','jpg');

•> clear all;

•> input('click key when ready');

•> X = imread('mask2','jpg');

•> image(X);

•> colormap(map);

•> input('click key when ready');

•> colormap(hot);

Matlab homework5

 

 

1.       durernoise.m

Create a program that creates 5 different versions of the durer image with increasing levels of noise, using the same grayscale(256) CLUT, all in one single image (BMP).

 Level of noise is measured the deviation of the noise from the actual value of the pixel.

use the following as an example:

>> z = mod(X + (rand(size(X)).*32 - 16),128);
>> image(z)

% Author; He, Jibo, 09/26/2008
%Goal:
%Create a program that creates 5 different versions of the durer image with
%increasing levels of noise, using the same grayscale(256) CLUT, all in one
%single image (BMP).
load durer;
colormap(gray(256));
% create the image with noise
NoiseLevel1=mod(X+(rand(size(X)).*4-2),128);
NoiseLevel2=mod(X+(rand(size(X)).*8-4),128);
NoiseLevel3=mod(X+(rand(size(X)).*16-8),128);
NoiseLevel4=mod(X+(rand(size(X)).*32-16),128);
NoiseLevel5=mod(X+(rand(size(X)).*64-32),128);
combine=[NoiseLevel1'; NoiseLevel2';NoiseLevel3';NoiseLevel4';NoiseLevel5'];
image(combine');

% to polish the graph
axis off;
axis image;
xlabel 'Figure 1. Durer image with increasing levels of noise from left to right';

% write image
imwrite(combine',gray(256),'Durer.bmp','bmp');

 

Professor's solution:

Indexed image:

 

load durer; %gives me X and map

 

%I am going to create five matrices Xnoi which will be copies of X with

%increasing noise

Xnoi = X + rand(648,509).*60 -30; %noise here is created by randomly

%varying the luminance of a pixel

%the total range of luminance is

%124, so 30 is about 1/4 of that.

 

%PROBLEM: some indeces will be lower than 1 or larger than 128...

% so we correct for that. However you would like to do it!

 

below = find(Xnoi < 1); %find values of Xnoi that go below the colormap %index of 1

Xnoi(below) = 1; %for those values, we reassign a low luminance %value.

% the same in all cases!

 

above=find(Xnoi>128); % we do the same for indeces larger than 128.

Xnoi(above)= 128;

 

2.       rgbnoise.m

Create a programm that takes the visionlab jpg logo and presents it in 2 different levels of black and white noise and two different levels of color noise, all in one single image that includes the untouched original. 3. Submit both images and corresponding script files.

www.psych.uiuc.edu/~alleras/courseImages.htm

---

--2008/10/02--

--Lecture 6--

Intermixing text and variables

textread

function.

•

•SYNTAX:

•A = textread('filename') transforms data in filename into Matrix A.

•ONLY WORKS WITH HOMOGENEOUS Matrices.

= v ns = "urn:schemas-microsoft-com:vml" />= o ns = "urn:schemas-microsoft-com:office:office" />= p ns = "urn:schemas-microsoft-com:office:powerpoint" />

 

•SYNTAX:

•[A,B,C] = textread('filename','%s%d%f')

•reads each column into a variable, of specified type.

 

strings are saved in "cell" arrays (multidimensional arrays whose elements are copies of other arrays, here a table of strings of different sizes).

 

~Cell Array~

names(1) is the cell itself

•so trash = name(1) makes trash a cell

•names{1} refers to the value in the cell

•so trash = name{1} makes trash a character array

•names{1}(j) is the jth element in the character array stored in the cell 1.

 

 

USE strcmp(string1,string2)
which is true if string1==string2.

name2f = input('what student?','s');

numstu = size(name,1);  %number of rows

for findex=1:numstu

 if (strcmp(name2f,names{findex}))

 whichisit =findex;

 end;

end;

Matlab homework6

 

OPTION A. electricgraphiti.m Write a program that asks a user for his/her name and displays an image graphiti of their name using the gif letter files in: www.psych.uiuc.edu/~alleras/courseImages.htm 

Save the name as three jpgs of poor, medium and high quality (as due to compression). Hint: beware of capitals. 

OPTION B. Write a program that creates the image of a white circle (128 pixel radius) centered on a 400x400 black square. For extra credit (+2), make the brightness of the circle increase smoothly from black in the center to white at its edge. TOUGH GRADING. Hint: one piece of pie, but make sure all the pieces add up to the real thing! 

OPTION C: Do both. Second one counts as extra credit. If you do both, plus the extra credit option, you'll get +4 of ec.

 

 

eval

 

example 1:

for

vowel=['a' 'e' 'i' 'o' 'u']

string = [

'let' vowel '.gif'] %concatenate file name string

[letter,map]= imread(eval(

' string '));

image(letter);

colormap(map);

axis

off;

axis

equal;

input(

'Ready for next? \n');

end

;

example 2:

for n=1:10

eval(['M' num2str(n) ' = zeros(n,n)'])

end

Avoid loops.

USE vectors of arrays of indeces to access all elements at once!

for example

%use find function to find elements of array satisfying a specific condition.

 

U=find(X<30) ;

X(U)=cos(t);

 

% use : to travel all elements of an array.

 

end: to reach the last element of a row or column

all: to test at once if all values are true/nonzero in a matrix (or along a dimension)

any: to test if ANY value is true/nonzero in a matrix

 

example:

 

string='A' :'Z'

if all(string<'a') % judge whether all are lower letter.

   do-something

end

 

repmat: repeat same matrix a number of times.

ind2sub   sub2ind: how to obtain indeces from arrays and vice-versa.

 

 

############################

Install Psychtoolbox

############################
 

Step 1. download at http://psychtoolbox.org/PTB-2/download.html

Psychtoolbox Win 2.54 (20 February 2004) requires Matlab 6.5 (Student or regular) or better.
Download zip archive (3.1 MB)

IMPORTANT UPDATES TO Win 2.54:

After you download Win Psychtoolbox 2.54, replace bug-ridden files with improved versions listed below:

WaitSecs.dll - Get it here.

CopyText.dll - Get it here.

 

Step 2: add psychtoolbox to the path folder of matlab

 

 

---

--2008/10/23--

--Lecture 8--

 

1. Structures

Multidimensional array elements accessed by textual designators. Each field can contain any type of Matlab data (numbers, strings, cells, etc).

 

Type:

Data.trial = 1;

Data.setsize = 3;

Data.tgtword = 'doctor';

Data.rt = 541;

Data.resp =1;

Data

 

Using the "struct" function:

Data = struct('label1', dummy1, 'label2',dummy2, etc);

Creates the structure:

data.label1 = dummy1

data.label2 = dummy2

 

 

Data(64) = struct('label1', dummy1, 'label2', dummy2, etc);

 

Access each element like a vector:

Data(34).label1 = 234;

 

data(n) =struct('field1',value1,'field2',value2)

initializes only the nth value. Others are set to empty matrices.

data = repmat(struct('trial',1,'rt',-1),1,64);

This way ALL values are initialized with values specified in the struct function.

Or: Use values saved in a cell array.

> a = cell(3,1);

> a{1} = 'bob';

> a{2} = 'where are you?';

> a{3} = 'I am here';

> data = struct('line',a);

What's data(2).line?

Planning an experiment

 

 

section 1:SETUP VARIABLES

-Initialize CONSTANTS (refresh rate)

-Initialize Variables (with comments so you know what each variable does)

-Load big files (images, sounds, mex…)

Section 2:BALANCE CONDITIONS

BALANCE your design:

-define conditions

-how many trials for each condition

-Randomize order of trials

 

Section 3: TRIAL LOOP

3.1 DRAW constant images

(fixation, blank screens)

3.2 Start Trial Loop

3.2.1 Draw trial specific stimuli (if any)

3.2.2 Present stimuli

3.2.3 Get response

3.2.4 Classify response (error?)

3.2.5 ERASE ANY TRIAL SPECIFIC STIMULI

 

Section 4:Save DATA

Open a subject file and write data to it. (personal preference).

save data after each block/trial to avoid data loss or software crash.

 

Section 5:CLEAN UP

Clear all the variables you used and

the images you created, and close

any opened files.

 

AND DON'T FORGET TO COMMENT AS MUCH AS YOU CAN!!!

AND DON'T FORGET SECTION 0: Comments at beginning of file for

"help".

 

The Psychophysics Toolbox

A set of functions to:

-Interact with Monitor (pg. 27-32)

-Interact with Keyboard and mouse

-Interact with your OS (Here, Windows XP)

 

Screen function:

-Function that helps us interact with our monitor.

- Many "sub-functions".

First thing:

OpenWindow:

[windowPtr,rect]=Screen(0,'OpenWindow',[color],[rectangle],[pixelSize]);

 

windowptr: a pointer to the space in memory we are allocating to work on this window (kinda like fid=fopen(..) keeps track of a file), can be called "fixation display",'practice' etc.

rect: (if specified, and I suggest you do) gives you the coordinates of the window you’ll be using in pixels, on the format [Xtop-left, Ytop-left, Xbotton-right, Ybottom-right]. e.g. [0 0 1280 1024]

0: refers to the main monitor (where you'll be presenting stimuli).

color: you want the window to be: if one number: an index (CLUT, between 0-255), or a RGB triplet [r g b]. Later we'll talk about

changing the CLUT.

rectangle: ignored in windows

pixelsize: you can set the pixelsize for your screens (8 bit -> 256 colors, 24 bit…). Default is unchanged.

 

 

Close Window

Two ways:

To close all windows:

Screen('CloseAll');

To close a specific window:

Screen(windowPtr,'Close');

VERY IMPORTANT!!!

 

"Hello World " to psychtoolbox

warning

off MATLAB:DeprecatedLogicalAPI

 

[windowPtr,rect]=Screen(0,

'OpenWindow',255); % CLUT 255 for white background, 0 for black

Screen(windowPtr,

'DrawText','Hello World',500,350,100);

KbWait; %wait for the user to push a key

Screen(

'CloseAll');

DETERMINE HOW LONG IT TOOK MATLAB

TO OPEN THE WINDOW AND WRITE TEXT

TO IT.

Use: GetSecs (returns seconds since computer was turned on);

 

OffScreenWindows

So, we can work "offline", prepare our screens and then quickly copy

them to the screen.

image = Screen(windowPtr,'OpenoffScreenWindow',color,smallerrect)

image is pointer to refer to this offscreen window

windowPtr is pointer to the monitor window (to which all windows are related)

smallerrect is size of offscreen window (can be smaller!)

So, Once we have our offscreen window, we THEN copy it to our main window.

Screen('CopyWindow', srcWinPtr,DestWindoPtr, [srcRect],[dstRect]);

srcWinPtr: is the windowPointer (name) of the window you want to copy (source)

Bugs: The color template in open window subcommand is BGR instead of RGB. The following command will create blue background.

[window,rect]=Screen(0,'OpenWindow',[255 0 0]);

 

image = Screen(window,

'OpenoffScreenWindow',[255 0 0],[0 0 100 100]);

HideCursor;

beep=MakeBeep(100,5);

Snd('Open');

Snd('Play',beep); % play beep sound.

 

PutImage

Screen(windowPtr,'PutImage', imagearray,[rect],[copymode]);

 

imagearray is a matrix like the ones we created a few weeks ago:MxN or MxNx3 (if 16 or 32 bits graphic card)

 

Help on Screen functions?

Type:

Screen(‘NameOfFunction?’)

Try:

Screen(‘PutImage?’)

 

Beeps and Wav sounds

Matlab can play wav sounds and beeps WHILE doing something else

(nice for experiments).

Just like we 'Open' a window, we need to 'Open' a sound channel:

Snd('Open');

and 'close' it when you are done:

Snd('Close');

 

Beeps

You can create beeps with MakeBeep, which creates a vector that will be interpreted by your sound card (just like a matrix of numbers is interpreted by your graphics card as an image).

beep = MakeBeep(frequency,duration,samplingrate);

Then you can play that beep:

Snd('Play',beep,samplingrate);

 

Wav files

Windows Audio files:

-> wavplay(y,FS) plays sound recorded in y vector at the sampling frequency specified in FS (same sound will be different if played faster (larger FS) or slower (smaller FS). For stereo playback, Y would be N-by-2 matrix (left, right channels).

->important:

wavplay(y,FS,'async') allows you to play that sound while continuing to do stuff in Matlab (non-blocking call).

 

wavrecord(N,FS,CH)

 

Synchronizing windows and monitor

To do so, we use the 'WaitBlanking' command in Screen.

Calling:

Screen(windowptr,'WaitBlanking')

will wait until your gun moves to the top of the monitor. (only true for CRT monitors and analog LCDs)

 

counting time with refresh rates:

->IF you want to present a stimulus for a specific amount of time, count time in refreshes.

 

TIME

counting time with GetSecs:

Cool and accurate. You can do something like:

t1=GetSecs;

t2=t1;

Screen('CopyWindow'…); %or whatever…

while ((t2-t1)< presentationTime)

t2=GetSecs;

end;

%This syntax is equivalent to WaitSecs.

 

tic (start counting time)

toc (count the time. )

 

 

###Get the refresh rate of monitor#######

%He, Jibo @UIUC, 10/30/2008

warning off MATLAB:DeprecatedLogicalAPI

[windowptr,rectangle]=screen(0,'OpenWindow',255);

Screen(windowptr,'WaitBlanking')

t1=GetSecs;

t2=t1;

counter=0;

while ((t2-t1)< 1)

Screen(windowptr,'WaitBlanking')

counter=counter+1;

t2=GetSecs;

end;

counter

Screen('CloseAll');

 ##############################

 

frames=FrameRate(window); %Matlab's inquiry function

 

 

###wait 100ms#######

frames=FrameRate(window)%present stimuli 100ms. to be accurate, the multiply of the refreshrate

Screen(window,'WaitBlanking',floor(frames./10))

 

 ##############################

 

Keyboard Management

List of useful functions:

KbCheck: status of keyboard

KbWait: waits for Keypress (returns GetSecs)

GetChar: waits for character

CharAvail: checks event queue for characters

FlushEvents: help manage event queue

EventAvail: checks for events

KbCheck

IS USEFUL TO CLEAN THE KEYBOARD

BUFFER!!

while KbCheck end;

will "clean" your buffer

(Technically, wait until it is clean)

 

[keyisDown,secs,keyCode]=KbCheck;

find(keyCode) which key in ASCII

or you can ask directly if a given key was hit:

if keyCode('Z') (would be 1 if the z key was hit)

%note capitalization.

 

KbName: toolbox function that allows us to name the different keys on the keyboard (primary label) (Note: '5' vs. '5%')

Usage: KbName(arg) if arg is a string ('z'): returns the keyCode for that key if arg is the array keyCode, KbName returns the label of the key.

KbName deals with KEYS not Characters!

leftTarget = KbName('left');

 

MOUSE Management

List of useful functions:

HideCursor

ShowCursor

GetClicks

GetMouse

SetMouse

 

[x,y,buttons]=GetMouse(windowPtr);

 

 

SetMouse(100,500) # move the mouse to (100,500)

 

 

PsychDemos

If you are wondering what kind of thing you can do with PTB and how some of your

ideas can be coded up, look at the demos that ship with PTB. Try typing:

>>help PsychDemos

This will give you a list of available demos and a short description of what they do. If you are curious what a certain

demo does you can inquire further. For example, type:

>>help MandelbrotDemo

This will tell you what this script does. If you are curious how this is implemented, type:

>>edit MandelbrotDemo

This will open the file MandelbrotDemo.m in an editor window. Don’t edit this file! You might cause some damage.

Instead, save the file under a new name. For example, ‘myMandelbrotDemo.m’. Now you can twiddle things in the file

and try to see what effect these changes have on the execution of the program. But before you start doing that, let’s get

acquainted with the single most important function in Psychtoolbox.

 

 

----------------------GUI in matlab--------------------

quesdlg

ButtonName=questdlg(Question,Title,Btn1,Btn2,

DEFAULT);

-up to three buttons.

-Default is optional.

ButtonName=questdlg('What is your wish?', ...

'Genie Question', ...

'Food','Clothing','Money','Money');

 

 

INPUTDLG

ANSWER = INPUTDLG(PROMPT) creates an input dialog box where

users can enter text, saved in the cell array ANSWER.

PROMPT is a cell array containing the PROMPT strings.

A Brief History of

Human Computer Interaction Technology

Brad A. Myers

Carnegie Mellon University School of Computer Science Technical
Report CMU-CS-96-163
and

Human Computer Interaction Institute
Technical Report CMU-HCII-96-103

December, 1996

Please cite this work as:

Brad A. Myers. "A Brief History of Human Computer Interaction
Technology."
ACM interactions. Vol. 5, no. 2, March, 1998. pp. 44-54.

Human Computer Interaction Institute


School of Computer Science

Carnegie Mellon University

Pittsburgh, PA 15213-3891

bam@a.gp.cs.cmu.edu

Abstract

This article summarizes the historical development of major advances in
human-computer interaction technology, emphasizing the pivotal role of
university research in the advancement of the field.

Copyright (c) 1996 -- Carnegie Mellon University

A short excerpt from this article appeared as part of "Strategic Directions
in
Human Computer Interaction," edited by Brad Myers, Jim Hollan, Isabel Cruz,
ACM Computing Surveys, 28(4), December 1996

This research was partially sponsored by NCCOSC under Contract No.
N66001-94-C-6037, Arpa Order No. B326 and partially by NSF under grant number
IRI-9319969. The views and conclusions contained in this document are those
of
the authors and should not be interpreted as representing the official
policies, either expressed or implied, of NCCOSC or the U.S. Government.

Keywords: Human Computer Interaction, History, User Interfaces,
Interaction Techniques.

---

1.
Introduction

Research in Human-Computer Interaction (HCI) has been spectacularly
successful,
and has fundamentally changed computing. Just one example is the ubiquitous
graphical interface used by Microsoft Windows 95, which is based on the
Macintosh, which is based on work at Xerox PARC, which in turn is based on
early research at the Stanford Research Laboratory (now SRI) and at the
Massachusetts Institute of Technology. Another example is that virtually
all
software written today employs user interface toolkits and interface builders,
concepts which were developed first at universities. Even the spectacular
growth of the World-Wide Web is a direct result of HCI research: applying
hypertext technology to browsers allows one to traverse a link across the
world
with a click of the mouse. Interface improvements more than anything else
has
triggered this explosive growth. Furthermore, the research that will lead
to
the user interfaces for the computers of tomorrow is happening at universities
and a few corporate research labs.

This paper tries to briefly summarize many of the important research
developments in Human-Computer Interaction (HCI) technology. By "research,"
I
mean exploratory work at universities and government and corporate research
labs (such as Xerox PARC) that is not directly related to products. By "HCI
technology," I am referring to the computer side of HCI. A companion article
on the history of the "human side," discussing the contributions from
psychology, design, human factors and ergonomics would also be appropriate.

A motivation for this article is to overcome the mistaken impression that
much
of the important work in Human-Computer Interaction occurred in industry,
and
if university research in Human-Computer Interaction is not supported, then
industry will just carry on anyway. This is simply not true. This paper
tries
to show that many of the most famous HCI successes developed by companies
are
deeply rooted in university research. In fact, virtually all of today's
major
interface styles and applications have had significant influence from research
at universities and labs, often with government funding. To illustrate this,
this paper lists the funding sources of some of the major advances. Without
this research, many of the advances in the field of HCI would probably not
have
taken place, and as a consequence, the user interfaces of commercial products
would be far more difficult to use and learn than they are today. As described
by Stu Card:

"Government funding of advanced human-computer interaction technologies built
the intellectual capital and trained the research teams for pioneer systems
that, over a period of 25 years, revolutionized how people interact with
computers. Industrial research laboratories at the corporate level in Xerox,
IBM, AT&T, and others played a strong role in developing this technology
and bringing it into a form suitable for the commercial arena." [6, p.
162]).

Figure 1 shows time lines for some of the technologies discussed in this
article. Of course, a deeper analysis would reveal much interaction between
the university, corporate research and commercial activity streams. It is
important to appreciate that years of research are involved in creating and
making these technologies ready for widespread use. The same will be true
for
the HCI technologies that will provide the interfaces of tomorrow.

It is clearly impossible to list every system and source in a paper of this
scope, but I have tried to represent the earliest and most influential systems.
Although there are a number of other surveys of HCI topics (see, for example
[1] [10] [33] [38]), none cover as many aspects as this one, or try to be
as
comprehensive in finding the original influences. Another useful resource
is
the video "All The Widgets," which shows the historical progression of a
number
of user interface ideas [25].

The technologies covered in this paper include fundamental interaction styles
like direct manipulation, the mouse pointing device, and windows; several
important kinds of application areas, such as drawing, text editing and
spreadsheets; the technologies that will likely have the biggest impact on
interfaces of the future, such as gesture recognition, multimedia, and 3D;
and
the technologies used to create interfaces using the other technologies,
such as user interface management systems, toolkits, and interface builders.

---

Figure 1: Approximate time lines showing where work was performed
on
some major technologies discussed in this article.

---

2.
Basic Interactions

• Direct Manipulation of graphical objects: The now ubiquitous
  direct
  manipulation interface, where visible objects on the screen are directly
  manipulated with a pointing device, was first demonstrated by Ivan Sutherland
  in Sketchpad [44], which was his 1963 MIT PhD thesis. SketchPad supported
  the
  manipulation of objects using a light-pen, including grabbing objects, moving
  them, changing size, and using constraints. It contained the seeds of myriad
  important interface ideas. The system was built at Lincoln Labs with support
  from the Air Force and NSF. William Newman's Reaction Handler [30], created
  at
  Imperial College, London (1966-67) provided direct manipulation of graphics,
  and introduced "Light Handles," a form of graphical potentiometer, that was
  probably the first "widget." Another early system was AMBIT/G (implemented
  at
  MIT's Lincoln Labs, 1968, ARPA funded). It employed, among other interface
  techniques, iconic representations, gesture recognition, dynamic menus with
  items selected using a pointing device, selection of icons by pointing, and
  moded and mode-free styles of interaction. David Canfield Smith coined the
  term "icons" in his 1975 Stanford PhD thesis on Pygmalion [41] (funded by
  ARPA
  and NIMH) and Smith later popularized icons as one of the chief designers
  of
  the Xerox Star [42]. Many of the interaction techniques popular in direct
  manipulation interfaces, such as how objects and text are selected, opened,
  and
  manipulated, were researched at Xerox PARC in the 1970's. In particular,
  the
  idea of "WYSIWYG" (what you see is what you get) originated there with systems
  such as the Bravo text editor and the Draw drawing program [10] The concept
  of
  direct manipulation interfaces for everyone was envisioned by Alan Kay of
  Xerox
  PARC in a 1977 article about the "Dynabook" [16]. The first commercial systems
  to make extensive use of Direct Manipulation were the Xerox Star (1981) [42],
  the Apple Lisa (1982) [51] and Macintosh (1984) [52]. Ben Shneiderman at
  the
  University of Maryland coined the term "Direct Manipulation" in 1982 and
  identified the components and gave psychological foundations [40].
  

  
  

• The Mouse: The mouse was developed at Stanford Research Laboratory
  (now SRI) in 1965 as part of the NLS project (funding from ARPA, NASA, and
  Rome
  ADC) [9] to be a cheap replacement for light-pens, which had been used at
  least
  since 1954 [10, p. 68]. Many of the current uses of the mouse were
  demonstrated by Doug Engelbart as part of NLS in a movie created in 1968
  [8].
  The mouse was then made famous as a practical input device by Xerox PARC
  in the
  1970's. It first appeared commercially as part of the Xerox Star (1981),
  the
  Three Rivers Computer Company's PERQ (1981) [23], the Apple Lisa (1982),
  and
  Apple Macintosh (1984).
  

  
  

• Windows: Multiple tiled windows were demonstrated in Engelbart's
  NLS
  in 1968 [8]. Early research at Stanford on systems like COPILOT (1974) [46]
  and at MIT with the EMACS text editor (1974) [43] also demonstrated tiled
  windows. Alan Kay proposed the idea of overlapping windows in his 1969
  University of Utah PhD thesis [15] and they first appeared in 1974 in his
  Smalltalk system [11] at Xerox PARC, and soon after in the InterLisp system
  [47]. Some of the first commercial uses of windows were on Lisp Machines
  Inc.
  (LMI) and Symbolics Lisp Machines (1979), which grew out of MIT AI Lab
  projects. The Cedar Window Manager from Xerox PARC was the first major tiled
  window manager (1981) [45], followed soon by the Andrew window manager [32]
  by
  Carnegie Mellon University's Information Technology Center (1983, funded
  by
  IBM). The main commercial systems popularizing windows were the Xerox Star
  (1981), the Apple Lisa (1982), and most importantly the Apple Macintosh (1984).
  The early versions of the Star and Microsoft Windows were tiled, but eventually
  they supported overlapping windows like the Lisa and Macintosh. The X Window
  System, a current international standard, was developed at MIT in 1984 [39].
  For a survey of window managers, see [24].
  

3.
Application Types

• Drawing programs: Much of the current technology was
  demonstrated in
  Sutherland's 1963 Sketchpad system. The use of a mouse for graphics was
  demonstrated in NLS (1965). In 1968 Ken Pulfer and Grant Bechthold at the
  National Research Council of Canada built a mouse out of wood patterned after
  Engelbart's and used it with a key-frame animation system to draw all the
  frames of a movie. A subsequent movie, "Hunger" in 1971 won a number of
  awards, and was drawn using a tablet instead of the mouse (funding by the
  National Film Board of Canada) [3]. William Newman's Markup (1975) was the
  first drawing program for Xerox PARC's Alto, followed shortly by Patrick
  Baudelaire's Draw which added handling of lines and curves [10, p. 326].
  The
  first computer painting program was probably Dick Shoup's "Superpaint" at
  PARC
  (1974-75).
  

  
  

• Text Editing: In 1962 at the Stanford Research Lab, Engelbart
  proposed, and later implemented, a word processor with automatic word wrap,
  search and replace, user-definable macros, scrolling text, and commands to
  move, copy, and delete characters, words, or blocks of text. Stanford's
  TVEdit
  (1965) was one of the first CRT-based display editors that was widely used
  [48]. The Hypertext Editing System [50, p. 108] from Brown University had
  screen editing and formatting of arbitrary-sized strings with a lightpen
  in
  1967 (funding from IBM). NLS demonstrated mouse-based editing in 1968.
  TECO
  from MIT was an early screen-editor (1967) and EMACS [43] developed from
  it in
  1974. Xerox PARC's Bravo [10, p. 284] was the first WYSIWYG editor-formatter
  (1974). It was designed by Butler Lampson and Charles Simonyi who had started
  working on these concepts around 1970 while at Berkeley. The first commercial
  WYSIWYG editors were the Star, LisaWrite and then MacWrite. For a survey
  of
  text editors, see [22] [50, p. 108].
  

  
  

• Spreadsheets: The initial spreadsheet was VisiCalc which was developed
  by Frankston and Bricklin (1977-8) for the Apple II while they were students
  at
  MIT and the Harvard Business School. The solver was based on a
  dependency-directed backtracking algorithm by Sussman and Stallman at the
  MIT
  AI Lab.
  

  
  

• HyperText: The idea for hypertext (where documents are linked
  to
  related documents) is credited to Vannevar Bush's famous MEMEX idea from
  1945
  [4]. Ted Nelson coined the term "hypertext" in 1965 [29]. Engelbart's NLS
  system [8] at the Stanford Research Laboratories in 1965 made extensive use
  of
  linking (funding from ARPA, NASA, and Rome ADC). The "NLS Journal" [10,
  p.
  212] was one of the first on-line journals, and it included full linking
  of
  articles (1970). The Hypertext Editing System, jointly designed by Andy
  van
  Dam, Ted Nelson, and two students at Brown University (funding from IBM)
  was
  distributed extensively [49]. The University of Vermont's PROMIS (1976)
  was
  the first Hypertext system released to the user community. It was used to
  link
  patient and patient care information at the University of Vermont's medical
  center. The ZOG project (1977) from CMU was another early hypertext system,
  and was funded by ONR and DARPA [36]. Ben Shneiderman's Hyperties was the
  first system where highlighted items in the text could be clicked on to go
  to
  other pages (1983, Univ. of Maryland) [17]. HyperCard from Apple (1988)
  significantly helped to bring the idea to a wide audience. There have been
  many other hypertext systems through the years. Tim Berners-Lee used the
  hypertext idea to create the World Wide Web in 1990 at the government-funded
  European Particle Physics Laboratory (CERN). Mosaic, the first popular
  hypertext browser for the World-Wide Web was developed at the Univ. of
  Illinois' National Center for Supercomputer Applications (NCSA). For a more
  complete history of HyperText, see [31].
  

  
  

• Computer Aided Design (CAD): The same 1963 IFIPS conference at
  which
  Sketchpad was presented also contained a number of CAD systems, including
  Doug
  Ross's Computer-Aided Design Project at MIT in the Electronic Systems Lab
  [37]
  and Coons' work at MIT with SketchPad [7]. Timothy Johnson's pioneering
  work
  on the interactive 3D CAD system Sketchpad 3 [13] was his 1963 MIT MS thesis
  (funded by the Air Force). The first CAD/CAM system in industry was probably
  General Motor's DAC-1 (about 1963).
  

  
  

• Video Games: The first graphical video game was probably SpaceWar
  by
  Slug Russel of MIT in 1962 for the PDP-1 [19, p. 49] including the first
  computer joysticks. The early computer Adventure game was created by Will
  Crowther at BBN, and Don Woods developed this into a more sophisticated
  Adventure game at Stanford in 1966 [19, p. 132]. Conway's game of LIFE was
  implemented on computers at MIT and Stanford in 1970. The first popular
  commercial game was Pong (about 1976).
  

4.
Up-and-Coming Areas

• Gesture Recognition: The first pen-based input device,
  the RAND
  tablet, was funded by ARPA. Sketchpad used light-pen gestures (1963).
  Teitelman in 1964 developed the first trainable gesture recognizer. A very
  early demonstration of gesture recognition was Tom Ellis' GRAIL system on
  the
  RAND tablet (1964, ARPA funded). It was quite common in light-pen-based
  systems to include some gesture recognition, for example in the AMBIT/G system
  (1968 -- ARPA funded). A gesture-based text editor using proof-reading symbols
  was developed at CMU by Michael Coleman in 1969. Bill Buxton at the University
  of Toronto has been studying gesture-based interactions since 1980. Gesture
  recognition has been used in commercial CAD systems since the 1970s, and
  came
  to universal notice with the Apple Newton in 1992.
  

  
  

• Multi-Media: The FRESS project at Brown used multiple windows
  and
  integrated text and graphics (1968, funding from industry). The Interactive
  Graphical Documents project at Brown was the first hypermedia (as opposed
  to
  hypertext) system, and used raster graphics and text, but not video (1979-1983,
  funded by ONR and NSF). The Diamond project at BBN (starting in 1982, DARPA
  funded) explored combining multimedia information (text, spreadsheets,
  graphics, speech). The Movie Manual at the Architecture Machine Group (MIT)
  was one of the first to demonstrate mixed video and computer graphics in
  1983
  (DARPA funded).
  

  
  

• 3-D: The first 3-D system was probably Timothy Johnson's 3-D CAD
  system mentioned above (1963, funded by the Air Force). The "Lincoln Wand"
  by
  Larry Roberts was an ultrasonic 3D location sensing system, developed at
  Lincoln Labs (1966, ARPA funded). That system also had the first interactive
  3-D hidden line elimination. An early use was for molecular modelling [18].
  The late 60's and early 70's saw the flowering of 3D raster graphics research
  at the University of Utah with Dave Evans, Ivan Sutherland, Romney, Gouraud,
  Phong, and Watkins, much of it government funded. Also, the
  military-industrial flight simulation work of the 60's - 70's led the way
  to
  making 3-D real-time with commercial systems from GE, Evans&Sutherland,
  Singer/Link (funded by NASA, Navy, etc.). Another important center of current
  research in 3-D is Fred Brooks' lab at UNC (e.g. [2]).
  

  
  

• Virtual Reality and "Augmented Reality": The original work on
  VR was
  performed by Ivan Sutherland when he was at Harvard (1965-1968, funding
  by Air
  Force, CIA, and Bell Labs). Very important early work was by Tom Furness
  when
  he was at Wright-Patterson AFB. Myron Krueger's early work at the University
  of Connecticut was influential. Fred Brooks' and Henry Fuch's groups at
  UNC
  did a lot of early research, including the study of force feedback (1971,
  funding from US Atomic Energy Commission and NSF). Much of the early research
  on head-mounted displays and on the DataGlove was supported by NASA.
  

  
  

• Computer Supported Cooperative Work. Doug Engelbart's 1968
  demonstration of NLS [8] included the remote participation of multiple people
  at various sites (funding from ARPA, NASA, and Rome ADC). Licklider and
  Taylor
  predicted on-line interactive communities in an 1968 article [20] and
  speculated about the problem of access being limited to the privileged.
  Electronic mail, still the most widespread multi-user software, was enabled
  by
  the ARPAnet, which became operational in 1969, and by the Ethernet from Xerox
  PARC in 1973. An early computer conferencing system was Turoff's EIES system
  at the New Jersey Institute of Technology (1975).
  

  
  

• Natural language and speech: The fundamental research for speech
  and
  natural language understanding and generation has been performed at CMU,
  MIT,
  SRI, BBN, IBM, AT&T Bell Labs and BellCore, much of it government funded.
  See, for example, [34] for a survey of the early work.
  

5.
Software Tools and Architectures

The area of user interface software tools is quite active now, and
many
companies are selling tools. Most of today's applications are implemented
using various forms of software tools. For a more complete survey and
discussion of UI tools, see [26].

• UIMSs and Toolkits: (There are software libraries and tools
  that
  support creating interfaces by writing code.) The first User Interface
  Management System (UIMS) was William Newman's Reaction Handler [30] created
  at
  Imperial College, London (1966-67 with SRC funding). Most of the early work
  was done at universities (Univ. of Toronto with Canadian government funding,
  George Washington Univ. with NASA, NSF, DOE, and NBS funding, Brigham Young
  University with industrial funding, etc.). The term "UIMS" was coined by
  David
  Kasik at Boeing (1982) [14]. Early window managers such as Smalltalk (1974)
  and InterLisp, both from Xerox PARC, came with a few widgets, such as popup
  menus and scrollbars. The Xerox Star (1981) was the first commercial system
  to
  have a large collection of widgets. The Apple Macintosh (1984) was the first
  to actively promote its toolkit for use by other developers to enforce a
  consistent interface. An early C++ toolkit was InterViews [21], developed
  at
  Stanford (1988, industrial funding). Much of the modern research is being
  performed at universities, for example the Garnet (1988) [28] and Amulet
  (1994) [27] projects at CMU (ARPA funded), and subArctic at Georgia Tech
  (1996,
  funding by Intel and NSF).
  

  
  

• Interface Builders: (These are interactive tools that allow interfaces
  composed of widgets such as buttons, menus and scrollbars to be placed using
  a
  mouse.) The Steamer project at BBN (1979-85; ONR funding) demonstrated many
  of
  the ideas later incorporated into interface builders and was probably the
  first
  object-oriented graphics system. Trillium [12] was developed at Xerox PARC
  in
  1981. Another early interface builder was the MenuLay system [5] developed
  by
  Bill Buxton at the University of Toronto (1983, funded by the Canadian
  Government). The Macintosh (1984) included a "Resource Editor" which allowed
  widgets to be placed and edited. Jean-Marie Hullot created "SOS Interface"
  in
  Lisp for the Macintosh while working at INRIA (1984, funded by the French
  government) which was the first modern "interface builder." Hullot built
  this
  into a commercial product in 1986 and then went to work for NeXT and created
  the NeXT Interface Builder (1988), which popularized this type of tool.
  Now
  there are literally hundreds of commercial interface builders.
  

  
  

• Component Architectures: The idea of creating interfaces by connecting
  separately written components was first demonstrated in the Andrew project
  [32]
  by Carnegie Mellon University's Information Technology Center (1983, funded
  by
  IBM). It is now being widely popularized by Microsoft's OLE and Apple's
  OpenDoc architectures.
  

6.
Discussion

It is clear that all of the most important innovations in Human-Computer
Interaction have benefited from research at both corporate research labs
and
universities, much of it funded by the government. The conventional style
of
graphical user interfaces that use windows, icons, menus and a mouse and
are in
a phase of standardization, where almost everyone is using the same, standard
technology and just making minute, incremental changes. Therefore, it is
important that university, corporate, and government-supported research
continue, so that we can develop the science and technology needed for the
user
interfaces of the future.

Another important argument in favor of HCI research in universities is that
computer science students need to know about user interface issues. User
interfaces are likely to be one of the main value-added competitive advantages
of the future, as both hardware and basic software become commodities. If
students do not know about user interfaces, they will not serve industry
needs.
It seems that only through computer science does HCI research disseminate
out
into products. Furthermore, without appropriate levels of funding of academic
HCI research, there will be fewer PhD graduates in HCI to perform research
in
corporate labs, and fewer top-notch graduates in this area will be interested
in being professors, so the needed user interface courses will not be
offered.

As computers get faster, more of the processing power is being devoted to
the
user interface. The interfaces of the future will use gesture recognition,
speech recognition and generation, "intelligent agents," adaptive interfaces,
video, and many other technologies now being investigated by research groups
at
universities and corporate labs [35]. It is imperative that this research
continue and be well-supported.

ACKNOWLEDGMENTS

I must thank a large number of people who responded to posts of earlier
versions of this article on the announcements.chi mailing list for their
very
generous help, and to Jim Hollan who helped edit the short excerpt of this
article. Much of the information in this article was supplied by (in
alphabetical order): Stacey Ashlund, Meera M. Blattner, Keith Butler, Stuart
K.
Card, Bill Curtis, David E. Damouth, Dan Diaper, Dick Duda, Tim T.K. Dudley,
Steven Feiner, Harry Forsdick, Bjorn Freeman-Benson, John Gould, Wayne Gray,
Mark Green, Fred Hansen, Bill Hefley, D. Austin Henderson, Jim Hollan,
Jean-Marie Hullot, Rob Jacob, Bonnie John, Sandy Kobayashi, T.K. Landauer,
John
Leggett, Roger Lighty, Marilyn Mantei, Jim Miller, William Newman, Jakob
Nielsen, Don Norman, Dan Olsen, Ramesh Patil, Gary Perlman, Dick Pew, Ken
Pier,
Jim Rhyne, Ben Shneiderman, John Sibert, David C. Smith, Elliot Soloway,
Richard Stallman, Ivan Sutherland, Dan Swinehart, John Thomas, Alex Waibel,
Marceli Wein, Mark Weiser, Alan Wexelblat, and Terry Winograd. Editorial
comments were also provided by the above as well as Ellen Borison, Rich
McDaniel, Rob Miller, Bernita Myers, Yoshihiro Tsujino, and the reviewers.

References

1. Baecker, R., et al., "A Historical and Intellectual Perspective,"
in
Readings in Human-Computer Interaction: Toward the Year 2000, Second
Edition, R. Baecker, et al., Editors. 1995, Morgan Kaufmann
Publishers, Inc.: San Francisco. pp. 35-47.

2. Brooks, F. "The Computer "Scientist" as Toolsmith--Studies in Interactive
Computer Graphics," in IFIP Conference Proceedings. 1977. pp.
625-634.

3. Burtnyk, N. and Wein, M., "Computer Generated Key Frame Animation."
Journal Of the Society of Motion Picture and Television Engineers,
1971.
8(3): pp. 149-153.

4. Bush, V., "As We May Think." The Atlantic Monthly, 1945.
176(July): pp. 101-108. Reprinted and discussed in
interactions,
3(2), Mar 1996, pp. 35-67.

5. Buxton, W., et al. "Towards a Comprehensive User Interface Management
System," in Proceedings SIGGRAPH'83: Computer Graphics. 1983. Detroit,
Mich. 17. pp. 35-42.

6. Card, S.K., "Pioneers and Settlers: Methods Used in Successful User
Interface Design," in Human-Computer Interface Design: Success Stories,
Emerging Methods, and Real-World Context, M. Rudisill, et al.,
Editors. 1996, Morgan Kaufmann Publishers: San Francisco. pp. 122-169.

7. Coons, S. "An Outline of the Requirements for a Computer-Aided Design
System," in AFIPS Spring Joint Computer Conference. 1963. 23.
pp. 299-304.

8. Engelbart, D. and English, W., "A Research Center for Augmenting Human
Intellect." Reprinted in ACM SIGGRAPH Video Review, 1994.,
1968.
106

9. English, W.K., Engelbart, D.C., and Berman, M.L., "Display Selection
Techniques for Text Manipulation." IEEE Transactions on Human Factors
in Electronics, 1967. HFE-8(1)

10. Goldberg, A., ed. A History of Personal Workstations. 1988,
Addison-Wesley Publishing Company: New York, NY. 537.

11. Goldberg, A. and Robson, D. "A Metaphor for User Interface Design," in
Proceedings of the 12th Hawaii International Conference on System
Sciences. 1979. 1. pp. 148-157.

12. Henderson Jr, D.A. "The Trillium User Interface Design Environment,"
in
Proceedings SIGCHI'86: Human Factors in Computing Systems. 1986. Boston,
MA. pp. 221-227.

13. Johnson, T. "Sketchpad III: Three Dimensional Graphical Communication
with
a Digital Computer," in AFIPS Spring Joint Computer Conference. 1963.
23. pp. 347-353.

14. Kasik, D.J. "A User Interface Management System," in Proceedings
SIGGRAPH'82: Computer Graphics. 1982. Boston, MA. 16. pp. 99-106.

15. Kay, A., The Reactive Engine. PhD Thesis, Electrical Engineering
and
Computer Science University of Utah, 1969,

16. Kay, A., "Personal Dynamic Media." IEEE Computer, 1977.
10(3): pp. 31-42.

17. Koved, L. and Shneiderman, B., "Embedded menus: Selecting items in
context." Communications of the ACM, 1986. 4(29): pp.
312-318.

18. Levinthal, C., "Molecular Model-Building by Computer." Scientific
American, 1966. 214(6): pp. 42-52.

19. Levy, S., Hackers: Heroes of the Computer Revolution. 1984, Garden
City, NY: Anchor Press/Doubleday.

20. Licklider, J.C.R. and Taylor, R.W., "The computer as Communication
Device." Sci. Tech., 1968. April: pp. 21-31.

21. Linton, M.A., Vlissides, J.M., and Calder, P.R., "Composing user interfaces
with InterViews." IEEE Computer, 1989. 22(2): pp. 8-22.

22. Meyrowitz, N. and Van Dam, A., "Interactive Editing Systems: Part 1 and
2." ACM Computing Surveys, 1982. 14(3): pp. 321-352.

23. Myers, B.A., "The User Interface for Sapphire." IEEE Computer
Graphics and Applications, 1984. 4(12): pp. 13-23.

24. Myers, B.A., "A Taxonomy of User Interfaces for Window Managers."
IEEE Computer Graphics and Applications, 1988. 8(5): pp. 65-84.

25. Myers, B.A., "All the Widgets." SIGGRAPH Video Review,
1990.
57

26. Myers, B.A., "User Interface Software Tools." ACM Transactions
on
Computer Human Interaction, 1995. 2(1): pp. 64-103.

27. Myers, B.A., et al., The Amulet V2.0 Reference Manual .
Carnegie Mellon University Computer Science Department Report, Number, Feb,
1996. System available from http://www.cs.cmu.edu/~amulet.

28. Myers, B.A., et al., "Garnet: Comprehensive Support for Graphical,
Highly-Interactive User Interfaces." IEEE Computer, 1990.
23(11): pp. 71-85.

29. Nelson, T. "A File Structure for the Complex, the Changing, and the
Indeterminate," in Proceedings ACM National Conference. 1965. pp.
84-100.

30. Newman, W.M. "A System for Interactive Graphical Programming," in AFIPS
Spring Joint Computer Conference. 1968. 28. pp. 47-54.

31. Nielsen, J., Multimedia and Hypertext: the Internet and Beyond.
1995, Boston: Academic Press Professional.

32. Palay, A.J., et al. "The Andrew Toolkit - An Overview," in
Proceedings Winter Usenix Technical Conference. 1988. Dallas, Tex.
pp.
9-21.

33. Press, L., "Before the Altair: The History of Personal Computing."
Communications of the ACM, 1993. 36(9): pp. 27-33.

34. Reddy, D.R., "Speech Recognition by Machine: A Review," in Readings
in
Speech Recognition, A. Waibel and K.-F. Lee, Editors. 1990, Morgan
Kaufmann: San Mateo, CA. pp. 8-38.

35. Reddy, R., "To Dream the Possible Dream (Turing Award Lecture)."
Communications of the ACM, 1996. 39(5): pp. 105-112.

36. Robertson, G., Newell, A., and Ramakrishna, K., ZOG: A Man-Machine
Communication Philosophy . Carnegie Mellon University Technical Report
Report, Number, August, 1977.

37. Ross, D. and Rodriguez, J. "Theoretical Foundations for the Computer-Aided
Design System," in AFIPS Spring Joint Computer Conference. 1963.
23. pp. 305-322.

38. Rudisill, M., et al., Human-Computer Interface Design: Success
Stories, Emerging Methods, and Real-World Context. 1996, San Francisco:
Morgan Kaufmann Publishers.

39. Scheifler, R.W. and Gettys, J., "The X Window System." ACM
Transactions on Graphics, 1986. 5(2): pp. 79-109.

40. Shneiderman, B., "Direct Manipulation: A Step Beyond Programming
Languages." IEEE Computer, 1983. 16(8): pp. 57-69.

41. Smith, D.C., Pygmalion: A Computer Program to Model and Stimulate
Creative Thought. 1977, Basel, Stuttgart: Birkhauser Verlag. PhD Thesis,
Stanford University Computer Science Department, 1975.

42. Smith, D.C., et al. "The Star User Interface: an Overview," in
Proceedings of the 1982 National Computer Conference. 1982. AFIPS.
pp.
515-528.

43. Stallman, R.M., Emacs: The Extensible, Customizable, Self-Documenting
Display Editor . MIT Artificial Intelligence Lab Report, Number, Aug,
1979, 1979.

44. Sutherland, I.E. "SketchPad: A Man-Machine Graphical Communication System,"
in AFIPS Spring Joint Computer Conference. 1963. 23. pp.
329-346.

45. Swinehart, D., et al., "A Structural View of the Cedar Programming
Environment." ACM Transactions on Programming Languages and
Systems, 1986. 8(4): pp. 419-490.

46. Swinehart, D.C., Copilot: A Multiple Process Approach to Interactive
Programming Systems. PhD Thesis, Computer Science Department Stanford
University, 1974, SAIL Memo AIM-230 and CSD Report STAN-CS-74-412.

47. Teitelman, W., "A Display Oriented Programmer's Assistant."
International Journal of Man-Machine Studies, 1979. 11: pp.
157-187. Also Xerox PARC Technical Report CSL-77-3, Palo Alto, CA, March
8,
1977.

48. Tolliver, B., TVEdit . Stanford Time Sharing Memo Report, Number,
March, 1965.

49. van Dam, A., et al. "A Hypertext Editing System for the 360,"
in
Proceedings Conference in Computer Graphics. 1969. University of
Illinois.

50. van Dam, A. and Rice, D.E., "On-line Text Editing: A Survey."
Computing Surveys, 1971. 3(3): pp. 93-114.

51. Williams, G., "The Lisa Computer System." Byte Magazine,
1983. 8(2): pp. 33-50.

52. Williams, G., "The Apple Macintosh Computer." Byte, 1984.
9(2): pp. 30-54.