Interactivity Reading
Group
Hangin' out at
Starbucks in the Ferg on Tuesday evenings at 5:30pm
Readings
- Tuesday, January 18
- Tuesday, January 25
- Meeting cancelled - Jamie and Joe are giving the Excel Workshop
- Tuesday, February 1
- Haslam &
McGarty (courtesy of Joe and Chris)
- More detailed review of the "Fundamentals of Interactivity"
paper
- Tuesday, February 8
- Haslam & McGarty paper (assigned Feb 1)
- Tuesday, February 15
- Chapters on IRT and CAT (delivered on paper)
- Tuesday, February 22
- No new readings (meeting on 2/15 was cancelled)
- Tuesday, March 1
- Again, no new readings. We still need to go through the
Rawlings & Andrieu paper and the IRT and CAT chapters
- Tuesday, March 17
- We will be meeting in Jamie's office (192 GP) to look at "The
Incredible Machine" and an old version of the Erasmatron.
- Tuesday, March 22
- Tuesday, April 5
- No new readings. Will discuss the two interactivity
articles assigned on 3/22.
- Tuesday, April 12
- Individual readings on learning theory (Jamie), problem solving
(Jana), and multiple intelligences (Michelle and Chris)
- Tuesday, April 19
Notes from Meetings
- Tuesday, January 18
- Attendees: Jamie, Joe, Michelle, Chris, Jana, Aimee
- Jamie emphasized that although our readings are going to
emphasize interactions with other people, he also feels that observing
interactions with an environment would be just as valuable. The
key is to examine how people choose to manipulate things around them or
move to alternate enviornments to achieve their goal, rather than
either asking them questions or observing their behavior in a
predefined situation.
- Michelle suggested that "speaking the same language" is going
to be important if we want to work with older adults. Older
people are typically unfamiliar with computers, and so will be unlikely
to work with them naturally. We may have to develop real,
physical environments for them to interact with.
- Joe suggested that using more realistic environments can make
the assessment situation more familiar and therefore more valid.
- The links between interactivity and computer adaptive testing
(CAT) came up several times. Both share an important
characteristic - both change in response to the performance of the
participant. CAT lacks several other aspects of interactivity,
but it also demonstrates how even limited interactivity can be
beneficial. We might apply the basic ideas of CAT to the
assessment of psychological phenomena - we might use information
collected early in the assessment to affect the way that we choose to
proceed with the assessment. This brings up some concerns of
construct validity, but I don't think that's a problem.
Ultimately I think that using the adaptive method is probably more
valid, and we could plan on a line of research specifically testing
this.
- Jamie thinks that there may be two broad uses of
interactivity. First, interactivity may provide us with better
ways to measure phenomena or characteristics that are already examined
by existing scales or procedures. Second, interactivity may
provide us with the opportunity to measure constructs that are
unavailable to static measures. Two examples of this second type
might be a measure of how people respond to feedback, and how well
people make use of a social network.
- Interactivity might provide us a way to integrate the best
ideas of quantitative and qualitative research.
- Tuesday, February 1
- Attendees: Jamie, Chris, Michelle, Jana, Joe
- Much of the discussion centered on "minimally interactive"
assessment techniques. We noted that simply including a cycle
into the way that we measure something is an advance over what is
currently being done. "Including a cycle" basically means that we
alter the way we decide to measure something based on the responses
that we receive from a subject.
- Chris mentioned the possibility of including interactivity as
an IV in studies. There may be a bit of an issue of
comparability, but it could lend support to interactive measures if we
can experimentally show that they perform differently (and hopefully
better than) their non-interactive counterparts.
- We discussed to major categories of minimally interactive
measures.
- Forked scales, where the particular measure we choose to
measure a construct depends on some other variable that we
measure. For example, we might choose to measure depression using
one scale for women and a different scale for men.
- Computer adaptive measures, where the assessment is performed
on a computer and the computer intelligently chooses which question to
ask based on prior responses. The computer can develop hypotheses
about the respondent based on prior items which it then tests using
specific questions.
- Forked scales will require a lot of prep work. It will be
necessary to demonstrate that the two different versions of the scales
are measuring the same thing. This is difficult because you would
only choose to use forked scales when the different versions of the
scales work differently for different people. We discussed three
different ways that this might be accomplished.
- Relate the different versions of the scale to a gold
standard. If there is a time-intensive method that will tell us
the "true" score on the construct (and is not influenced by the type of
person we are measuring), we can show that the different measures have
approximately equivalent relations to the true score.
- Show that the different versions have similar relations to
other variables. If scale 1 has the same relations to other
variables in group 1 as scale 2 has in group 2, we have evidence that
scale 1 and scale 2 may be measuring the same construct.
- Show that they different versions have similar
distributions. If we assume that the actual distribution of the
construct we are trying to measure is the same across our groups, then
finding that the different scales produce similar distributions
provides evidence that they are measuring the same construct.
- When demonstrating the validity and reliability of computer
adaptive measures, we may want to treat the process by which we obtain
the measures as a "black box." Essentially we might just want to
show that the resulting measurement can predict other variables and is
consistent over repeated assessments without trying to validate the
processes by which we obtain those measures. The fact that the
measurement method is different for each subject sort of prevents us
from having to demonstrate that it's doing the same thing in each case,
since this would be impossible to do.
- Jamie mentioned a storytelling architecture that seems to
incorporate some of the hypothesis-testing ideas that we discussed in
relation to the computer adaptive measures.
- Joe suggested that we could examine the specific path by which
the result is chosen to see if this provides any useful information.
- These minimally interactive techniques can probably be applied
more immediately, and so we might want to focus on them for our grant.
- Researchers on group processes have spent years investigating
the way that group members interact with each other. We might
want to look to this literature when we want to develop ways of looking
at more complicated interactions.
- We discussed trying to measure people's ability to manipulate
and combine elements of their environment, similar to the way that
MacGuyver invents things on his TV show or how the contestants on
"Junkyard Wars" have to construct unusual devices. This relates
to research on insight, but is a bit more mundane and practical.
Most insight problems seem to focus on a single instance of thinking of
something in a new way. It might be more useful to try to measure
a subject's ability to put things together to solve a task that doesn't
necessarily involve any insight. Jamie brought up the example of
writing a computer program, but anything where you build something up
from component parts would also work. This would include things
like an architect designing a building or an engineer designing an
airplane. Michelle suggested that we might look for research on
"problem solving" to see what has already been done. Jana
indicated that some research on this may have been done in Germany.
- Tuesday, February 8
- Attendees: Jamie, Chris, Michelle, Aimee, Joe, Kelly
- Spend some time revisiting the three research programs that we
have currently identified.
- Using/testing/validating forked scales
- Performing adaptive measurement of psychological constructs
- Measuring a general ability to combine objects to solve
problems (Junkyard wars ability)
- As part of our discussion, we noticed that when using a forked
scale, we cannot interpret differences among the groups that define the
forks in a meaningful fashion. It is debatable whether we will be
able to interpret the results of interaction effects. An
interaction specifically would test whether the differences between the
levels of an IV is the same across the different forks. The
question is whether having an incomparability between the groups will
invalidate looking at interactive differences. We did not resolve
the issue (mainly because it would be boring for non-statoids), but
Jamie did suggest that we can probably could if we wanted to dedicate
some thought to it.
- Aimee commented on the importance of using a task for the
junkyard wars ability that was not itself biased in favor of specific
genders or specific races. However, it was also mentioned that we
might observe real differences between groups on this variable.
In this case we must really make sure that there is no logical reason
to suspect that particular demographic groups might have greater
familiarity with the elements of the task than others, or else any
observed group differences will be called into question.
- It appears that there may be strong ties between item response
theory and computer adaptive testing, so we should probably become more
familiar with IRT.
- Jamie reiterated that we may want to draw from the research on
group interaction when we want to use measures that actually examine
interactive process and not just the product of an interaction, or some
summary evaluation of the quality of the interaction.
- We discussed the implications of the Haslam & McGarty
article for our research. The development of these interactive
measures would definitely fit into the category of "uncertainty
expansion" since we will be using new measures and potentially
examining new psychological constructs. However, because of this
we might expect that our tasks and measures might not be as reliable or
valid at the start. We may therefore want to emphasize what our
new measures can predict or describe above and beyond what is captured
by static measures, since they will be psychometrically less sound at
first.
- Tuesday, February 22
- Attendees: Jamie, Michelle, Kelly, Aimee, Chris
- We spend a decent part of this evening discussing the
possibility of using interactive storytelling in psychological
research. Interactive storytelling gives people a lot of freedom
with regard to the choices they can make, but will consequently produce
data that is more complex and more difficult to analyze. Michelle
specifically discussed using interactive storytelling to assess
motivational differences. For example, younger adults tend to
expand their social networks while older adults tend to restrict their
social networks. You might consider measuring these motives by
seeing whether people choose to follow more or less social elements of
a story.
- Kelly discussed providing an interactive framework to measure
decision-making strategies. She mentioned the "shopping list"
technique, where you give people a broad goal of what they need to
purchase and then observe how they develop and implement a shopping
plan.
- We discussed the role of interactivity in life-decision
tasks. An individuals ability to function socially is in many
ways dependent on their abilities to successfully interact with people
and objects in the environment. We discussed specific examples of
the role of interactivity in the social skills training group at the
PDD clinic, and the importance of successful interactions with the
environment for older adults.
- Jamie discussed the possible applicability of virtual reality,
which is growing in popularity in social psychology. As part of
our discussion of this, we came to realize that it may be more
important to be able to measure a person's bodily movement than it
would be to actually present items in virtual reality. Chris
mentioned that devices to measure body movements are actually much
cheaper than a full VR setup.
- Jamie mentioned that there is software that will record what is
displayed on a computer monitor as an avi file. This works
reasonably well, although if there is a lot going on you get noticeable
pauses as the computer saves the information. It is also possible
to get a graphics card that will send the output from the computer to a
VCR.
- We discussed the possibility of developing quality control
measures for interactive storytelling. Right now there are very
few interactive stories available, and people don't have good ways of
evaluating them.
- Chris mentioned an interesting in studying
creativity/interactivity in real time. He discussed the
importance of actually being able to examine, handle, and interact with
a tool to be able to fully understand how it can be used. There
might be some ties to implicit learning there - you might develop ideas
of how something can be used not through logical inference, but instead
through a more automatic process keyed by implicit associations.
- We might want to consider how to measure the psychological
impacts of an interactive experience. Crawford says that
interactive stories and games are better, but there really isn't any
data on that. They are likely better in some ways and worse in
others - identifying these relations could be useful when trying to
sell the idea of interactivity. An interesting side note that we
discussed was that people often come away from the CAT version of the
GRE feeling unhappy about the experience. This could be caused by
a lack of familiarity, but could also be related to other aspects of
interactivity. Perhaps the evaluation seems more personal and
more invasive.
- We discussed the specific interests of individual group members.
- Michelle: forked scales, McGuyver task, interactive
storytelling
- Chris: CAT, McGuyver task
- Aimee: none
- Kelly: CAT, McGuyver task, interactive storytelling
- Tuesday, March 1
- Attendees: Jamie, Aimee, Joe, Jana, Michelle
- Continued our discussion of the specific interests of
individual group members
- Joe: McGuyver task, interactive storytelling, forked
scales
- Jana: MacGuyver task, combination of quantitative and
qualitative methods
- We discussed the Rawlings & Andrieu article on interactive
storytelling. It provided a decent insight as to what people are
trying to do with interactive storytelling. One of the main
reasons we were interested in that article was that they specifically
mention that they incorporate hypothesis testing in their
framework. Unfortunately the authors did not describe this system
in great detail.
- Joe mentioned that he has a friend that is working on some
intelligent decision-making system for the millitary. He might be
a useful person to talk to about the idea of developing a
hypothesis-testing scale.
- We might consider using a neural network to guide responses on
the hypothesis-guessing scale.
- We then started our discussion of the specific lines of
research.
- Forked scales.
At this time we don't have a specific scale for which we want to create
different versions for different groups. It may be best to put
this line on hold until someone comes up with a specific application.
- CAT. There is
currently not a lot of research applying CAT to psychological
measurement. However, Jamie mentioned that he did see a link on
the CAT central website (now listed below in the links section) to a
group that got a grant to study applications of CAT to the assessment
of mental health. To find this, go to CAT central and click on
"Current CAT research programs." The research project of interest
is listed near the top.
- MacGuyver task.
Jamie suggested that we start by looking at a computer game called "The
incredible machine" that is based on combining a set of components to
accomplish an overall goal. We discussed that there is probably
some existing research in creativity and problem solving that is
related to our ideas. Aimee suggested that we might talk to Bev
RE or Tom Ward to see if they know of any literature that we should be
looking at. Jamie later had a chance to talk to both of them, and
they seem interested in our ideas. We might consider setting up a
meeting with both of them in the next week or two.
- Interactive storytelling.
Jamie mentioned that he should provide more readings on this topic so
that everyone can have a better idea of the issues in the area.
He also mentioned that it might help to actually look at some existing
interactive stories. Right now the field is so new that most of
the people are talking about ways to accomplish interactive
storytelling without actually providing any interactive stories.
However, Jamie mentioned that he had used Crawford's storytelling
engine "Erasmatron" in the past, so that might be a good place to start.
- Tuesday, March 17
- Attendees: Jamie, Joe, Aimee, Chris, Jana
- We examined the Incredible Machine program. For next week
people should try to at least make their way through the tutorial
puzzles.
- We talked a little bit about how the program could be
used. Some suggestions are:
- The time to solve puzzles could be used as a criterion to
test a scale measuring McGuyver ability
- We could directly examine the process by which people solve
problems.
- We could look at how people behave during practice to see how
this affects a subject's ability to solve problems.
- We could manipulate various aspects of practice and see how
this affects a subject's ability to solve problems.
- Tuesday, March 22
- Attendees: Jamie, Joe, Kelly, Jana, Michelle
- Postponed discussion of the incredible machine and interactive
storytelling to give people a chance to catch up.
- Talked about applying CAT to psychology. Some fruitful
areas might be the Big 5 personality measurement, and the MMPI.
Jamie agreed to talk to the researchers listed on the CAT Central
website who are currently applying CAT to mental health assessment.
- We came up with a big, crazy grant idea. Originally we
discussed picking a specific area of psychology and developing CAT
versions of the most commonly used scales in the area. Jamie then
mentioned that it would be handy to implement this in Java, since the
programs would then be accessible without regard to the platform that
people use. If we were going to write the programs in Java
anyway, we decided that we might as well make them available on a
website. Researchers could enter their email address and a
subject number, and the website would then take care of measuring their
subjects. The data would automatically be sent to the provided
email address. From there we discussed the possibility of
providing people with a large list of scales, allowing them to check
the ones that they want, and having the website generate a customized
survey for them. We could even include non-CAT scales, although
there would definitely be an advantage of creating CAT versions, since
it would allow people to use more scales in the same time period.
This service would encourage people who are not computer-savvy to use
computerized testing, facilitating their research and saving tons of
paper. It would also help the researchers whose scales are posted
on the site, since people would be more likely to include a scale in
their research if it only required them to check an additional
box. We would of course make the citations for the scales
available on the website, and the website itself would probably get a
lot of citations as well. The task itself would be pretty huge -
we'd need to collect scales, do all the IRT work, do the CAT
programming, set up the website, and write a program to take the data
from the emails and convert it to an excel file. We would
probably need to hire a professional programmer. We would want to
start in a specific topic area, and get one or more collaborators who
have developed scales in that area. We would also want to have
someone experienced in CAT on our team. That either means getting
one of us trained, or bringing in a CAT collaborator.
- Friday, March 25
- Attendees: Jamie, Bev R-E, Aimee, Joe, Chris, Jana,
Michelle
- We discussed some of our interactivity ideas with Bev
R-E. We primarily focused on the possibility of working with TIM
(The Incredible Machine).
- Bev mentioned that the problems presented by TIM do have some
strong similarities to those investigated in the classic problem
solving literature. We should therefore try to become more
familiar with this literature. She also mentioned that our work
might relate to the research on "multiple intelligences." We
should specifically check this literature to see what they have done to
represent mechanical ability.
- The standard way of using TIM would be to provide people with a
puzzle and then ask people to use a set of parts to create a
solution. We might alternatively think about putting people in
front of the puzzle generator and ask them to come up with their own
machines. We could also build this into a puzzle where we select
specific parts for people to use (rather than actually putting people
into the puzzle creation part of the program). We might also
think about asking people to come up with a puzzle that we would give
to other participants.
- We could potentially look at the relation between sensation
seeking and exploratory tendencies with TIM. Could also determine
situational factors that might encourage people to do more exploring,
and look at the relation between exploration and performance.
Some possible measures of exploration would be the number of different
parts that are examined, how many non-standard features of a puzzle are
used, how often people look at hints, and how often people read
information about a part.
- We could examine how training affects peoples' ability to solve
puzzles. We could train people on one collection of parts and
then examine their ability to work with entirely new parts.
- Bev's own work on wayfinding would likely be a good place to
look at the role of interactivity. Most of her experiments are
interactive in one way or another.
- We should take a look at J. J. Gibson's work on ecological
perception to get a better idea of how people might think about and use
different components.
- Tuesday, April 5
- Attendees: Jamie, Michelle, Jana, Chris
- Discussed the CAT grant idea some more. If we are able to
actually develop the website as we propose, Chris suggested that we
might obtain a second grant to provide multilingual versions of the
scales. Along a similar vein, Michelle suggested that we might
provide voice recordings of the surveys to make them easier to apply to
young children or illiterate populations.
- Jamie mentioned that he will offer a CAT seminar in the fall,
and that anyone interested in working on the grant should attend that
seminar.
- We then spent some time talking about TIM. Jana mentioned
that she sometimes became frustrated when she had to quit a puzzle
halfway through and then had to start over entirely from the
beginning. The effects of starting over might be something that
would be interesting to look at. In real life we sometimes have
the ability to put a problem aside for awhile, but there are certainly
times when we have to recreate something from scratch.
- Performing well at TIM definitely appears to involve both
implicit and explicit learning (although talking about it as
declarative and procedural learning would be better). The hints
that appear in the puzzles provide declarative learning, while the
practical experience of moving things around would provide procedural
learning.
- Before getting much further into the TIM studies, we decided
that we probably need to learn more about several areas of cognitive
psychology, including:
- Problem solving
- Multiple intelligences
- Learning theory
- Creativity
- We decided to give each person the responsibility of
investigating one of the topics above. We decided to skip
creativity for now, since it's probably the part of the literature that
we already know the best. Jamie will investigate learning theory,
Jana will investigate problem solving, and both Michelle and Chris will
investigate multiple intelligences. For next week all that we
hope to do is to at least identify one or more simple review papers or
chapters on the topics.
- Tuesday, April 12
- Attendees: Jamie, Joe, Jana, Michelle
- We started by discussing the future of our projects.
- The work on forked scales is on indefinite hold until someone
comes up with a specifc scale and a specific population they want to
examine.
- The work on interactive storytelling is on hold until we are
able to obtain an actual interactive story to work with. Jamie
plans to continue his conversations with Chris Crawford, and it is
possible that some new ideas will come from that. However, the
idea of examining behavior in interactive stories itself will likely
not come to pass for several years.
- We are going to hold off on doing much more with the CAT
project until we all have a chance to learn more about CAT. Jamie
plans to offer a CAT seminar in the fall, and we will probably start
working seriously on the grant in the spring. In the meantime we
should just try to read the CAT primer book and decide what specific
content area we want to work in.
- The TIM/MacGuyver task is therefore the only project that
doesn't have any immediate holds. So for now we plan to
concentrate on learning more about research related to this task and
will take up whichever experimental idea we come across first.
- Jana found a number of articles related to problem solving,
some of which seemed to have direct relevance to our research.
However, most of them were about 10 years old, and many were not
available in our library. Jana will try to get the articles
through ILL, and Joe is going to look through some recent handbooks to
see if he can find a summary of the literature.
- Jamie discovered that the specific aspect of learning that
interests us is usually called "cognitive skill acquisition" in the
literature. Research by Rich Carlson seems to be relevant to our
research. Jamie plans to chat with Rich sometime over this
summer, and we will see what he has to say about our work.
- Joe has decided that he will start investigating problem
solving along with Jana.
- Fall 2005 Meeting
- Attendees: Jamie, Chris, Joe
- Discussed the "History of Thinking" article.
- It clearly has some strong relations to existing dual-process
models in psychology, though with some differences.
"Pattern-recognition" thinking corresponds to automatic processing
while "Sequential" or "Verbal" thinking corresponds to controlled
processing. There are some aspects of psychological models that were
not included in this discussion. Existing models emphasize the
interactions between automatic and controlled processing, such that
automatic processing can bias controlled processing, and conscious
processing can direct our attention to specific stimuli, affecting what
we automatically process. They also present these two types of
processing as occurring simultaneously, which was not really discussed
in this article (although it isn't inconsistent with it, either).
- We discussed the implication of the different ways of
thinking for teaching. As Chris Crawford points out, most teaching
relies heavily on sequential thinking, although there are clearly some
ways that pattern-recognition thinking could be brought in. Anything
organized around providing "guided experiences" would provide people
with the opportunity to use automatic processing to learn from their
experiences. This type of teaching, however, is bound to be much less
efficient overall, although it might be necessarily to learn certain
types of skills. To try to make this type of learning more efficient we
might consider how we might create "summarized" versions of
experiences, basically guiding people though a nonrandom selection of
experiences that would provide them with the most important
information. The key would be to identify what are the most important
things that people might learn through this type of learning, and what
aspects of the experience are critical. We discussed how this line of
thought might have some implications for the way upper-level
undergraduate and graduate seminars are structured. It might be
worthwhile to provide a larger perspective of how different articles
link to each other instead of organizing discussions around specific
topics. There are some textbooks that try to do this explicitly (Smith
& Mackie social psych text) but these connections are often ignored
in favor of providing greater content. It might be useful for
instructors to indicate some of these links directly on their course
syllabi.
- Information that is embedded in a network is more
generalizeable, since it links an event to its causes and its effects.
This view of the process makes it easier to draw parallels to other
circumstances.
- We discussed the possiblity that different types of
processing might differ in terms of the pleasure that they provide.
Some people might receive more pleasure out of performing a
pattern-recognition task, while others may get more pleasure out of
sequential learning. This might be what the well-known "Need for
Cognition" scale is getting at, even more than a tendency to use these
two different types of processes. This might have some implications for
enjoyment of and performance on TIM. We discussed a possible new
experiment. In the Ashby categorization task, people are asked to learn
about a categorization that is either verbalizeable or not
verbalizeable. We might look to see if people enjoy one type of task
more than the other. The enjoyment may specificly depend on other
variables, such as Need for Cognition or gender. We might also see
whether the pleasure rating is related to various performance measures.
The basic design of the study would be to have people perform the NFC
task, then the categorization task, then the pleasure rating task. We
might consider giving the NFC task in a prior session to prevent it
from affecting the pleasure ratings, although the categorization task
itself might be distracting enough to prevent any biases.
Links