Interactivity Reading Group

Notes from Meetings

Tuesday, January 18

Attendees: Jamie, Joe, Michelle, Chris, Jana, Aimee
Jamie emphasized that although our readings are going to emphasize interactions with other people, he also feels that observing interactions with an environment would be just as valuable. The key is to examine how people choose to manipulate things around them or move to alternate enviornments to achieve their goal, rather than either asking them questions or observing their behavior in a predefined situation.

Michelle suggested that "speaking the same language" is going to be important if we want to work with older adults. Older people are typically unfamiliar with computers, and so will be unlikely to work with them naturally. We may have to develop real, physical environments for them to interact with.
Joe suggested that using more realistic environments can make the assessment situation more familiar and therefore more valid.
The links between interactivity and computer adaptive testing (CAT) came up several times. Both share an important characteristic - both change in response to the performance of the participant. CAT lacks several other aspects of interactivity, but it also demonstrates how even limited interactivity can be beneficial. We might apply the basic ideas of CAT to the assessment of psychological phenomena - we might use information collected early in the assessment to affect the way that we choose to proceed with the assessment. This brings up some concerns of construct validity, but I don't think that's a problem. Ultimately I think that using the adaptive method is probably more valid, and we could plan on a line of research specifically testing this.
Jamie thinks that there may be two broad uses of interactivity. First, interactivity may provide us with better ways to measure phenomena or characteristics that are already examined by existing scales or procedures. Second, interactivity may provide us with the opportunity to measure constructs that are unavailable to static measures. Two examples of this second type might be a measure of how people respond to feedback, and how well people make use of a social network.
Interactivity might provide us a way to integrate the best ideas of quantitative and qualitative research.

Tuesday, February 1

Attendees: Jamie, Chris, Michelle, Jana, Joe
Much of the discussion centered on "minimally interactive" assessment techniques. We noted that simply including a cycle into the way that we measure something is an advance over what is currently being done. "Including a cycle" basically means that we alter the way we decide to measure something based on the responses that we receive from a subject.
Chris mentioned the possibility of including interactivity as an IV in studies. There may be a bit of an issue of comparability, but it could lend support to interactive measures if we can experimentally show that they perform differently (and hopefully better than) their non-interactive counterparts.
We discussed to major categories of minimally interactive measures.

Forked scales, where the particular measure we choose to measure a construct depends on some other variable that we measure. For example, we might choose to measure depression using one scale for women and a different scale for men.
Computer adaptive measures, where the assessment is performed on a computer and the computer intelligently chooses which question to ask based on prior responses. The computer can develop hypotheses about the respondent based on prior items which it then tests using specific questions.

Forked scales will require a lot of prep work. It will be necessary to demonstrate that the two different versions of the scales are measuring the same thing. This is difficult because you would only choose to use forked scales when the different versions of the scales work differently for different people. We discussed three different ways that this might be accomplished.

Relate the different versions of the scale to a gold standard. If there is a time-intensive method that will tell us the "true" score on the construct (and is not influenced by the type of person we are measuring), we can show that the different measures have approximately equivalent relations to the true score.
Show that the different versions have similar relations to other variables. If scale 1 has the same relations to other variables in group 1 as scale 2 has in group 2, we have evidence that scale 1 and scale 2 may be measuring the same construct.
Show that they different versions have similar distributions. If we assume that the actual distribution of the construct we are trying to measure is the same across our groups, then finding that the different scales produce similar distributions provides evidence that they are measuring the same construct.

When demonstrating the validity and reliability of computer adaptive measures, we may want to treat the process by which we obtain the measures as a "black box." Essentially we might just want to show that the resulting measurement can predict other variables and is consistent over repeated assessments without trying to validate the processes by which we obtain those measures. The fact that the measurement method is different for each subject sort of prevents us from having to demonstrate that it's doing the same thing in each case, since this would be impossible to do.
Jamie mentioned a storytelling architecture that seems to incorporate some of the hypothesis-testing ideas that we discussed in relation to the computer adaptive measures.
Joe suggested that we could examine the specific path by which the result is chosen to see if this provides any useful information.
These minimally interactive techniques can probably be applied more immediately, and so we might want to focus on them for our grant.
Researchers on group processes have spent years investigating the way that group members interact with each other. We might want to look to this literature when we want to develop ways of looking at more complicated interactions.
We discussed trying to measure people's ability to manipulate and combine elements of their environment, similar to the way that MacGuyver invents things on his TV show or how the contestants on "Junkyard Wars" have to construct unusual devices. This relates to research on insight, but is a bit more mundane and practical. Most insight problems seem to focus on a single instance of thinking of something in a new way. It might be more useful to try to measure a subject's ability to put things together to solve a task that doesn't necessarily involve any insight. Jamie brought up the example of writing a computer program, but anything where you build something up from component parts would also work. This would include things like an architect designing a building or an engineer designing an airplane. Michelle suggested that we might look for research on "problem solving" to see what has already been done. Jana indicated that some research on this may have been done in Germany.

Tuesday, February 8

Attendees: Jamie, Chris, Michelle, Aimee, Joe, Kelly
Spend some time revisiting the three research programs that we have currently identified.

Using/testing/validating forked scales
Performing adaptive measurement of psychological constructs
Measuring a general ability to combine objects to solve problems (Junkyard wars ability)

As part of our discussion, we noticed that when using a forked scale, we cannot interpret differences among the groups that define the forks in a meaningful fashion. It is debatable whether we will be able to interpret the results of interaction effects. An interaction specifically would test whether the differences between the levels of an IV is the same across the different forks. The question is whether having an incomparability between the groups will invalidate looking at interactive differences. We did not resolve the issue (mainly because it would be boring for non-statoids), but Jamie did suggest that we can probably could if we wanted to dedicate some thought to it.
Aimee commented on the importance of using a task for the junkyard wars ability that was not itself biased in favor of specific genders or specific races. However, it was also mentioned that we might observe real differences between groups on this variable. In this case we must really make sure that there is no logical reason to suspect that particular demographic groups might have greater familiarity with the elements of the task than others, or else any observed group differences will be called into question.
It appears that there may be strong ties between item response theory and computer adaptive testing, so we should probably become more familiar with IRT.
Jamie reiterated that we may want to draw from the research on group interaction when we want to use measures that actually examine interactive process and not just the product of an interaction, or some summary evaluation of the quality of the interaction.
We discussed the implications of the Haslam & McGarty article for our research. The development of these interactive measures would definitely fit into the category of "uncertainty expansion" since we will be using new measures and potentially examining new psychological constructs. However, because of this we might expect that our tasks and measures might not be as reliable or valid at the start. We may therefore want to emphasize what our new measures can predict or describe above and beyond what is captured by static measures, since they will be psychometrically less sound at first.

Tuesday, February 22

Attendees: Jamie, Michelle, Kelly, Aimee, Chris
We spend a decent part of this evening discussing the possibility of using interactive storytelling in psychological research. Interactive storytelling gives people a lot of freedom with regard to the choices they can make, but will consequently produce data that is more complex and more difficult to analyze. Michelle specifically discussed using interactive storytelling to assess motivational differences. For example, younger adults tend to expand their social networks while older adults tend to restrict their social networks. You might consider measuring these motives by seeing whether people choose to follow more or less social elements of a story.
Kelly discussed providing an interactive framework to measure decision-making strategies. She mentioned the "shopping list" technique, where you give people a broad goal of what they need to purchase and then observe how they develop and implement a shopping plan.
We discussed the role of interactivity in life-decision tasks. An individuals ability to function socially is in many ways dependent on their abilities to successfully interact with people and objects in the environment. We discussed specific examples of the role of interactivity in the social skills training group at the PDD clinic, and the importance of successful interactions with the environment for older adults.
Jamie discussed the possible applicability of virtual reality, which is growing in popularity in social psychology. As part of our discussion of this, we came to realize that it may be more important to be able to measure a person's bodily movement than it would be to actually present items in virtual reality. Chris mentioned that devices to measure body movements are actually much cheaper than a full VR setup.
Jamie mentioned that there is software that will record what is displayed on a computer monitor as an avi file. This works reasonably well, although if there is a lot going on you get noticeable pauses as the computer saves the information. It is also possible to get a graphics card that will send the output from the computer to a VCR.
We discussed the possibility of developing quality control measures for interactive storytelling. Right now there are very few interactive stories available, and people don't have good ways of evaluating them.
Chris mentioned an interesting in studying creativity/interactivity in real time. He discussed the importance of actually being able to examine, handle, and interact with a tool to be able to fully understand how it can be used. There might be some ties to implicit learning there - you might develop ideas of how something can be used not through logical inference, but instead through a more automatic process keyed by implicit associations.
We might want to consider how to measure the psychological impacts of an interactive experience. Crawford says that interactive stories and games are better, but there really isn't any data on that. They are likely better in some ways and worse in others - identifying these relations could be useful when trying to sell the idea of interactivity. An interesting side note that we discussed was that people often come away from the CAT version of the GRE feeling unhappy about the experience. This could be caused by a lack of familiarity, but could also be related to other aspects of interactivity. Perhaps the evaluation seems more personal and more invasive.
We discussed the specific interests of individual group members.

Michelle: forked scales, McGuyver task, interactive storytelling
Chris: CAT, McGuyver task
Aimee: none
Kelly: CAT, McGuyver task, interactive storytelling

Tuesday, March 1

Attendees: Jamie, Aimee, Joe, Jana, Michelle
Continued our discussion of the specific interests of individual group members

Joe: McGuyver task, interactive storytelling, forked scales
Jana: MacGuyver task, combination of quantitative and qualitative methods

We discussed the Rawlings & Andrieu article on interactive storytelling. It provided a decent insight as to what people are trying to do with interactive storytelling. One of the main reasons we were interested in that article was that they specifically mention that they incorporate hypothesis testing in their framework. Unfortunately the authors did not describe this system in great detail.
Joe mentioned that he has a friend that is working on some intelligent decision-making system for the millitary. He might be a useful person to talk to about the idea of developing a hypothesis-testing scale.
We might consider using a neural network to guide responses on the hypothesis-guessing scale.
We then started our discussion of the specific lines of research.

Forked scales. At this time we don't have a specific scale for which we want to create different versions for different groups. It may be best to put this line on hold until someone comes up with a specific application.
CAT. There is currently not a lot of research applying CAT to psychological measurement. However, Jamie mentioned that he did see a link on the CAT central website (now listed below in the links section) to a group that got a grant to study applications of CAT to the assessment of mental health. To find this, go to CAT central and click on "Current CAT research programs." The research project of interest is listed near the top.
MacGuyver task. Jamie suggested that we start by looking at a computer game called "The incredible machine" that is based on combining a set of components to accomplish an overall goal. We discussed that there is probably some existing research in creativity and problem solving that is related to our ideas. Aimee suggested that we might talk to Bev RE or Tom Ward to see if they know of any literature that we should be looking at. Jamie later had a chance to talk to both of them, and they seem interested in our ideas. We might consider setting up a meeting with both of them in the next week or two.
Interactive storytelling. Jamie mentioned that he should provide more readings on this topic so that everyone can have a better idea of the issues in the area. He also mentioned that it might help to actually look at some existing interactive stories. Right now the field is so new that most of the people are talking about ways to accomplish interactive storytelling without actually providing any interactive stories. However, Jamie mentioned that he had used Crawford's storytelling engine "Erasmatron" in the past, so that might be a good place to start.

Tuesday, March 17

Attendees: Jamie, Joe, Aimee, Chris, Jana
We examined the Incredible Machine program. For next week people should try to at least make their way through the tutorial puzzles.
We talked a little bit about how the program could be used. Some suggestions are:

The time to solve puzzles could be used as a criterion to test a scale measuring McGuyver ability
We could directly examine the process by which people solve problems.
We could look at how people behave during practice to see how this affects a subject's ability to solve problems.
We could manipulate various aspects of practice and see how this affects a subject's ability to solve problems.

Tuesday, March 22

Attendees: Jamie, Joe, Kelly, Jana, Michelle
Postponed discussion of the incredible machine and interactive storytelling to give people a chance to catch up.
Talked about applying CAT to psychology. Some fruitful areas might be the Big 5 personality measurement, and the MMPI. Jamie agreed to talk to the researchers listed on the CAT Central website who are currently applying CAT to mental health assessment.
We came up with a big, crazy grant idea. Originally we discussed picking a specific area of psychology and developing CAT versions of the most commonly used scales in the area. Jamie then mentioned that it would be handy to implement this in Java, since the programs would then be accessible without regard to the platform that people use. If we were going to write the programs in Java anyway, we decided that we might as well make them available on a website. Researchers could enter their email address and a subject number, and the website would then take care of measuring their subjects. The data would automatically be sent to the provided email address. From there we discussed the possibility of providing people with a large list of scales, allowing them to check the ones that they want, and having the website generate a customized survey for them. We could even include non-CAT scales, although there would definitely be an advantage of creating CAT versions, since it would allow people to use more scales in the same time period. This service would encourage people who are not computer-savvy to use computerized testing, facilitating their research and saving tons of paper. It would also help the researchers whose scales are posted on the site, since people would be more likely to include a scale in their research if it only required them to check an additional box. We would of course make the citations for the scales available on the website, and the website itself would probably get a lot of citations as well. The task itself would be pretty huge - we'd need to collect scales, do all the IRT work, do the CAT programming, set up the website, and write a program to take the data from the emails and convert it to an excel file. We would probably need to hire a professional programmer. We would want to start in a specific topic area, and get one or more collaborators who have developed scales in that area. We would also want to have someone experienced in CAT on our team. That either means getting one of us trained, or bringing in a CAT collaborator.

Friday, March 25

Attendees: Jamie, Bev R-E, Aimee, Joe, Chris, Jana, Michelle
We discussed some of our interactivity ideas with Bev R-E. We primarily focused on the possibility of working with TIM (The Incredible Machine).
Bev mentioned that the problems presented by TIM do have some strong similarities to those investigated in the classic problem solving literature. We should therefore try to become more familiar with this literature. She also mentioned that our work might relate to the research on "multiple intelligences." We should specifically check this literature to see what they have done to represent mechanical ability.
The standard way of using TIM would be to provide people with a puzzle and then ask people to use a set of parts to create a solution. We might alternatively think about putting people in front of the puzzle generator and ask them to come up with their own machines. We could also build this into a puzzle where we select specific parts for people to use (rather than actually putting people into the puzzle creation part of the program). We might also think about asking people to come up with a puzzle that we would give to other participants.
We could potentially look at the relation between sensation seeking and exploratory tendencies with TIM. Could also determine situational factors that might encourage people to do more exploring, and look at the relation between exploration and performance. Some possible measures of exploration would be the number of different parts that are examined, how many non-standard features of a puzzle are used, how often people look at hints, and how often people read information about a part.
We could examine how training affects peoples' ability to solve puzzles. We could train people on one collection of parts and then examine their ability to work with entirely new parts.
Bev's own work on wayfinding would likely be a good place to look at the role of interactivity. Most of her experiments are interactive in one way or another.
We should take a look at J. J. Gibson's work on ecological perception to get a better idea of how people might think about and use different components.

Tuesday, April 5

Attendees: Jamie, Michelle, Jana, Chris
Discussed the CAT grant idea some more. If we are able to actually develop the website as we propose, Chris suggested that we might obtain a second grant to provide multilingual versions of the scales. Along a similar vein, Michelle suggested that we might provide voice recordings of the surveys to make them easier to apply to young children or illiterate populations.
Jamie mentioned that he will offer a CAT seminar in the fall, and that anyone interested in working on the grant should attend that seminar.
We then spent some time talking about TIM. Jana mentioned that she sometimes became frustrated when she had to quit a puzzle halfway through and then had to start over entirely from the beginning. The effects of starting over might be something that would be interesting to look at. In real life we sometimes have the ability to put a problem aside for awhile, but there are certainly times when we have to recreate something from scratch.
Performing well at TIM definitely appears to involve both implicit and explicit learning (although talking about it as declarative and procedural learning would be better). The hints that appear in the puzzles provide declarative learning, while the practical experience of moving things around would provide procedural learning.
Before getting much further into the TIM studies, we decided that we probably need to learn more about several areas of cognitive psychology, including:

Problem solving
Multiple intelligences
Learning theory
Creativity

We decided to give each person the responsibility of investigating one of the topics above. We decided to skip creativity for now, since it's probably the part of the literature that we already know the best. Jamie will investigate learning theory, Jana will investigate problem solving, and both Michelle and Chris will investigate multiple intelligences. For next week all that we hope to do is to at least identify one or more simple review papers or chapters on the topics.

Tuesday, April 12

Attendees: Jamie, Joe, Jana, Michelle

We started by discussing the future of our projects.

The work on forked scales is on indefinite hold until someone comes up with a specifc scale and a specific population they want to examine.
The work on interactive storytelling is on hold until we are able to obtain an actual interactive story to work with. Jamie plans to continue his conversations with Chris Crawford, and it is possible that some new ideas will come from that. However, the idea of examining behavior in interactive stories itself will likely not come to pass for several years.
We are going to hold off on doing much more with the CAT project until we all have a chance to learn more about CAT. Jamie plans to offer a CAT seminar in the fall, and we will probably start working seriously on the grant in the spring. In the meantime we should just try to read the CAT primer book and decide what specific content area we want to work in.
The TIM/MacGuyver task is therefore the only project that doesn't have any immediate holds. So for now we plan to concentrate on learning more about research related to this task and will take up whichever experimental idea we come across first.

Jana found a number of articles related to problem solving, some of which seemed to have direct relevance to our research. However, most of them were about 10 years old, and many were not available in our library. Jana will try to get the articles through ILL, and Joe is going to look through some recent handbooks to see if he can find a summary of the literature.
Jamie discovered that the specific aspect of learning that interests us is usually called "cognitive skill acquisition" in the literature. Research by Rich Carlson seems to be relevant to our research. Jamie plans to chat with Rich sometime over this summer, and we will see what he has to say about our work.
Joe has decided that he will start investigating problem solving along with Jana.

Fall 2005 Meeting

Attendees: Jamie, Chris, Joe
Discussed the "History of Thinking" article.

It clearly has some strong relations to existing dual-process models in psychology, though with some differences. "Pattern-recognition" thinking corresponds to automatic processing while "Sequential" or "Verbal" thinking corresponds to controlled processing. There are some aspects of psychological models that were not included in this discussion. Existing models emphasize the interactions between automatic and controlled processing, such that automatic processing can bias controlled processing, and conscious processing can direct our attention to specific stimuli, affecting what we automatically process. They also present these two types of processing as occurring simultaneously, which was not really discussed in this article (although it isn't inconsistent with it, either).
We discussed the implication of the different ways of thinking for teaching. As Chris Crawford points out, most teaching relies heavily on sequential thinking, although there are clearly some ways that pattern-recognition thinking could be brought in. Anything organized around providing "guided experiences" would provide people with the opportunity to use automatic processing to learn from their experiences. This type of teaching, however, is bound to be much less efficient overall, although it might be necessarily to learn certain types of skills. To try to make this type of learning more efficient we might consider how we might create "summarized" versions of experiences, basically guiding people though a nonrandom selection of experiences that would provide them with the most important information. The key would be to identify what are the most important things that people might learn through this type of learning, and what aspects of the experience are critical. We discussed how this line of thought might have some implications for the way upper-level undergraduate and graduate seminars are structured. It might be worthwhile to provide a larger perspective of how different articles link to each other instead of organizing discussions around specific topics. There are some textbooks that try to do this explicitly (Smith & Mackie social psych text) but these connections are often ignored in favor of providing greater content. It might be useful for instructors to indicate some of these links directly on their course syllabi.
Information that is embedded in a network is more generalizeable, since it links an event to its causes and its effects. This view of the process makes it easier to draw parallels to other circumstances.
We discussed the possiblity that different types of processing might differ in terms of the pleasure that they provide. Some people might receive more pleasure out of performing a pattern-recognition task, while others may get more pleasure out of sequential learning. This might be what the well-known "Need for Cognition" scale is getting at, even more than a tendency to use these two different types of processes. This might have some implications for enjoyment of and performance on TIM. We discussed a possible new experiment. In the Ashby categorization task, people are asked to learn about a categorization that is either verbalizeable or not verbalizeable. We might look to see if people enjoy one type of task more than the other. The enjoyment may specificly depend on other variables, such as Need for Cognition or gender. We might also see whether the pleasure rating is related to various performance measures. The basic design of the study would be to have people perform the NFC task, then the categorization task, then the pleasure rating task. We might consider giving the NFC task in a prior session to prevent it from affecting the pleasure ratings, although the categorization task itself might be distracting enough to prevent any biases.

Links