Methodology Grant
Readings
Notes from Meetings
- May 19, 2006
- Meeting attendees: Jamie D, Jamie O, Bev RE, Anne-Marie
- Bev RE provided information about calls for proposals from both
NIH and NSF dealing with methodological issues. Links to these program
announcements are provided below.
- Discussed three possible projects
- Data archiving
- Cognitive abilities and reverse coding
- Measuring well-being in reaction to use of technology
- Our group decided to focus on the first two topics. Bev RE is
involved in another group that will be investigating the use of
technology.
- Purpose of the data archiving project would be to develop
methods (and potentially software) to organize, store, and document the
results from large research projects in a way that makes it easy for
multiple researchers to access the same data, to make it easy to
provide the data to those not originally involved in the project, and
to go back later on and rexamine the results or reanalyze the data.
- Purpose of the reverse coding project would be to determine
what cognitive abilities are important for people to accurately respond
to reverse coded items. We will also use this knowledge to develop
specific ways of reverse coding items so that they can be understood
easily.
- We plan to submit the data archiving and reverse coding grants
separately. We will try to get the reverse coding grant written up for
Fall 2006 as an R21 or R03 equivalent. We are uncertain with regard to
the mechanism for the data archiving grant. We might try to submit it
as a standard R21/R03, or else we might try submitting it as an
SBIR/STTR
(business grant). Jamie's friend Angie's brother-in-law works for a
computer firm in Atlanta and might be a potential resource when trying
to develop an SBIR.
- Tasks for data archiving
- Literature search on data archiving (Jamie O)
- Talk with Alabama's office for research (Bev RE)
- Discussions with investigators of large resarch projects
- Ed Mulvey (AM - delayed)
- John Bolland - MYS (Jamie O)
- Debra MacCallum - Capstone poll (Jamie O)
- Lou Burgio - Reach, etc. (Jamie D)
- Talk to Corina Davidson about SBIR grants (Bev RE)
- Talk with Angie & her brother-in-law about SBIR
opportunity (Jamie D)
- Tasks for reverse coding
- Literature search on reverse coding (Jamie O)
- Literature search on the processing of negations (Bev RE)
- Get access to data that has measures of cognitive functioning
and scales with reverse coded items
- SUDS/MYS (Jamie O)
- Laura Phillips (Jamie D)
- Randy Salekin (AM)
- John Lochman (AM)
- Ed Mulvey' (AM - delayed)
- Contact Norbert Schwarz about his work on surveys (Jamie D -
delayed)
- May 26, 2006
- Meeting attendees: Jamie D, Jamie O, Bev RE
- Bev RE talked to Alabama's office of research and found that
Marianne Woods is the person most involved with documenting research.
We decided not to contact her immediately, but we will arrange a
meeting with her once we have a more specific plan for the data
archiving grant.
- Bev RE mentioned that there are specific theories about how
people process negative statements that we can use for our reverse
coding project.
- Bev RE and Jamie O presented some information about individuals
who respond poorly to RC items. Reading ability, age, IQ, etc. However,
there does not appear to be research relating poor performance on
reverse coded items to specific cognitive abilities like working
memory, context processing, etc.
- Jamie O mentioned that there is basically nothing describing
the best ways of archiving data. There are some articles that do
provide general recommendations, but no one tries to present a
systematic method. Jamie provided us with the keywords she used in her
literature search:
- data archive/archiving problems
- data archive/archiving process
- data archive/archiving
- data manage/management
- The grant examining reverse coding will have two foci
- Identify specific cognitive abilities related to the
comprehension of reverse coded items.
- Test different methods of reverse coding items to find those
that prevent thoughtless responding and yet provide valid responses.
- Bev RE suggested the following organizaiton for the reverse
coding grant
- Describe effects of reverse coding on survey quality
- Discuss presence of this problem in special populations
- Establish the relation between general cognitive abilities
and comprehension of reverse coding
- Investigate the relation between specific cognitive abilities
and the comprehension of reverse coding
- Test different methods of reverse coding items to see which
is easisest to understand
- We've decided to try to make the reverse coding project an
extension of the SUDS grant. This will involve bringing John Bolland
and Joan Barth in on the project. There are several advantages of this.
- Provides a specific domain where this is a problem (at risk
youth), highlighting the importance of the issue.
- We can work with John's infastructure to obtain our sample.
- We all have experience working with the population.
- John and Joan have a great deal of experience working with
surveys.
- Jamie O reviewed her discussion with Debra MacCallum about the
capstone poll. The data set is not terribly organized, and they do not
have a specific, explicit method of archiving the data. Jamie O and
Jamie D will arrange a time to meet with the data manager (Mike
Connaway) of the capstone poll to get more information.
- We might potentially use the data from the Capstone Poll to
look at reverse coding. Every poll collects basic demographic
information including educational level, and a number use scales that
have reverse coded items.
- Jamie D brought up the issue that we might have some difficulty
with the analysis of our reverse coded studies. If we want to compare
mutiple methods of reverse coding items we will need to compare the
correlations obtained with one method to the correlations obtained with
other methods. Comparing correlations is a very low-powered test.
- There may be linguisitic issues affecting comprehension of
reverse coded items. Bev RE will talk with Catherine Davies in the
English department to see if there is anything particular about Black
English (spoken by many people in the MYS sample) that might have
implications for comprehension of negatives
- June 2, 2006
- Meeting attendees: Jamie O, Bev RE
- June 7, 2006
- Meeting attendees: Jamie D, Jamie O, Bev RE, Anne-Marie, Joan
- Joan suggested that we plan to run the study in Tuscaloosa
since there are too many projects already running down in Mobile. We
should be able to get our desired sample size if we recruit and test in
the homes. We can talk over the details with Debra MacCallum.
- Plan to originally start as an R03, but might move to an R21 if
we need more money to complete the study.
- John may be able to access the standardized test scores for
participants in the MYS.
- We talked about looking at the influence of scale response
formats more generally, although this would involve different theories.
We also discussed making the focus of the grant more empirical and less
theoretical.
- Jamie's friend Jennifer Dail may know theory from the education
literature that would be relevant.
- Might consider examining the National Youth Survey for
reverse-coded items.
- Joan will talk to John to see his level of interest in this
project.
- Considering using 3 reverse-coded scales, 5-10 cognitive
measures, measures of context processing, and the "dot response task"
by Braver & MacDonald.
- There could be some difficulty using computers in the homes.
May consider using MacMillan as the center for adminitering the study.
- June 29, 2006
- Meeting attendees: Jamie D, Jamie O, Anne-Marie, Joan
- Started with a discussion of Joan's conversation with John.
Mobile is unavailable during the summer, but may be available during
the school year. However, we would probably be able to get better study
administrators (as GRAs) here in Tuscaloosa. We could consider running
some sort of summer internship program to attract administrators. In
this circumstance people would get course credit from Alabama in
addition to pay. John has had very good success with this, and has
attracted people from across the country.
- Joan mentioned that we can get the local addresses of kids
within specific age groups to help target recruiting. Person we'd want
to talk to is Rick Herbert - this is who Joan talked to when recruiting
for SUDS. AM talked to the same person, but he suggested that she work
through the YMCA instead of directly recruiting participants. AM fround
that working under the YMCA rules was very restrictive, and suggests
that we not try to work with them. We should therefore tlak to Rick
Herbert about going door-to-door to recruit participants directly.
- Discussed the administration of the surveys. AM mentioned that
she would not feel comfortable testing alone in homes. John writes in
his grants that administrators test people in pairs, although they will
often separate. Some people also administer surveys on their own. Jamie
O mentioned that if we are going to send people out in pairs they
should administer the survey together in the same home instead of going
to nearby homes. The incidents she has had have always been in the home
instead of simply being out on the street.
- Recruitment will be more difficult if we either decide to
collect a truly random sample or else wanted to collect longitudinal
data. However, neither of these should be necessary for our grant.
- Considered collecting data in large groups instead of
individually in the homes. We might also consider using both methods,
specifically using large groups when we recruit participants from
specific organizations.
- There are differences in the understandability of items between
when people listen to them and when they read them for themselves. We
need to consider this when designing our study.
- Joan suggests using the Peabody instead of the Kbit as an IQ
measure.
- Research from the concept attainment literature indicates that
it is easier to confirm hypotheses than it is to disconfirm them.
- July 26, 2006
- Bev searched the literature and didn't come up with a clear
answer regarding the best way to phrase reverse-coded items.
- One common error (supported by the literature) is that people
will read the sentence as if the "not" isn't there.
- Working memory capacity is clearly necessary to interpret the
reverse-coded items correctly. We might try to rephrase sentences so
that the "not" is at the end or the beginning so that it is easier to
identify and takes less memory. May also consider using naturally
negative words instead of using the "not".
- Could try changing the response scales while leaving the items
the same.
- Our discussions seem to be focusing more on the practical
aspects then on the theory, but it is important to emphasize that we
are basing our ideas on cognitive psychology research.
- Should develop the multidisciplinary focus of the grant: Jen
from education, Barbara Kucharski from communicative disorders.
- Anticipated age ranges for study = 10-16. 10 is earliest age
for functional literacy, 16 is oldest age for required schooling.
- Anticipated authorship on grant: Bev = PI, Jamie = Co-PI, Joan
= Co-I.
- First task is to develop methods on how to reverse-code items.
Joan and AM will examine existing scales to ook at methods that have
already been use.
- Plan to organize the grant around the following questions:
- How do you know that a person doesn't get a reverse-coded
item?
- When do people not get reverse-coded items?
- What do you do with the data where people misunderstand the
item?
- Why do people not understand reverse-coded items?
- What response formats are easy and what response formats are
difficult?
- How do you write reverse-coded items so that they are easily
understood?
- Who has difficulty with reverse coded items?
- How do you intervene to help people understand reverse-coded
items?
- Some potential methods to help people understand the items are
to have regular reminders of the instructions, and do make intelligent
use of format and font changes.
Links
- NIH program announcements
- NSF program announcement