Speaker 2: Welcome back everyone.
Speaker 1: Today we're gonna get back into the weeds of still
into research. We're covering psychometrics, reliability, and validity.
Speaker 2: So every test you've ever given a.
Speaker 1: Client, every questionnaire, every checklist, is making a silent promise
and it is promising that the number it spets out
actually means something. Today we're pulling that promise apart. So
we're going to be talking about psychometrics, the mathema logic
behind whether your tools can be trusted. So let's look
at some of the things we're going to look at.
Scale of measurement. First, nominal scales or categorical with no order.
Think diagnosis categories, gender, eye color. There's no more then
or less then blue eyes. Here are just different buckets.
Because there's no order, you're allowed analysis are limited to
frequency counts and what they call Ki square tests, which
we've talked about before. The ordinal scales rank categories, but
the distance between ranks is not equal. So the classic
psychic example is a LIKEERD scale that strongly agree or
strongly disagree you see on almost every self report measure.
A client moving from moderate to severe depression on a
rating scale tells you direction, but not really the magnitude
allowed analysis. Here are medians, percentiles, and non parametric tests
like the Man Whitney you, which will be important to understand.
Interval scales of equal intervals, but no true zero, So IQ.
Speaker 2: Scores are the TechBook textbook.
Speaker 1: Example zero and IQ score does not mean the absence
of intelligence. It just means a raw score of zero.
You can compute means and standard deviations and run T
tests and Anova's, but you cannot meaningfully say someone with
a one forty RQ is twice as smart as someone
with a seventy because there's no true zero. Now, if
you have a ratio, which is have equal intervals and
a true zero, reaction time and number of errors on
a task are ratio data. So here all mathematical operations
since they're fair game, including ratios. So when you say
a test score doubled that's true, Ask yourself, is this
actually a racial scale? Am I assuming one where it
does not exist? I accuse the perfect trap here because
it feels like it should be ratio, but it's not
because there's no true zero. Segment two reliability consistency of
measurement reliabilities about consistency, it does not ask whether a
test measures the right thing only whether it measures the
same thing in the same way every time. Test retest
reliability measures temporal stability.
Speaker 2: You administer the same test.
Speaker 1: Twice and look for a high correlation between the two administrations.
Let me give you a clinical example, giving a client
the Beck depression inventory at intake and again two weeks
later to see if scores hold steady absent in any intervention.
The threats here are practice effects and maturation, meaning the
person might get better at taking the test or might
actually change between sessions for reasons.
Speaker 2: Unrelated to the construct.
Speaker 1: In tonal consistency most often measured with something called the
Kronback's alpha. This asks whether items within the test measure
the same construct. An alpha above zero point seven is
generally considered acceptable. Here's a trap you gotta be careful with.
An alpha above ninety is not automatically better. It can't
actually indicate item redundancy, meaning your items are just restating
each other rather than each contributing something useful. So split
half reliability is a related method where you divide the
test and two and correlate the halves. Another one is
intirator reliability. It's as agreement between observers or scores calculating
calculator using something called the Kappa Kappa or inner intra
class correlation the ICC. This matter is enormously in behavioral coding,
clinical diagnosis, and qualitative scoring. Picture two trained clinicians independently
coding videotape sessions for therapists empathy. Their agreement or lack
is inturrator reliability.
Speaker 2: And action parallel forms.
Speaker 1: Reliability uses two versions of a test measuring the same construct,
administered to the same people, where scores should correlate highly.
This controls for form specific area, which is why standardized
tests like the SAT have multiple versions across administrations. Another
part is validity, the accuracy of measurement. Here's a line
that should be tattooed in every person's forearm. You can
have reliability without validity, but you cannot have validity without reliability.
Speaker 2: A broken scale and it always.
Speaker 1: Reads five pounds too heavy is perfectly reliable and completely invalid.
Content Validity asks whether the test items addequately cover the construct.
Speaker 2: It is evaluated by.
Speaker 1: Expert judgment, not statistics, and it is key in achievements
and personality scores. An example would be a generalized anxiety scale.
Should cover worry, physical tensions, sleep disturbance, and concentration difficulty,
not just one slice.
Speaker 2: Of the construct.
Speaker 1: Criterion validity asks how well the test correlates with an
external standard.
Speaker 2: There are two favors.
Speaker 1: Flavors Concurrent validity is measured at the same time, like
comparing a new depression screener against the same day clinical diagnosis.
Predictive validity is about the test predicting a future outcome,
the classic example being SAT scores predicting college gpa.
Speaker 2: Both are measured using correlation coefficients.
Speaker 1: Are Construct validity asks whether the test is measuring the
theoretical construct that claims to measure. That splits into convergent
validity where the test correlates with measures of similar constructs,
and discriminate validity whether it where it does not correlate
with unrelated constructs. An example would be a new social
anxiety scale should correlate with an established social anxiety measure
that's convergent, but should not correlate strongly with a measure
of an unrelated trait like extraversion.
Speaker 2: That would be a discriminate validity.
Speaker 1: This is demonstrated through factor analysis and multi trade method matrixes.
Speaker 2: Face validity simply.
Speaker 1: Asked whether the test looks like it measures with it
should It is not scientific. It can, but it can't
influence client engagement and social acceptability. And item like I
feel sad most days has high face validity for depression,
even if it's statistical properties still need to be tested.
Speaker 2: Another aspect is measurement issues.
Speaker 1: This is where good instruments can go wrong in practice.
Response sets are systematic ways people respond, regardless of item content.
Acquiescence is always agreeing. Extreme responding means only using the
end points of a scale. Central tendency bias means avoiding
extremes altogether and clustering in the middle. You mitigate these
with reverse coded items and balance scales. Social desirability this
is the tendency to answer in a way that looks
good and can inflate or obscure a client's true traits.
I think of a client under reporting substance use or
minimizing anger because they want to be see seen favorably.
This is exactly why instruments like the MMMPI two build
in validity scales, and why clinicians sometimes use indirect items
to detect it. Ceiling and floor effects happen when the
test is too easy or too hard for the population.
A ceiling effect means scores bunch at the top, like
giving a standard i Q test to a gifted population
where everyone tops out. A floor means scores bunch at
the bottom, like using an adult cognitive measure where the
population has significant intellectual disability. Both reduced variability, which weakens
correlations and discrimination. Cultural bias occurs when items reflect the
dominant culture's norm or language, potentially disadvantaging non native speakers
or neuroindividuent in nerve neurodivergent individuals. Measurement models, we look
at classical tests. Theory CTTT arrests on one core equation.
Every observes score equals the true score plus error. Reliability
under this model reflects the proportion of true score variance,
and its limitation is that it treats eras random and
does not account for individual item difficulty. Item response theory
goes deeper. It examines the relationship between a person's latent
trait level and their probability of responding correctly to each
specific item. Each item carries its own parameters difficulty, how
hard it is, discrimination, how well it separates people of
different ability levels, and guessing the chance of getting it
right without actually having knowledge about it. IRT allows for
adaptive testing and better scale precision, which is why you
see it in large scale assessments of the GRE and
in certain adaptations of the MMPI. The six steps of
scale development. If you ever had to build a clinical
measure from scratch, here's the order of operation. To find
the construct is Number one, what exactly are you measuring
based on theory, literature, clinical input? Number two generate items.
These are questions using experts, clients, and focus groups that
create a broad item pool. Number three pilot testing deministering
items to a sample and analyzing item total, correlations, difficulty,
and discrimination. Number four factor analysis, which splits into exploratory
factor analysis, uncovering underlying dimensions you did not specify in advance.
In confirmatory factor analysis, testing a hypothesized structure you already
believe is true. The goal is to identify and retain
strongly loading items. Number five establish reliability and validity, checking
internal consistency, test retests, constructing criteria validity, and number six norming,
which is administering the final instrument to a large representative sample.
Speaker 2: To create standard scores and cutoffs.
Speaker 1: When selecting a sec test, match into the referral question
and the population in front of you, check its norms
reliability and validity, and considered language, cultural context ability. When
interpreting a test to always factor in measurement, error and
confidence intervals. Avoid over reliance on a single score by
looking for patterns across the data and straight transparent about
the limits of your interpretation. That's it for now, folks. Oh, actually,
I forgot some practice questions.
Speaker 2: A researcher.
Speaker 1: Develops a new generalized anxiety scale to validated. She administers
the new scale in a structured clinical interview for anxiety
disorders to the same group of participants in the same day,
then compares the results. Which type of validity is it
as a predictive A, content B or concurrent C. If
you said concurrent.
Speaker 2: Validity, you're right.
Speaker 1: This is the validity measured at the same point in time.
Comparing the new instrument against an external standard of the
clinical interview. Question number two, the clinical researcher trains two
raids to an independently code video tape therapy sessions for
a level of therapist empathy. She wants to know how
consistently the two vadors agree with each other which reliability
MU should she calculate, Test retest reliability, parallel forms reliability
or interrator reliability. If you said C, you're right interrator reliability.
Typically calculator using Kappa or interraclass correlation ICC to determine this.
Speaker 2: That's it for now, folks,