Speaker 1: Welcome back everybody.
Speaker 2: Today we're going to be looking at validity concepts and
psychological research. We're taking a deeper dive into research and
we'll be looking at validity. Some of them must know definitions.
Internal validity asked whether the independent variable truly caused the
change and the dependent variable. So, if you remember research
we are looking for here, is there really the independent variable?
Do we have other confounding variables playing a role? External
validity asks whether findings can be generalized beyond the study
to other people, other settings and times, So between lab
work compared to the real world. Construct validity asked whether
we actually measured what we intended to measure, and we'll look.
Speaker 1: At these deeper in a minute.
Speaker 2: Statistical conclusion validity ask whether the relationships between variables.
Speaker 1: Are accurate and reliable.
Speaker 2: Threats to ability undermine the trustworthiness of conclusions and must
be addressed in study design, interpretation, and application. Strong research
design uses tools like randomization, manipulation checks, multi method approaches,
and sampling strategies. So back to internal validity, the backbone
of causal inference. The core question can be can we
be confident the independent of aerial cause a change in
the dependent variable. Common threats to internal validity include to history.
For instance, external events occurring during the study that influence
results independently of the intervention. It's an example of history.
Here are another example of it. A therapist is studying
the effect of a UCBT.
Speaker 1: Protocol on depression.
Speaker 2: Halfway through, a major natural disaster occurs in the region,
and participants mood scores worse than regardless of treatment. The disaster,
not the treatment, is the driving change. Another factor could
be maturation, the natural development of or biological changes and
participants that occur over time independent of the treatment. Clinical
examples A school psychologists tests of reading intervention was second greater.
Over eight months, kids improve and much of the game
reflects normal reading development that would have occurred anyway, not
the program itself. Another one it's testing effects. Exposure to
a pretest influences performance on the post test independent of
the intervention. So a researcher administers the BDI before and
after an eight week mindfulness program. Participants score lower on
the post test, partly because they have already seen the
questions and know how to respond, not solely because of
mindfulness practice instrumentation changes in measurement tools raiders for scoring
criteria over the course of a study. Clinical example, in
a longitudinal outcome studies, supervisors rating patient progress become more
lenient in their scoring after several months of familiarity with participants,
producing artificially inflated improvement scores. Another one in statistical regression
definition and participants are selected because.
Speaker 1: Of extreme scores.
Speaker 2: Scores naturally drift toward the mean or retesting regardless of intervention.
The clinician recruits clients scoring in the severe range on
an anxiety measure for a new exposure therapy trial.
Speaker 1: At protests post.
Speaker 2: Tests, many show improvement simply because extreme core scores statistically
regress toward average, not necessarily because of treatment efficacy.
Speaker 1: You can see this in athletes too.
Speaker 2: Just to make it if you're inclined to watching sports.
If you look at a basketball player and they average
thirty five thirty points a game, but one game they
hit fifty, more than likely there's going to be games
in the future that they're going to score under thirty and.
Speaker 1: The average will bounce.
Speaker 2: The average will drop right into that thirty range over
a course of a series of games.
Speaker 1: Selection bias is another one.
Speaker 2: Non random group assignment produces pre existing differences between groups
that confound results. An example would be a hospital assigns
patients who voluntarily request therapy to the treatment group and
those on a wait list to the control group. The
treatment grouping are ready be more motivated, making any outcome
difference attribute to motivation.
Speaker 1: Rather than the therapy itself.
Speaker 2: Attrition definition is systematic drop out of participants that skews
the remaining sample and distorted results. An example would be
a twelve month substance abuse treatment study. The most severe
users drop out early. The remaining completers show high success rates,
but the findings reflect survivor bias but not true treatment effectiveness.
Another one is a diffusion of treatment, where control group
participants become exposed in the intervention, contaminating the comparison. So
in a DBT's skill study, participants in the control group
begin learning distress tolerance techniques from friends in the treatment group,
reducing the observable difference between the conditions. Strategies of strengthen
internal validity or random assignment. It distributes participant characteristics evenly
across groups, eliminating selection bias at the outset. Control groups
provide a comparison baseline so that orth changes can be
attributed to the independent variable rather than extraneous variables. Consistent
measurement protocols reduce instrumentation drift by standardizing how when data
is collected across old time points. Blinding raiders or participants
prevents expectancy effects from influencing data collection or participant behavior.
Pretesting to assess baseline equivalence confirms that groups start from
the same point before the intervention begins. External validity can
we generalize these fightings through.
Speaker 1: The real world or other people or other settings.
Speaker 2: Some of the major threats to external validity include population validity,
the degree to which findings generalize from the study sample
to the broader intended population. The landmark study on social
anxiety treatment was conducted entirely with white college aged women
recruited from US a university psychology pool. Applying those findings
to middle aged Latino men settings stretches a sample beyond
on its original boundaries.
Speaker 1: Another one is ecological validity.
Speaker 2: The degree to which the study environment mirrors the real
world conditions. For instance, a memory intervention tested in a quiet,
well lit lab with no distractions may show strong effects,
but when applied and an assisted living facility with ambient
noise and caregiver interruptions.
Speaker 1: The benefits diminish substantially.
Speaker 2: Temporal validity the degree to which findings hold across different
historical time periods and cultural moments, For instance, and.
Speaker 1: Stress and oculation.
Speaker 2: Protocol validated during a period of economic stability may not
generalize the populations experiencing a pandemic, mass employment, or a
political crisis.
Speaker 1: Strengthening the external validity.
Speaker 2: Well representative sampling ensures the study sample reflects the demographics
and characteristics of the target population. Also, realistic settings or
field studies move research out of labs and into the
environments where findings will actually be applied. Replication across samples
and settings builds confidence that an effect is robust rather
than an artifact of one particular context. Constructibilities Are we
measuring the theoretical construct who claim to be measuring?
Speaker 1: It?
Speaker 2: Is about the fit between concept and method. If you
claim to measure anxiety, are you capturing anxiety?
Speaker 1: Or are you capturing general.
Speaker 2: Distress, social discomfort, or physiological arousal. Here are some threats
to construct validity monomethod bias. Relying on only one measurement
method such a self report alone limits the scope of
what is captured and introduces shared method variance that inflates correlations.
A researcher assesses both depression severity and quality of life
using only self report questionnaires administered in the same session.
Any correlation between scores may partially reflect the participant's general
negative mood in the state, not a true relationship between
the constructs. Inadequate operationalization, poorly defined, or mismat which measures
that do not adequately represent the theoretical construct being studied.
The researcher claims to measure empathy, but uses only a
cognitive perspective taking scale. Affect of empathy that felt resonance
with another's emotions goes entirely unmeasured, leaving the operationalization incomplete.
Experimental expectancy, Subtle behavioral cues from researchers unconsciously influenced participant responses,
contaminating the measurement. A therapist researcher who believes a new
intervention will work may ask questions in a warmer, more
affirming tone during treatment sessions, inadvertently shaping participant responses in
the direction of the hypothesis. Hypothesis Guessing participants offer and
sorry explain that participants infer the study's purpose and alter
their behavior to conform to perceived expectations, producing demand characteristics. Example,
participants in a mindfulness study figure out that researchers expected
to report reduce stress. They minimize stress complaints on questionnaires,
even when their actual experience has not changed meaningfully. Confounding constructs,
overlapping variables are not separated from one another, making an
impossible determine which construct.
Speaker 1: Is actually driving the effect.
Speaker 2: An example would be a study claims to demonstrate the
effects of self esteem on academic performance, but the measure
of self esteem is heavily saturated with general positive affect
it becomes unclear whether self esteem or.
Speaker 1: Mood is the active ingredient.
Speaker 2: So how do we improve construct validity? Multi method assessments
combined self report, physiological measures, and behavioral observation to triangulate
a construct from multiple angles.
Speaker 1: Right, you want to understand this from multiple sides.
Speaker 2: Manipulation of checks confirm that the independent variable actually produce
the intended psychological state before examining.
Speaker 1: Its downstream effects.
Speaker 2: Pilot testing measures for clarity and precision catches ambiguous or
double barreled items before the full study launches. Established a
validated scales carry a history or psychometric evidence supporting their
ability to measure the intended construct. Another idea is statistical
conclusion validity. The question are the statistical conclusions accurate to
the data analysis, preserve a real relationship, or miss one
that existed. Some of the threats are low statistical power.
An underpowered study lacks the sensitivity to detect the real effect,
increasing the risk of type.
Speaker 1: Two error false negatives.
Speaker 2: A pilot study comparing two therapy modalities uses only twelve
participants per group. A genuine medium sized treatment difference exists,
but fails to reach significance because the sample size is
too small to detected reliably. Speaking of that, unreliable measures
is another one. Measures with poor reliability introduce random noise
that obscures true relationships between variables. For example, a researcher
uses a hastily assembled depression scale of poor internal consistency.
The noise is in the measure swamps the signal, producing
a non significant result even though the treatment genuinely reduced
depressive symptoms. How about violation of assumptions applying parametric statistical
test to data that does not meet the required distributional
or scaling assumptions.
Speaker 1: Let me give you an example of this one.
Speaker 2: A therapists researcher runs a standard A NOVA, so you're
looking at multiple independent variables on a heavily skewed trauma
symptom scale without transformation, producing misleading f statistics and potentially
false conclusions about group differences. So let me try to
explain this a little bit more simple because it kind
of gets into the weeds when it comes to parametric
statistical tests. So imagine you're entering a high performance sports
car into a race. Before you can race, the officials
give you a strict rule book. The car must use
premium racing fuel, they must race on a smooth, paved track,
and the tires must be properly inflamed. If you ignore
the rules, say you fill the tank with cheap gas
and try to drive the car through a muddy swamp,
the car is going to stall, sputter, and give you
terrible performance and statistics. The violation of assumptions is exactly
like trying to drive a sports car through the mud.
You're using a highly powerful, sensitive mathematical.
Speaker 1: Tool on data that doesn't fit the rules.
Speaker 2: So parametric statistical tests, these are the high performance sports
cars of math, like a lenova or a T test.
They're incredibly accurate, but the only work if your data
follows strict rules distributional assumptions. The biggest rules that your
data must form a perfect symmetrical Barel curve called the
normal distribution. Most people should score in the middle, with
a few scoring very high and few little low.
Speaker 1: Scaling assumptions, the.
Speaker 2: Data must be measured on a consistent even scale like
height or temperature, not just random categories. If violation happens
when a researcher gets impatient or careless ignores these rules
and forces messy lopsided data into one of these strict
math formulas.
Speaker 1: Anyway, so here plays out.
Speaker 2: So a therapist runs a standard in nova we talked
about in a heavily skewed trauma symptom scale without transformation,
producing misleading f statistics and potentially false conclusions. The messy
data the mud. The researcher is measuring trauma symptoms and patience.
Because trauma severe, almost everyone in the study scores incredibly
high on the symptoms scale, while almost no one scores low.
Speaker 1: This means the data is heavily skewed lopsided.
Speaker 2: It looks like a giant cliff, not a smooth, symmetrical
Bell curve. The violation the researcher wants to compare three
different therapy groups to which one is the best. Instead
of fixing the lopside of data, they immediately run a
standard of nova, a test that mathematically demands a perfect
bell curve that we talked about. So this is the violation.
Because the data broke the rules. The math formula breaks down,
the test spits out a warped, incorrect number and misleading
f statistic. The test might say, Wow, therapy A is
a miracle cure. In reality, therapy didn't work at all.
The broken math created an illusion. It's created a false
positive type one error. So you have to transform the
data with algebra and other things.
Speaker 1: But we'll leave it out of that. Out of it
for now.
Speaker 2: So let's go back another when it's fishing and multiple comparisons.
Running numerous statistical tests without correction inflates the teeth, inflates
the type one error rate, increasing the probability of false positives.
The researcher tests twenty different outcome variables in a single
study without applying it. What they call it bond Feroni
or false discovery or rate correction. By chance alone one
or two will reach the P value less than point
zero five. But these are likely fake findings, what they
call spurious findings. So you can keep running statistical tests
and eventually your false positive will appear. Restriction of range,
limited variability and variable reduces the ability to detect true
correlations or effects. So let me see example of study
examining the relation between therapist experience and client outcomes. Recruits
only licensed clinicians with ten to twenty five years of experience.
The narrow range of experience prevents detection of a gradient
that might be visible of trainees and seasoned veterans were
both included. Error statistical analysis well, misuse of tests, incorrect
model specification, or misinterpretation of P values will affect the conclusions,
and researcher treats a P value zero point four to
nine as definitive proof of a real effect without reporting
effects sizes or confidence intervals, ignoring that defining may be
statistically significant but clinically trivial.
Speaker 1: That's where the.
Speaker 2: Effect sizes and confidence in themals come. In power analysis,
this is for strengthening statistical conclusion validity. Power analysis determines
the minimum sample size needed to reliably detect and effect
on the expected magnitude. Correct statistical models match the analytic
approach to the data structure.
Speaker 1: Research question.
Speaker 2: Measurement level reliability of measures insures that it iss are internally
consistent and stable across time. Adjusting for comparisons using Bonferoni
correction or similar procedures. Controls the family wise air rate,
reporting effects sizes, and confidence intervals alongside p values. Conveys
the magnitude and precision of findings. Using validity concepts to
evaluate research. When reading or designing research, use these four
validity questions as a filter. Does this study rule out
alternative explanations? That's internal validity? Would these findings apply to
my client or setting external validity? Do the measures match
the construct construct validity? And can I trust the data analysis?
Statistical conclusion validity question one to practice for you, The
researcher conducts a six month study evaluating a new cognitive
behavioral intervention for generalized anxiety disorder. Participants are recruited from
a single university counseling center and assessed using only a
self reporting aliety questionnaire. Halfway through the studies, several participants
begin comparing notes with each other in the waiting room.
Speaker 1: And start adjusting their questionnaire responses.
Speaker 2: Which two validity threats are most directly illustrated in this
scenario history and statistical regression, attrition and ecological validity or
hypothesis guessing and monomethod bias. The answer is b B.
Hypothesis guessing and monomethod bias. Participants Altering responses based on
perceived expectations is the definition of hypothesis guessing.
Speaker 1: It constructs validity threat.
Speaker 2: The exclusive use of a single self report measure represents
monomethod bias. Population validiti is also arguably compromised by the
university sample that wasn't given in your choices.
Speaker 1: Another one.
Speaker 2: A clinical researcher publishes findings showing a new mindfulness based
interventions significantly reduced burnout scores sample of emergency room nurses.
The study used random assignment, a control group, standardized measures,
and appropriate statistical modeling. However, all data were collected in
a single large urban hospital in the Pacific Northwest.
Speaker 1: During a period of unusually high staffing levels.
Speaker 2: A colleague wants to implement the intervention which validity concern
is most relevant to the colleague's hesitation because they want
to implement in a rural community health clinic in the
Deep South. So as an A internal validity due to
lack of random assignment, B statistical conclusion validity due to
small sample size, or C external validity due to population
and ecological differences. If you said C, you're right. The
original study had strong internal validity given random assignment and
control conditions. The concern here's where they're finding some urban
Pacific Northwest nurses with high staffing generalized to rural Southern clinicians.
This directly invokes population validity and ecological validity.
Speaker 1: That's it for now. It's a little longer than
Speaker 2: Normal, but hopefully you enjoyed the podcast and have a
good week.