Speaker 1: Welcome to the deep dive, your shortcut to truly understanding
Speaker 1: complex topics. Today we're taking a fresh look at something
Speaker 1: often dismissed as well tedious, maybe just a box checking activity.
Speaker 1: Software testing.
Speaker 2: Yeah, it definitely gets that reputation sometimes.
Speaker 1: But what if I told you it's actually a deeply creative,
Speaker 1: intricate craft, almost like what a skilled artisan brings to
Speaker 1: their masterpiece.
Speaker 2: It's absolutely true. Software testing is far more than just
Speaker 2: you know, finding bugs. It's really about understanding the very
Speaker 2: fabric of how software is built, how it behaves, and
Speaker 2: ultimately how it's validated for quality.
Speaker 1: That's exactly right. Today we're going to pull back the
Speaker 1: layers on the craft of software testing, drawing insights from
Speaker 1: Pusey Jorgensen's Software Testing a crafts Fund's approach.
Speaker 2: It's a foundational text in the field.
Speaker 1: Our goal here is to unpack the fundamental concepts, the
Speaker 1: diverse techniques, and maybe some surprising real world applications, so
Speaker 1: you walk away with a much richer understanding and maybe
Speaker 1: even a new appreciation for this qui discipline that really
Speaker 1: underpins almost everything digital around us.
Speaker 2: Sounds like a great plan. Where should we start?
Speaker 1: So let's start right at the beginning, what testing really is.
Speaker 1: Jorgensen gives us this clear progression of terms that are fundamental.
Speaker 1: It all begins with a human error, a.
Speaker 2: Mistake, right, someone makes a mistake while coding.
Speaker 1: That's the error, and that human error then shows up
Speaker 1: in the software as.
Speaker 2: A fault, which is the bug or defect people talk about. Yeah,
Speaker 2: it's a representation of that mistake in the code.
Speaker 1: And it's crucial to remember this human element, isn't it.
Speaker 1: We test because we know that we're fallible.
Speaker 2: Especially in something as complex as software.
Speaker 1: We make mistakes, and it continues from there. When that
Speaker 1: code containing the fault actually gets executed.
Speaker 2: That's when you get a failure. The software doesn't do
Speaker 2: what it's supposed to do.
Speaker 1: Okay, And finally, the incident is what the user actually
Speaker 1: sees exactly.
Speaker 2: It's the symptom, the alert, the thing that makes someone say, hey,
Speaker 2: something went wrong here. So it's this chain reaction.
Speaker 1: Error leads to fault, leads to failure, leads to incident.
Speaker 2: You got it, human mistake, bugging code, wrong behavior, user notices.
Speaker 1: So given that chain, a test becomes this deliberate act
Speaker 1: right exercising the software with specific.
Speaker 2: Test cases, right and it has two main goals. Either
Speaker 2: you're trying to find those failures hut them down, or
Speaker 2: you're trying to demonstrate confidently that the software is working
Speaker 2: correctly under certain conditions.
Speaker 1: And a test case. It's not just throwing random stuff
Speaker 1: at the program.
Speaker 2: No, no, not at all. A craftsman builds a test
Speaker 2: case carefully. It needs an identity, a clear purpose like
Speaker 2: testing a specific business rule. Okay, it needs to find preconditions,
Speaker 2: specific inputs, and crucially the exact expected outputs.
Speaker 1: You need to know what right looks.
Speaker 2: Like absolutely, and even the expected state of the system
Speaker 2: after the test runs. Plus you keep track of its
Speaker 2: execution history. It's a complete, thoughtful construction.
Speaker 1: So that's the what and why. Now how does a
Speaker 1: craftsman actually approach testing? Jorgensen highlights too sort of fundamental philosophies.
Speaker 2: Yeah, specification based testing versus code based testing.
Speaker 1: Let's unpack specification based first. That's also known as black
Speaker 1: box testing.
Speaker 2: Right, or functional testing. The core idea is you're designing
Speaker 2: your tests based purely on the software's requirements the specs,
Speaker 2: without looking at the internal code.
Speaker 1: Like testing a car, you use the steering wheel of
Speaker 1: the pedals, check the.
Speaker 2: Lights, But you don't need to know how the engine
Speaker 2: or the wiring actually works inside. You just know what
Speaker 2: it should do based on the user manual. Basically, that sounds.
Speaker 1: Really useful during development. What are the big advantages there?
Speaker 2: Well, A huge one is that the test cases are
Speaker 2: independent of the actual implementation, meaning even if the developers
Speaker 2: completely rewrite a section of code, as long as it's
Speaker 2: supposed to do the same thing according to the spec,
Speaker 2: your test case is still valid.
Speaker 1: Oh okay, that's powerful.
Speaker 2: And it also means testing can start earlier, maybe even
Speaker 2: happen in parallel with development, which can speed things up.
Speaker 1: But I'm guessing there's a downside.
Speaker 2: I catch there is because you're not looking inside, you
Speaker 2: can end up with redundant tests, multiple tests checking the
Speaker 2: same underlying logic without realizing it wasted effort, and more critically,
Speaker 2: you can have gaps, big blind spots where parts of
Speaker 2: the software might just never get tested because the spec
Speaker 2: didn't explicitly cover some weird internal case.
Speaker 1: Okay, So that leads us to the other side. Code
Speaker 1: based testing right.
Speaker 2: Sometimes called white box testing. Here you are looking at
Speaker 2: the source code. The tests are designed based on the
Speaker 2: program's structure, its paths its conditions.
Speaker 1: What's the main limitation there? Then it sounds more thorough.
Speaker 2: It can be for certain things, but its main weakness
Speaker 2: is identifying behaviors that were never programmed at all but
Speaker 2: should have been.
Speaker 1: Ah omissions, things missing from the requirements in the code exactly.
Speaker 2: Or think about something malicious like a trojan warp someone
Speaker 2: slipped in if it wasn't in the spec and it's
Speaker 2: just extra code doing bad stuff. Spec based won't find it,
Speaker 2: and code based testing might just test the paths through
Speaker 2: it without realizing its malicious intent. It struggles with things
Speaker 2: that aren't there but should be, or things that are
Speaker 2: there but shouldn't be if they weren't specified.
Speaker 1: So this sounds like a classic debate, black box versus
Speaker 1: white box? Is one just better?
Speaker 2: Well, if you step back, you see pretty quickly that
Speaker 2: neither one alone is really sufficient.
Speaker 1: Why not?
Speaker 2: Code based testing, like we said, won't find requirements that
Speaker 2: were completely missed in the lamentation, and spec based testing
Speaker 2: won't find extra, maybe unwanted behaviors that got coded in
Speaker 2: but weren't in the spec.
Speaker 1: So the craftsman's approach isn't about picking a side in
Speaker 1: this great debate.
Speaker 2: Exactly, it's not either. The real answer, the craftsman's answer
Speaker 2: is a smart combination.
Speaker 1: Okay, how does that work?
Speaker 2: You use both You design tests based on the specification
Speaker 2: to ensure functionality, and you use code based techniques, especially
Speaker 2: coverage metrics, to see what parts of the actual code
Speaker 2: those tests are exercising.
Speaker 1: Ah, So the code coverage tells you about the gaps
Speaker 1: and redencies in your spec based tests.
Speaker 2: Precisely, it gives you that measurement, that confidence. You get
Speaker 2: the functional assurance from spec based testing and the structural
Speaker 2: assurance and efficiency check from code based testing. It's about
Speaker 2: blending the strengths of both views.
Speaker 1: That makes a lot of sense, combining perspectives for a
Speaker 1: fuller picture. Okay, So with these foundational ideas, the error chain,
Speaker 1: the two approaches, how does the craftsmen apply this in
Speaker 1: a real project. Jorgensen talks about levels of testing often
Speaker 1: shown using the V model.
Speaker 2: Right. The V model is a variation of the older
Speaker 2: Waterfall model, but it's really useful because it visually emphasizes
Speaker 2: how testing activities should mirror development activities. How So, well,
Speaker 2: on the left side of the V you have the
Speaker 2: development phases going down requirements high level design, detailed design
Speaker 2: coding okay, and on the right side. Going up, you
Speaker 2: have the corresponding testing levels. Unit testing validates the code
Speaker 2: from detailed design, Integration testing validates the interfaces from high
Speaker 2: level design, and system testing validates the whole thing against
Speaker 2: the initial requirements.
Speaker 1: So each test level connects back to a design level exactly.
Speaker 2: It builds in quality checks at each stage rather than
Speaker 2: waiting until the very end. The idea is to catch
Speaker 2: faults as close as possible to where they.
Speaker 1: Were introduced, cheaper to fix them earlier, much much cheaper. Okay,
Speaker 1: So let's dive into that first level unit testing. This
Speaker 1: is where the craftsman is working on individual components, right,
Speaker 1: like a single function or class.
Speaker 2: Yep, the smallest testable pieces of the software. And there's
Speaker 2: a whole toolbox of techniques.
Speaker 1: For this level. Let's start with boundary value testing. You
Speaker 1: called these the off by one detectives earlier.
Speaker 2: Ah, yeah, that's a good way to think about it.
Speaker 2: The core idea is simple. Experience shows that programmers often
Speaker 2: make mistakes right at the edges the boundaries of input ranges,
Speaker 2: like using when.
Speaker 1: They meant or starting a loop counter at one instead
Speaker 1: of zero.
Speaker 2: Exactly those kinds of things off by one errors. So
Speaker 2: boundary value testing says, Okay, if an input is valid
Speaker 2: between one and one hundred, don't just test fifty.
Speaker 1: Test the edges.
Speaker 2: Test the minimum one, minimum plus one two a nominal
Speaker 2: value like fifty, maximum one ninety nine, and the maximum
Speaker 2: one hundred.
Speaker 1: Makes sense, and what about robust testing?
Speaker 2: Robust boundary value testing goes one step further. It says,
Speaker 2: let's also test values just outside the valid range, so
Speaker 2: minimum one zero in our example, and maximum plus one
Speaker 2: one oh one.
Speaker 1: Why test invalid inputs.
Speaker 2: Because that's often where unexpected crashes or even security vulnerabilities happen.
Speaker 2: How does the system handle bad data? Robust testing checks that.
Speaker 2: There's also worst case testing, which gets pretty complex testing
Speaker 2: combinations of boundary values.
Speaker 1: Okay, boundaries are critical. Yeah, but testing every boundary for
Speaker 1: every variable sounds like it could create a lot of tests.
Speaker 2: It can't, And that leads nicely into the next technique,
Speaker 2: equivalence class testing.
Speaker 1: Oh did that help?
Speaker 2: It's all about smart simplification. It tackles the potential redundancy
Speaker 2: you might get from just boundary testing. The idea comes
Speaker 2: from math, from partitions. Okay, you try to identify groups
Speaker 2: or classes of inputs that the program should treat exactly
Speaker 2: the same way. If you put in three, four, or five,
Speaker 2: and the code follows the exact same logic path for
Speaker 2: all of them, they form an equivalence class.
Speaker 1: Ah, so you don't need to test all three exactly.
Speaker 2: The assumption is if you test one representative value from
Speaker 2: that class, say four, it tells you how the program
Speaker 2: behaves for all the other values in that class three
Speaker 2: and five.
Speaker 1: That sounds much more efficient.
Speaker 2: Hugely efficient. It aims for completeness. Every possible input belongs
Speaker 2: to some class, and non redundancy no input belongs to
Speaker 2: more than one class. Ideally you get great coverage with
Speaker 2: fewer tests.
Speaker 1: Are there different types like with boundary testing.
Speaker 2: Yes, similar ideas. Weak normal tests one value from each
Speaker 2: valid class, Strong normal tests combinations of valid classes, and
Speaker 2: then weak robust and strong robust ad testing for the
Speaker 2: invalid equivalence classes inputs that should cause an error.
Speaker 1: So boundary values handle the edges. Equivalence classes handle the
Speaker 1: broad range efficiently. What if the logic itself is really complicated,
Speaker 1: lots of nested if statements or complex conditions.
Speaker 2: That's where decision table based testing shines. It's a very
Speaker 2: rigorous logical way to approach complex decision logic.
Speaker 1: How does it work? You literally build a.
Speaker 2: Table you do you list all the conditions, the inputs
Speaker 2: or system states that affect the decision, and all the
Speaker 2: possible actions what the software should do. Then you systematically
Speaker 2: map out every possible combination of condition outcomes true, false,
Speaker 2: and the corresponding action.
Speaker 1: Sounds like it could get huge.
Speaker 2: It can initially, but the real power comes when you
Speaker 2: analyze the table. You often find don't care conditions, situations
Speaker 2: where the outcome of one condition doesn't actually matter if
Speaker 2: another condition is met.
Speaker 1: Ah, So you can simplify the.
Speaker 2: Table exactly you collapse rules. Jorgensen uses the next date
Speaker 2: function exam, a function to calculate the date after a
Speaker 2: given date. The initial table might seem to have hundreds
Speaker 2: of rules when you consider day, month, year, leap yer rules.
Speaker 1: Yeah yeah, that sounds complicated.
Speaker 2: But by using decision tables and identifying those don't care
Speaker 2: conditions they could reduce it down to just a handful
Speaker 2: of essential test cases that covered all the logic. It
Speaker 2: shows how testing can actually improve the program's design by
Speaker 2: clarifying the logic.
Speaker 1: Pesting clarifying the code, not just finding bugs. I like that. Okay,
Speaker 1: so we've looked at inputs in logic. What about the
Speaker 1: actual flow of the code, the paths.
Speaker 2: It takes, right, that's path testing here. The graftsmanship's focus
Speaker 2: to the control flow through the program.
Speaker 1: How do you visualize that?
Speaker 2: You often use program graphs. Think of nodes as chunks
Speaker 2: of code like statement fragments, and edges as the flow
Speaker 2: of control between them, like an if statement creating a branch.
Speaker 2: We often look at ddpaths decision to decision.
Speaker 1: Paths, okay, paths between decisions.
Speaker 2: The goal is to design tests that exercise different paths
Speaker 2: through this graph. There are different levels of coverage you
Speaker 2: might aim for, Like what the simplest is node coverage
Speaker 2: or statement coverage. Just make making sure every single statement
Speaker 2: in the code gets executed at least once by some.
Speaker 1: Test seems like a minimum baseline.
Speaker 2: It is stronger is edge coverage or branch coverage. Making
Speaker 2: sure every possible outcome of every decision, like the shrewd
Speaker 2: and false branches of an if statement, gets executed at
Speaker 2: least once.
Speaker 1: That sounds more thorough it generally is.
Speaker 2: And this is where we often hear about cyclomatic complexity
Speaker 2: right VG.
Speaker 1: What does that number actually tell us?
Speaker 2: It's a metric calculated from the program graph number of
Speaker 2: edges minus number of nodes plus one. Essentially, it gives
Speaker 2: you the number of independent paths through the code. The
Speaker 2: independent paths basically the minimum number of paths you'd need
Speaker 2: to test to ensure you've covered every edge at least once.
Speaker 2: It's a measure of the code's structural complexity.
Speaker 1: Is there a rule of thumb for it?
Speaker 2: A common guideline is that if the cyclomatic complexity HERMBG
Speaker 2: gets above ten for a single function or module, that
Speaker 2: code is getting pretty complex. It'll likely be harder to understand,
Speaker 2: harder to test, and potentially more prone to errors.
Speaker 1: So it's a warning sign for developers and testers exactly.
Speaker 2: It suggests maybe breaking the code down into smaller, simpler pieces.
Speaker 1: But wait, can you have paths in the graph that
Speaker 1: look possible but you can't actually execute them?
Speaker 2: Absolutely? Those are called infeasible paths. It's a major headache
Speaker 2: in path testing. Why way happen because the structure of
Speaker 2: the graph doesn't always capture the semantic dependencies. Maybe path
Speaker 2: A set's a variable to true and path B requires
Speaker 2: that variable to be false. You can draw the path,
Speaker 2: but you can never actually make the program follow it.
Speaker 2: Designing tests for infeasible path is wasted effort.
Speaker 1: Tricky, and you mentioned something even more rigorous for critical systems.
Speaker 2: Yes, for safety critical stuff like aviation software level A,
Speaker 2: they often require modified condition decision coverage or MCDC.
Speaker 1: What does that involved.
Speaker 2: It's pretty intense. For every decision with multiple conditions like
Speaker 2: A and B or C, you need test cases that
Speaker 2: show that each individual condition A, B, and C can
Speaker 2: independently affect the outcome of the entire decision, while the
Speaker 2: other conditions are held constant.
Speaker 1: Wow, that's ensuring every part of the logic really matters precisely.
Speaker 2: It's about preventing situations where a condition seems to be
Speaker 2: tested but its effect is masked by other conditions. Extremely thorough.
Speaker 1: Okay, so path testing covers the flow, but what about
Speaker 1: the data itself? What happens to variables is they move
Speaker 1: along these paths?
Speaker 2: Excellent question. That brings us to data flow testing. This
Speaker 2: technique shifts the focus from just the control flow paths
Speaker 2: to the life cycle of variables within those paths.
Speaker 1: LIFECYC.
Speaker 2: Yeah, where does a variable get defined, get a value,
Speaker 2: and where does it get used? Its value is read.
Speaker 2: Data flow testing looks for paths between a definition of
Speaker 2: a variable and a subsequent use of that same variable.
Speaker 2: These are called definition use.
Speaker 1: Paths or do paths. Why is that important?
Speaker 2: Well, it helps catch errors like using a variable before
Speaker 2: it's been initialized, or defining a variable and then never
Speaker 2: actually using its value. It acts as a kind of
Speaker 2: reality check on pure path.
Speaker 1: Testing, so it connects the control flow with what's actually
Speaker 1: happening to the data exactly.
Speaker 2: There's a whole hierarchy of dataflow coverage criteria like all
Speaker 2: deaths tests at least one path from every definition, all
Speaker 2: uses test paths to every use. All the upaths test
Speaker 2: every simple definition use path. It's particularly good for object
Speaker 2: oriented code where data interactions can be complex.
Speaker 1: Makes sense, okay? One more in the unit testing toolbox
Speaker 1: program slicing. This sounds different.
Speaker 2: It is a bit different, but incredibly useful, especially for
Speaker 2: debugging and understanding code. A slice of a program relative
Speaker 2: to a specific variable at a specific point is the
Speaker 2: subset of program statements that could possibly affect the value
Speaker 2: of that variable at that point.
Speaker 1: So it's like highlighting only the relevant code exactly.
Speaker 2: You can do a backward slice starting from a vary goal,
Speaker 2: trace back everything that could have influenced its value, or
Speaker 2: a forward slice starting from where a variable is defined,
Speaker 2: see everything that it could possibly influence later on.
Speaker 1: I can see how that would help with debugging. Focuses
Speaker 1: your attention tremendously.
Speaker 2: It helps eliminate all the irrelevant detail and lets the
Speaker 2: craftsmen focus precisely where the problem might be. Jorgensen even
Speaker 2: suggests that developing programs in terms of compilable slices could
Speaker 2: be a powerful way to build and understand complex software.
Speaker 1: Interesting idea. Okay, so that's a lot of techniques just
Speaker 1: for unit testing, boundary values, equivalence classes, decision tables, path testing,
Speaker 1: data flow slicing. How does the craftsmen put it all together?
Speaker 1: You mentioned Jorgensen's testing pendulum earlier, right.
Speaker 2: Let's bring that back. It's that metaphor of testing swinging
Speaker 2: between the specification based view, the black box.
Speaker 1: High level functional requirement.
Speaker 2: Focused and the code based view, the white box.
Speaker 1: Low level structural implementation focused.
Speaker 2: And the pendulum highlights that neither extreme is sufficient on
Speaker 2: its own. The real skill the craft lies in the combination.
Speaker 1: How does that play out.
Speaker 2: Practically, A common approach is to start by choosing a
Speaker 2: specification based technique, maybe equivalence classes or boundary values, to
Speaker 2: define an initial set of tests based on what the
Speaker 2: system should do.
Speaker 1: Okay, get the functional coverage first.
Speaker 2: Then you run those tests and use code coverage tools
Speaker 2: which come from the code based world, to measure which
Speaker 2: parts of the code were actually executed.
Speaker 1: Ah, And that measurement reveals the gas exactly.
Speaker 2: It shows you the parts of the code your spec
Speaker 2: based tests didn't reach. It might also reveal redundancies. If
Speaker 2: multiple tests exercise the exact same code path, then you
Speaker 2: can design additional tests, perhaps using path or data flow
Speaker 2: ideas specifically to fill those gaps.
Speaker 1: So it's an iterative refinement using both perspectives.
Speaker 2: Precisely, you leverage the strengths of both. Jorgensen uses an
Speaker 2: insurance premium case study to show this how you'd pick
Speaker 2: different techniques based on whether variables are physical quantities or
Speaker 2: logical flags, whether faults are likely independent or interacting, and
Speaker 2: how to handle exceptions. It really emphasizes choosing the right
Speaker 2: tool or combination of tools for this job.
Speaker 1: The hallmark of a craftsman. Okay, let's zoom out. Now
Speaker 1: we've covered the unit level in detail. How does testing
Speaker 1: fit into the bigger picture across different software development life cycles?
Speaker 1: Good question.
Speaker 2: The traditional waterfall model, as we touched on with the
Speaker 2: V model, was very linear. Design everything first, then build it,
Speaker 2: then test it in stages unit integrash system. It kind
Speaker 2: of assumed you could know everything perfectly upfront.
Speaker 1: Which rarely happens in reality.
Speaker 2: Exactly, so iterative models emerge, things like incremental development, evolutionary prototyping,
Speaker 2: the spiral model. The big shift there was moving away
Speaker 2: from doing everything in one big chunk to doing things
Speaker 2: in smaller cycles, building and testing parts of the system,
Speaker 2: getting feedback, and then.
Speaker 1: Iterating more flexible, more adaptive.
Speaker 2: Much more. And this really paved the way for agile testing.
Speaker 1: What are the key characteristics of testing in an agile world?
Speaker 2: Agile testing is fundamentally driven by customer needs and feedback.
Speaker 2: It's typically bottom up, focusing on delivering working software components
Speaker 2: early and often, and it absolutely embraces changing requirements.
Speaker 1: Flexibility is key.
Speaker 2: Absolutely. Two big examples you hear about are Extreme Programming
Speaker 2: XP and Test driven development TDDDDD.
Speaker 1: That's the one where we write the test first.
Speaker 2: Right that's the one. It sounds backward, but it's powerful.
Speaker 2: In TDD, developers work in very short cycles. First, write
Speaker 2: an automated test case for a tiny piece of functionality
Speaker 2: that doesn't exist yet. The test will obviously.
Speaker 1: Fail because the code isn't there right.
Speaker 2: Then write the minimum amount code needed to make that
Speaker 2: test pass okay, and then importantly refactor the code, clean
Speaker 2: it up, improve its design, while continually rerunning all the
Speaker 2: tests to make sure nothing broke.
Speaker 1: So the tests act like a specification in a safety
Speaker 1: net precisely.
Speaker 2: It requires good automated testing frameworks like JUnit for Java,
Speaker 2: for instance, and it really keeps the focus on building
Speaker 2: exactly what's needed and keeping the code based clean.
Speaker 1: What about Scrum? How does testing fit in there?
Speaker 2: Rum organizes work into sprints, usually two four weeks long.
Speaker 2: Teams have daily stand up meetings to coordinate, and the
Speaker 2: goal is to produce a potentially shippable increment of software
Speaker 2: at the end of each sprint. Testing is integrated throughout
Speaker 2: the sprint, often with continuous integration and automated builds happening.
Speaker 1: Daily, so very rapid cycles and feedback very rapid.
Speaker 2: Some say scrum is mostly new names for old ideas,
Speaker 2: but that intense focus on short intervals and daily integration
Speaker 2: really forces testing to be a continuous activity, not an afterthought.
Speaker 1: Beyond specific methodologies, there's also model based testing MBT. What's
Speaker 1: the idea there?
Speaker 2: NBT takes a more abstract approach. Instead of deriving tests
Speaker 2: directly from requirements, text, or code, you first build a
Speaker 2: formal model of the system's.
Speaker 1: Behavior, like a flow chart or a state machine.
Speaker 2: Could be finite state machines, petri nets, state charts. There
Speaker 2: are various modeling languages. The key advantage, Jorgensen argues is
Speaker 2: that the act of building the model forces you to
Speaker 2: gain deeper insights and understanding of the system. You have
Speaker 2: to really think through the state's transitions and behaviors.
Speaker 1: Now the model itself is valuable immensely.
Speaker 2: Then you use algorithms or heuristics to automatically or semi
Speaker 2: automatically generate test cases from the model. You run the tests,
Speaker 2: analyze results, and potentially refine the model or the system.
Speaker 2: Some models are more powerful than others. Peterson's lattice shows
Speaker 2: a hierarchy where things like beatri nets can express concurrency
Speaker 2: that simpler finite state machines can't.
Speaker 1: Interesting okay, moving up the V model. Let's top integration
Speaker 1: testing you mentioned earlier. This is often seen as tricky.
Speaker 2: Yeah. It's often called the lyst well understood, and most
Speaker 2: poorly done phase. This is where you start combining those
Speaker 2: units that were tested individually, and you're looking for problems
Speaker 2: at the interfaces between them where things connect exactly. Maybe
Speaker 2: Unit A passes data incorrectly to unit B, or they
Speaker 2: make different assumptions about shared data. These kinds of issues
Speaker 2: only appear when you put them together.
Speaker 1: How did people traditionally approach this?
Speaker 2: Strategies were often decomposition based, like top down integration, where
Speaker 2: you start with the main module and use stubs dummy
Speaker 2: modules to simulate the ones it calls, or bottom up
Speaker 2: where you start with the lowest level units and use
Speaker 2: drivers to call them, building upwards or sandwich a mix
Speaker 2: of both. What was the problem with those A key
Speaker 2: issue is that the decomposition tree used for planning might
Speaker 2: not match the actual call structure of the program. This
Speaker 2: could lead you to design tests for interactions between units
Speaker 2: that never actually call each other directly impossible.
Speaker 1: Test pair wasted effort again, right.
Speaker 2: So a better approach is call graph based integration. You
Speaker 2: look at the real call graph units or nodes, calls
Speaker 2: or edges and base your integration tests on actual feasible
Speaker 2: paths between units.
Speaker 1: Makes more sense. What about for object oriented systems?
Speaker 2: For UVO software, the concept of MM paths method message
Speaker 2: paths is useful. An MM path represents a sequence of
Speaker 2: method executions within a single object, interspersed with messages sent
Speaker 2: to other objects, which in turn trigger their methods. It
Speaker 2: helps trace feasible execution flows across object boundaries during integration.
Speaker 1: Okay, ensuring those interactions work. Finally, we reached the top
Speaker 1: of the v system, testing the whole show bang.
Speaker 2: Yep, testing the entire integrated system as a black box.
Speaker 2: Usually from the end user's perspective. Here, we often think
Speaker 2: in terms of atomic system functions ASFS. It's basically the
Speaker 2: smallest observable interaction a user can have with the system
Speaker 2: that accomplishes something meaningful, like inserting your card at an ATM,
Speaker 2: entering your PI, requesting a balance. Each of those is
Speaker 2: an ASF.
Speaker 1: Got it and we also use use cases here right heavily.
Speaker 2: Use cases describe sequences of actions involving the user and
Speaker 2: the system to achieve a specific goal. They can be
Speaker 2: high level user stories or very detailed step by step
Speaker 2: real use cases specifying inputs, outputs, preconditions, post conditions. They
Speaker 2: really define the system's behavior from that user viewpoint.
Speaker 1: So system testing is very much about does it do
Speaker 1: with the user needs exactly?
Speaker 2: And to prioritize system testing, we often use operational profiles.
Speaker 1: What do those tell you?
Speaker 2: An operational profile tries to estimate how frequently different features
Speaker 2: or sequences of actions threads will actually be used in
Speaker 2: the real world. You gather statistics or make educated guesses
Speaker 2: about usage.
Speaker 1: Patterns that it's so useful.
Speaker 2: It allows you to focus your testing effort on the
Speaker 2: high traffic areas, the parts of the system that users
Speaker 2: will interact with most often. Finding and fixing bugs there
Speaker 2: gives you the biggest quality improvement for the user experience.
Speaker 2: It's based on probabilities, often derived from models like finite
Speaker 2: state machines representing usage smart prioritization.
Speaker 1: Yeah, and system testing isn't just about function right, there's
Speaker 1: non functional testing.
Speaker 2: Too, absolutely crucial. This includes things like performance testing, security testing,
Speaker 2: usability testing, and stress testing.
Speaker 1: Pushing the system to its limit.
Speaker 2: Yeah, seeing how it behaves under extreme load or adverse conditions.
Speaker 2: How many users can handle concurrently? What happened when the
Speaker 2: database gets huge? How does it recover from network failures?
Speaker 2: Craftsmen use strategies like compression simulating large loads with smaller
Speaker 2: representative test sets, or replication simulating potentially destructive tests in
Speaker 2: a controlled environment.
Speaker 1: Okay, that covers the main levels. Let's shift to some
Speaker 1: more advanced perspectives. How do we even test systems made
Speaker 1: of other systems? Systems of systems sos?
Speaker 2: AH? Yes, SOS testing is a huge challenge. These are
Speaker 2: systems built by integrating large scale, often independently managed, constituent
Speaker 2: systems think air traffic control or complex financial networks.
Speaker 1: They weren't necessarily designed to work together initially.
Speaker 2: Often not. Jorgensen describes different types based on how much
Speaker 2: central control exists. Directed sos are built for a specific purpose,
Speaker 2: like a smart home system. Acknowledged sos have independent systems
Speaker 2: that know about each other and work together loosely, like
Speaker 2: maybe coordinating traffic lights across the city. Collaborative sos have
Speaker 2: systems that volunteer to work together for a shared goal,
Speaker 2: maybe during an emergency response. And virtual sos have no
Speaker 2: central authority at all. They just interoperate dynamically when needed.
Speaker 1: Like parts of the Internet, testing those interactions must be
Speaker 1: incredibly complex.
Speaker 2: It is you have to understand the communication protocols, the
Speaker 2: potential emergent behaviors that arise from the interactions. Techniques involve
Speaker 2: modeling the communications, perhaps using things like Petree nets mapped
Speaker 2: to interface specifications, to understand how these independent giants influence
Speaker 2: each other.
Speaker 1: A whole other level of complexity. Now, let's come back
Speaker 1: to something fundamental, evaluating our own tests. After all this effort,
Speaker 1: how does a craftsmen know if their test cases are
Speaker 1: actually any good?
Speaker 2: That's the million dollar question, isn't it? How good are
Speaker 2: my tests? You need ways to assess their effectiveness. One classic,
Speaker 2: powerful technique is mutation testing.
Speaker 1: You mentioned this briefly. How does it work again?
Speaker 2: You take the original program that presumably passes all your
Speaker 2: current tests. Then you automatically introduce lots of small, single
Speaker 2: chain mutants. Each mutant version of the program has one
Speaker 2: tiny fault deliberately inserted.
Speaker 1: Like changing a plus to a flaint or to a frame.
Speaker 2: Exactly simple syntactic changes. Then you rerun your entire test
Speaker 2: suite against each mutant program.
Speaker 1: What are you looking for?
Speaker 2: You're looking for your test suite to kill the mutant,
Speaker 2: meaning at least one test case fails when run against
Speaker 2: that mutated code. If a mutant runs through your entire
Speaker 2: test suite and doesn't cause any test to fail, it's
Speaker 2: called a live mutant.
Speaker 1: Yeah, that's bad.
Speaker 2: That's bad. It means your test suite has a blind spot.
Speaker 2: It wasn't sensitive enough to detect that specific small change,
Speaker 2: which might represent a real potential bug. The ratio of
Speaker 2: killed mutants to total mutants gives you a mutation score,
Speaker 2: a quantitative measure of your test suite's thoroughness.
Speaker 1: That urban legend about the mariner I probe.
Speaker 2: Yeah, the story goes it failed due to a single
Speaker 2: character typo in the code. Something. Mutation testing is designed
Speaker 2: to catch. Whether the legend is perfectly accurate or not,
Speaker 2: It highlights the principle small mistakes can have huge consequences,
Speaker 2: and mutation testing helps find weaknesses and tests that might
Speaker 2: miss them, so.
Speaker 1: It tests the tests. What about fuzzing? That sounds less systematic.
Speaker 2: It is less systematic, but surprisingly effective, especially for finding
Speaker 2: security vulnerabilities and robustness issues. Fuzzing basically involves throwing large
Speaker 2: amounts of random, semi random or malformed data out of
Speaker 2: program's inputs.
Speaker 1: Just trying to break it pretty much.
Speaker 2: You're hoping that some unexpected inport combination will trigger a crash,
Speaker 2: an assertion failure, a memory leak, or some other vulnerability.
Speaker 2: It was famously discovered by accident when random noise on
Speaker 2: a modem line caused Unix utilities to crash. The noise
Speaker 2: was effectively fuzzing them.
Speaker 1: Accidental discovery. Okay, And then there's that analogy for ecology, phishing,
Speaker 1: creole counts, or fault insertion.
Speaker 2: Right. This is another way to estimate test effectiveness, or
Speaker 2: rather estimate the number of remaining faults. The analogy is
Speaker 2: wildlife managers tagging some fish, releasing them, and then using
Speaker 2: the portion of tagged fish caught later to estimate the
Speaker 2: total fish population.
Speaker 1: How does that apply to software?
Speaker 2: You tag the software by intentionally inserting a known number
Speaker 2: of representative faults seated faults. These should ideally be similar
Speaker 2: in nature to the kinds of real faults you expect.
Speaker 2: Then you run your test suite.
Speaker 1: And see how many of the seated faults it finds.
Speaker 2: Exactly. If your tests find, say eighty percent of the faults,
Speaker 2: you deliberately inserted. You might infer that they've also likely
Speaker 2: found about eighty percent of the original wild false that
Speaker 2: were already in the code. It gives you a statistical
Speaker 2: estimate of the remaining defects.
Speaker 1: Clever, very clever. Okay, Finally, let's talk of the human
Speaker 1: side of quality software. Technical reviews. Are these really a
Speaker 1: form of testing?
Speaker 2: Jorgenson argues strongly that they are. They're a form of
Speaker 2: static testing without executing the code. Reviews aim to find
Speaker 2: faults errors in code design documents, requirements before they have
Speaker 2: a chance to become failures during execution.
Speaker 1: And the economics are compelling, right.
Speaker 2: Hugely compelling. Barry Boehm's classic research showed graphically that the
Speaker 2: cost to fix a fault increases exponentially the later in
Speaker 2: the development cycle.
Speaker 1: It's found.
Speaker 2: Finding a fault in a requirements document during a review
Speaker 2: might cost pennies compared to finding the same underlying misunderstanding
Speaker 2: after the system is deployed. It's the ultimate stitch in
Speaker 2: time saves nine.
Speaker 1: So what kinds of reviews are there?
Speaker 2: There is a spectrum of formality. Walkthroughs are usually the
Speaker 2: least formal, often led by the author of the work product,
Speaker 2: code design doc their effectiveness can vary a lot depending
Speaker 2: on the author's goals. At the other end is the
Speaker 2: technical inspection, pioneered by Michael Fagan at IBM in the seventies.
Speaker 2: These are highly structured and generally considered the most effective
Speaker 2: type of review.
Speaker 1: What makes them so.
Speaker 2: Formal inspections involve documented processes, specific roles aither producer, moderator, leader, reader,
Speaker 2: recorder inspectors, formal training for participants, budgeted time, detailed checklists
Speaker 2: based on common error types, and a focus on finding
Speaker 2: and logging defects, not fixing them during the meeting. Places
Speaker 2: like telephone switching Labs developed incredibly rigorous inspection processes over
Speaker 2: decades to achieve near fault free systems.
Speaker 1: It sounds like a very disciplined process. What does it
Speaker 1: take to make reviews work well in an organization? It's
Speaker 1: not just process, is it?
Speaker 2: Definitely not? Reviews are a deeply social process. For them
Speaker 2: to be effective, the culture has to support them as
Speaker 2: they need to be seen as valuable, a worthwhile investment
Speaker 2: of time, not just overhead. Time needs to be budgeted
Speaker 2: for preparation and participation. There needs to be clear etiquette.
Speaker 2: You review the product, not the producer, the goal is
Speaker 2: defect detection, not solution brainstorming or blaming. Keep it constructive, absolutely,
Speaker 2: and generally it's best of direct managers don't participate in
Speaker 2: the actual review meeting itself. That helps create a psychologically
Speaker 2: safer environment where people feel more comfortable pointing out potential
Speaker 2: issues without fear of evaluation.
Speaker 1: Makes sense, create the right environment for finding faults. So
Speaker 1: this up, What does this whole deep dive mean for you,
Speaker 1: our listener? We've journeyed through this intricate world of software testing.
Speaker 2: Yeah, from the basic definitions of errors and failures, through
Speaker 2: all those unit testing techniques, the different life cycle approaches, integration,
Speaker 2: system testing.
Speaker 1: And finally looking at advanced topics like systems of systems,
Speaker 1: evaluating our tests, and the critical role of technical reviews.
Speaker 2: And throughout it all we've seen that software testing, when
Speaker 2: done well, really is a craft.
Speaker 1: What does that mean?
Speaker 2: Ultimately, it means it requires deep knowledge of the principles
Speaker 2: and techniques. It requires this skill to choose the right
Speaker 2: tools for the job, just like any artisan, It requires
Speaker 2: extensive experience to build intuition, and fundamentally, it requires a
Speaker 2: dedication to high quality work, a sense of pride in
Speaker 2: doing the job well.
Speaker 1: Well said, it's about building confidence and quality through skill
Speaker 1: and diligence. So for your provocative thought this week, step
Speaker 1: back and consider how these principles of software testing might
Speaker 1: apply beyond just code. Think about identifying folds early, rigorously
Speaker 1: evaluating the effectiveness of your methods, or understanding complex systems
Speaker 1: through their inputs, outputs, and interactions. Where in your own work,
Speaker 1: or even your life might you benefit from applying a
Speaker 1: bit of this craftsmanlike testing mindset. Where could a deep
Speaker 1: dive using some of these concepts reveal ways to improve
Speaker 1: quality or prevent problems?
Speaker 2: Something to think about