WEBVTT 1 00:00:00.120 --> 00:00:03.240 You know that feeling right when you've just finished building 2 00:00:03.279 --> 00:00:04.240 a data visualization. 3 00:00:04.559 --> 00:00:05.719 Oh, I know it all too well. 4 00:00:05.799 --> 00:00:08.800 Right, it looks pristine. I mean, the axes aligned perfectly, 5 00:00:09.279 --> 00:00:12.160 the color palette is accessible, and the data is just 6 00:00:12.400 --> 00:00:14.119 it tells this beautifully clear story. 7 00:00:14.240 --> 00:00:15.880 It's a masterpiece exactly. 8 00:00:16.359 --> 00:00:19.079 So you share your screen with a team and someone 9 00:00:19.160 --> 00:00:24.600 leans in, squints and says, hey, this is fantastic. Could 10 00:00:24.600 --> 00:00:26.440 we just make one tiny tweak. 11 00:00:26.920 --> 00:00:28.199 It's never a tiny tweak. 12 00:00:28.320 --> 00:00:31.039 Never, they say, maybe add a second version of that 13 00:00:31.160 --> 00:00:33.159 exact chart right next to it, but you know, just 14 00:00:33.200 --> 00:00:37.200 filtered for a different region. And suddenly your stomach just drops. 15 00:00:36.880 --> 00:00:38.679 Because you know exactly what's about to happen. 16 00:00:38.840 --> 00:00:41.920 Yeah, because to make that one quote unquote tiny tweak, 17 00:00:42.320 --> 00:00:45.479 you are about to get totally lost in like five 18 00:00:45.560 --> 00:00:49.600 hundred lines of fragile, unreadable spaghetti code. You pull on 19 00:00:49.600 --> 00:00:51.840 one thread and the whole layout collapses. 20 00:00:52.039 --> 00:00:54.399 Is the ultimate developer's curse. I mean, it's a request 21 00:00:54.439 --> 00:00:58.159 that sounds conceptually so simple to the user, but structurally 22 00:00:58.439 --> 00:01:01.719 it basically requires tearing down the entire foundation of the 23 00:01:01.759 --> 00:01:03.679 application to execute. 24 00:01:03.840 --> 00:01:07.480 Well, we are tackling that exact curse today. Our mission 25 00:01:07.480 --> 00:01:09.480 for this deep dive is to rescue you from the 26 00:01:09.560 --> 00:01:12.920 dreaded one off visualization trap. We are looking at a 27 00:01:12.920 --> 00:01:16.439 book called Developing a D three JS edge. 28 00:01:16.400 --> 00:01:21.120 By Chris Viao and uh Andrew Thornton ger Hobbled and 29 00:01:21.239 --> 00:01:22.400 Rolling done. 30 00:01:22.280 --> 00:01:25.359 Right, that's the one, and we're pulling apart the architecture 31 00:01:25.480 --> 00:01:28.120 used by Mike Bostock, the creator of D three himself, 32 00:01:28.159 --> 00:01:32.560 to transform static, you know, messy code into a scalable 33 00:01:32.840 --> 00:01:34.920 modular visualization engine, which is. 34 00:01:34.879 --> 00:01:38.120 So critical because before we can build anything scalable, we 35 00:01:38.159 --> 00:01:41.560 really need to look at why the normal way people 36 00:01:41.640 --> 00:01:44.040 learn to code these charts eventually breaks down. 37 00:01:44.159 --> 00:01:47.159 Yeah, let's start there. The book outlines this classic scenario, right, 38 00:01:47.200 --> 00:01:49.400 a standard bar chart showing the frequency of letters in 39 00:01:49.439 --> 00:01:51.480 the English alphabet, so A shows up eight point one 40 00:01:51.480 --> 00:01:53.519 percent of the time, E is at twelve point seven percent, 41 00:01:53.560 --> 00:01:53.959 and so on. 42 00:01:54.400 --> 00:01:56.599 And for a beginner, the typical flow for building that 43 00:01:56.719 --> 00:01:58.640 chart is honestly, it's completely. 44 00:01:58.280 --> 00:01:59.359 Logical, right, it makes sense. 45 00:01:59.359 --> 00:02:03.719 At first, you established some conventional margins for the kansas, 46 00:02:03.920 --> 00:02:06.200 You define an ordinal scale for your X axis, which 47 00:02:06.439 --> 00:02:09.000 discreetly spaces out those twenty six letters, and. 48 00:02:09.000 --> 00:02:11.400 Then you set a linear scale for your y axis 49 00:02:11.479 --> 00:02:15.479 right to dynamically map those percentage of values to actual 50 00:02:15.520 --> 00:02:17.000 pixel heights on the screen. 51 00:02:17.199 --> 00:02:19.159 Yeah, and then you run into the core mechanic of 52 00:02:19.240 --> 00:02:22.719 D three, which is the enter, update and exit. 53 00:02:22.439 --> 00:02:24.039 Pattern, the famous pattern. 54 00:02:24.159 --> 00:02:27.719 Right, you bind your alphabet array to SVG rectangle elements. 55 00:02:28.599 --> 00:02:32.000 The interphase looks for data points that don't have a 56 00:02:32.039 --> 00:02:35.159 matching rectangle yet, and it creates a placeholder. 57 00:02:35.199 --> 00:02:37.800 You style those rectangles, set the width and height, and 58 00:02:37.919 --> 00:02:40.360 boom you have a beautiful bar chart right there on 59 00:02:40.400 --> 00:02:40.759 the screen. 60 00:02:40.840 --> 00:02:43.439 And it works flawlessly, I mean for a single instance. 61 00:02:44.120 --> 00:02:48.080 But the critical flaw emerges the exact second you attempt 62 00:02:48.120 --> 00:02:50.759 to introduce that second bar chart to the page. 63 00:02:50.479 --> 00:02:53.240 Which is what they always asked for. And are my instinct, 64 00:02:53.280 --> 00:02:55.680 and I think honestly the instinct of like ninety percent 65 00:02:55.719 --> 00:02:59.199 of developers working on tight deadlines is to just highlight 66 00:02:59.240 --> 00:03:01.759 that entire block of code, hit copy and paste it 67 00:03:01.840 --> 00:03:03.159 right below the original block. 68 00:03:03.199 --> 00:03:04.439 Oh, absolutely, we've all done it. 69 00:03:04.639 --> 00:03:07.360 Yeah, you just quickly renamed some variables, like charting with 70 00:03:07.599 --> 00:03:10.120 becomes chart two with. But if you need, say ten 71 00:03:10.240 --> 00:03:13.800 charts on a dashboard, you now have ten identical, massive 72 00:03:13.919 --> 00:03:14.719 code blocks. 73 00:03:15.000 --> 00:03:18.719 And consider what happens when a layout requirement changes, because 74 00:03:18.759 --> 00:03:24.840 it always does. The client reviews the dashboard and decides, actually, 75 00:03:24.879 --> 00:03:28.719 the padding between the bars is too tight. Oh yeah, right. 76 00:03:28.840 --> 00:03:31.439 You can't just change a global variable anymore because each 77 00:03:31.560 --> 00:03:34.400 chart has slightly different constraints. Now you have to hunt 78 00:03:34.400 --> 00:03:37.520 down the padding logic in ten different places within the 79 00:03:37.520 --> 00:03:38.159 code bases. 80 00:03:38.280 --> 00:03:40.639 You've introduced massive redundancy. 81 00:03:40.159 --> 00:03:44.199 And mathematically, you have draftically increased the probability of human error. 82 00:03:44.520 --> 00:03:46.879 So imagine it like this. You need to go to 83 00:03:46.879 --> 00:03:49.960 the grocery store, right, So you spend a week building 84 00:03:50.000 --> 00:03:51.800 a custom car from scratch in your. 85 00:03:51.719 --> 00:03:53.199 Driveway, as one does, right. 86 00:03:53.400 --> 00:03:56.319 Naturally, you drive to the store and it works perfectly. 87 00:03:56.680 --> 00:03:58.319 But the next day you need to go to the 88 00:03:58.319 --> 00:04:00.759 hardware store, and instead of getting back the car you 89 00:04:00.759 --> 00:04:04.840 already built, you build a second, completely identical car from scratch, 90 00:04:05.199 --> 00:04:07.360 specifically to drive to the hardware store. 91 00:04:07.560 --> 00:04:10.280 That is exactly what standard D three scripting is. Like, 92 00:04:10.719 --> 00:04:13.599 it functions, but it is a horrific way to operate 93 00:04:13.639 --> 00:04:14.800 a transportation system. 94 00:04:14.919 --> 00:04:17.160 Exactly. It's so wildly inefficient. 95 00:04:17.399 --> 00:04:19.560 But you know there is a reason tutorials are written 96 00:04:19.600 --> 00:04:22.879 this way. D three tutorials are structured to teach core 97 00:04:22.920 --> 00:04:27.120 functionality right, like how SVG coordinates work or how data 98 00:04:27.199 --> 00:04:28.240 joins function. 99 00:04:28.040 --> 00:04:31.920 They aren't meant to be enterprise architecture guides exactly. 100 00:04:31.959 --> 00:04:35.399 They are not designed to teach large scale software architecture. 101 00:04:35.639 --> 00:04:39.639 D three gives you extreme low level freedom. You aren't 102 00:04:39.680 --> 00:04:43.680 restricted to basic configuration menus like you are in say 103 00:04:44.120 --> 00:04:48.560 Excel or Tableau. But that freedom dictates that you are 104 00:04:48.720 --> 00:04:52.279 entirely responsible for architecting your own application state. It's on 105 00:04:52.360 --> 00:04:53.959 you to keep your own house organized. 106 00:04:54.240 --> 00:04:56.800 Okay, so we need to stop building single use cars 107 00:04:56.839 --> 00:04:59.600 in the driveway. We need to stop copy pasting code blocks. 108 00:05:00.079 --> 00:05:02.600 The source material argues that the solution is moving away 109 00:05:02.639 --> 00:05:05.720 from building a chart and moving toward building an API. 110 00:05:05.519 --> 00:05:08.639 An application programming interface. Yes, the fundamental goal has to 111 00:05:08.680 --> 00:05:11.680 shift here. Instead of writing a script that executes sequentially 112 00:05:11.720 --> 00:05:13.680 to draw a single chart, we want to write a 113 00:05:13.720 --> 00:05:16.959 chart generator. A generator, right, We want a standalone function 114 00:05:17.000 --> 00:05:20.000 that we can call with different parameters which then returns 115 00:05:20.040 --> 00:05:23.519 a brand new chart object that is perfectly isolated from 116 00:05:23.519 --> 00:05:24.759 any other chart on the screen. 117 00:05:25.040 --> 00:05:27.800 And to achieve that, the authors lean into a very 118 00:05:27.839 --> 00:05:33.480 specific programming philosophy, which is composition over inheritance. 119 00:05:33.279 --> 00:05:39.360 Yes, composition over inheritance. In traditional object oriented programming, developers 120 00:05:39.439 --> 00:05:40.879 often rely heavily. 121 00:05:40.519 --> 00:05:42.720 On inheritance, like having a parent class. 122 00:05:43.040 --> 00:05:46.079 Exactly, you define a base chart class with a title 123 00:05:46.120 --> 00:05:49.000 and an access and then a bar chart class inherits 124 00:05:49.040 --> 00:05:52.079 those properties, yea, and maybe a stacked bar chart inherits 125 00:05:52.079 --> 00:05:52.319 from that. 126 00:05:52.439 --> 00:05:54.360 But D three doesn't really like that, does it. No. 127 00:05:54.600 --> 00:05:58.839 D three's ecosystem strongly prefers composition. You don't inherit a 128 00:05:58.879 --> 00:06:02.759 monolithic chart. You assemble one by composing smaller independent modules. 129 00:06:02.800 --> 00:06:05.360 So you take a rectangle module, an access module, and 130 00:06:05.360 --> 00:06:07.560 maybe a tooltip module, and you just snap them together 131 00:06:07.639 --> 00:06:08.199 like legos. 132 00:06:08.279 --> 00:06:10.120 Exactly like legos, the text. 133 00:06:09.800 --> 00:06:12.759 Outlines a Hello World pattern for this. To start building 134 00:06:12.759 --> 00:06:15.680 our modular API, we basically wrap all of our chart 135 00:06:15.720 --> 00:06:18.360 generating code inside a single outer function. 136 00:06:18.319 --> 00:06:21.319 Which creates what's called a closure in JavaScript. 137 00:06:20.800 --> 00:06:23.480 Right, a closure, And from what I understand, this closure 138 00:06:23.560 --> 00:06:27.839 effectively builds a wall around our internal variables, things like 139 00:06:27.879 --> 00:06:30.600 the width of the chart or the font color. Those 140 00:06:30.680 --> 00:06:34.600 variables become entirely private, yes, like nothing else running on 141 00:06:34.639 --> 00:06:36.319 the web page can access. 142 00:06:35.959 --> 00:06:39.480 Them, and keeping those variables private prevents the name space 143 00:06:39.519 --> 00:06:43.639 pollution that causes so many bugs in complex dashboards. If 144 00:06:43.639 --> 00:06:46.040 a separate script on the page happens to use a 145 00:06:46.120 --> 00:06:49.639 variable named width, it won't accidentally overwrite the width of 146 00:06:49.680 --> 00:06:50.600 your specific chart. 147 00:06:50.800 --> 00:06:52.279 Okay. I have to push back on this a little 148 00:06:52.279 --> 00:06:57.720 bit because wrapping functions inside functions, creating these artificial walls 149 00:06:57.759 --> 00:07:00.319 just to draw some bars on a screen, it feels 150 00:07:00.360 --> 00:07:02.879 like we are adding a massive layer of complexity for 151 00:07:02.959 --> 00:07:05.920 no reason. I mean, why not just pass a configuration 152 00:07:06.000 --> 00:07:07.959 object into a standard function. 153 00:07:07.839 --> 00:07:08.800 Like a static object? 154 00:07:08.920 --> 00:07:11.199 Yeah, just hand the chart an object that says, you know, 155 00:07:11.279 --> 00:07:14.759 with five hundred color blue and be done with it. 156 00:07:15.120 --> 00:07:18.000 Why all the inception style functions, It's a fair question. 157 00:07:18.439 --> 00:07:21.519 Passing a static configuration object works fine if the chart 158 00:07:21.600 --> 00:07:24.399 never needs to change after it is drawn. Okay, but 159 00:07:24.560 --> 00:07:28.759 data visualizations in modern applications are rarely static. They need 160 00:07:28.800 --> 00:07:34.240 to respond to window resizing, data filtering, user inputs. If 161 00:07:34.240 --> 00:07:37.839 you just pass a static object, updating the chart requires 162 00:07:37.879 --> 00:07:40.879 tearing it down entirely and passing a whole new object. 163 00:07:40.959 --> 00:07:41.560 Oh. I see. 164 00:07:41.839 --> 00:07:44.720 The closure pattern allows the chart to maintain its own 165 00:07:44.759 --> 00:07:48.120 internal state, and to interact with that secure internal state, 166 00:07:48.560 --> 00:07:52.600 we use a very specific mechanism getter and setter functions. 167 00:07:52.680 --> 00:07:55.120 Right. Okay, so if the closure is a wall, the 168 00:07:55.120 --> 00:07:56.759 getters and setters are the windows. 169 00:07:56.920 --> 00:07:58.399 That's a great way to look at it, But the. 170 00:07:58.360 --> 00:08:00.759 Way they implement it here is pretty unique. The getter 171 00:08:00.800 --> 00:08:03.199 and the setter are actually the exact same function. 172 00:08:03.360 --> 00:08:03.959 Yes they are. 173 00:08:04.120 --> 00:08:06.079 So if I want to control the width, I call 174 00:08:06.120 --> 00:08:09.319 a function named dot and if I pass an argument 175 00:08:09.360 --> 00:08:11.920 into it, say the number five hundred, it acts as 176 00:08:11.920 --> 00:08:15.399 a setter. It updates the internal private variable to five hundred. 177 00:08:15.680 --> 00:08:18.720 What does that setter function return after it updates the variable? 178 00:08:19.000 --> 00:08:22.399 It returns this, like literally the word this. It returns 179 00:08:22.399 --> 00:08:25.759 the execution context of the chart instance itself exactly. And 180 00:08:25.759 --> 00:08:28.360 because it returns itself, you can chain the functions together. 181 00:08:28.439 --> 00:08:30.439 I don't have to write three separate lines of code 182 00:08:30.480 --> 00:08:32.919 to update the chart. I can just type dot width 183 00:08:32.919 --> 00:08:36.879 five hundred, dot height three hundred, dot color blue, all 184 00:08:36.919 --> 00:08:38.360 in one continuous chain. 185 00:08:38.519 --> 00:08:41.679 That chainability is the absolute hallmark of the D three 186 00:08:41.720 --> 00:08:45.000 API pattern. And if you call that exact same dot 187 00:08:45.039 --> 00:08:48.399 width function without passing an argument into it, the fund 188 00:08:48.399 --> 00:08:50.840 can detect that the argument is missing and it acts 189 00:08:50.879 --> 00:08:51.399 as a getter. 190 00:08:51.799 --> 00:08:53.519 Oh so it just tells you the current width. 191 00:08:53.600 --> 00:08:56.519 It simply returns the current internal value. Yeah, you have 192 00:08:56.559 --> 00:09:00.399 an elegant, unified way to query or my notify the 193 00:09:00.399 --> 00:09:02.399 state of the chart at any point in its life cycle. 194 00:09:02.480 --> 00:09:04.879 That is super clean. Now, while we are looking at 195 00:09:04.919 --> 00:09:06.919 the code for these modules, I have to bring up 196 00:09:06.960 --> 00:09:09.200 a moment where I completely geeked out over the source 197 00:09:09.240 --> 00:09:12.759 material which part I noticed a very strange syntax choice. 198 00:09:12.799 --> 00:09:15.679 When they were generating random data to test the charts, 199 00:09:16.360 --> 00:09:19.840 instead of using standard math functions to round numbers, they 200 00:09:19.919 --> 00:09:23.960 placed a double tilled operator like two little squiggly lines 201 00:09:23.960 --> 00:09:27.320 in front of the variables like till the variable. 202 00:09:27.399 --> 00:09:31.960 Ah. Yes, the bit wise not operator applied twice. 203 00:09:32.240 --> 00:09:34.080 Okay, but why do that if you want to drop 204 00:09:34.120 --> 00:09:36.440 the decimal from a random number like four point nine? 205 00:09:36.600 --> 00:09:39.799 Almost every developer on Earth just types math dot floor. 206 00:09:40.080 --> 00:09:42.960 Using a double bitwise knot. Seems like writing code in 207 00:09:43.039 --> 00:09:44.799 a secret language just to show off. 208 00:09:45.000 --> 00:09:47.440 It does look cryptic. I'll give you that, but it 209 00:09:47.519 --> 00:09:51.000 is actually a profound performance optimization at the hardware level. 210 00:09:51.080 --> 00:09:53.519 Wait, really, just dropping a decimal? 211 00:09:53.639 --> 00:09:57.039 Yeah? In JavaScript, all numbers are inherently sixty four bit 212 00:09:57.120 --> 00:10:00.240 floating point numbers. They consume memory and processing psych to 213 00:10:00.279 --> 00:10:02.960 handle decimals even when you only need a whole number. 214 00:10:03.200 --> 00:10:05.879 When you run math dot floor, the JavaScript engine has 215 00:10:05.879 --> 00:10:08.360 to spit up a function execution, check if the input 216 00:10:08.399 --> 00:10:11.320 is a valid number, handle edge cases, and then return 217 00:10:11.360 --> 00:10:11.840 the value. 218 00:10:11.919 --> 00:10:14.080 That sounds like a lot of overhead for rounding a number. 219 00:10:14.320 --> 00:10:17.519 It is, but the bitwise operator bypasses all of that 220 00:10:17.559 --> 00:10:22.000 function overhead completely. When the JavaScript engine sees a bitwise 221 00:10:22.039 --> 00:10:25.759 operator like a tilled, it temporarily forces that sixty four 222 00:10:25.759 --> 00:10:29.519 bit float into a thirty two bit signed integer. The 223 00:10:29.559 --> 00:10:33.279 decimal portion isn't mathematically rounded off, it is literally truncated 224 00:10:33.360 --> 00:10:34.840 and discarded at the bit level. 225 00:10:35.039 --> 00:10:35.559 Wow. 226 00:10:35.840 --> 00:10:38.039 The first tiled flips all the binary bits, and the 227 00:10:38.039 --> 00:10:40.720 second till flips them back to their original state. The 228 00:10:40.799 --> 00:10:43.679 result is the exact same as math dot floor, but 229 00:10:43.759 --> 00:10:47.000 it executes significantly faster because it operates directly on the 230 00:10:47.039 --> 00:10:51.080 CPU's registers. It bypasses the language's math library entirely. 231 00:10:51.480 --> 00:10:54.279 That is wildly efficient. It just chops off the decimal 232 00:10:54.279 --> 00:10:57.120 at the bit level and connecting that optimization back to 233 00:10:57.159 --> 00:11:00.759 our broader architecture. You know this API pattern and the closures, 234 00:11:00.759 --> 00:11:03.320 the getters and setters. This isn't an arbitrary choice by 235 00:11:03.360 --> 00:11:06.159 the authors. This is the genetic makeup of D three itself. 236 00:11:06.240 --> 00:11:09.039 Exactly when a developer calls a core function like D 237 00:11:09.120 --> 00:11:12.039 three do spg dot axis, they are interacting with this 238 00:11:12.200 --> 00:11:15.879 exact closure pattern. That consistency is why third party plugins 239 00:11:15.879 --> 00:11:19.080 in the D three ecosystem mesh together so flawlessly. 240 00:11:19.320 --> 00:11:23.600 Okay, So, moving from unstructured scripts to this intentional API 241 00:11:23.759 --> 00:11:27.120 architecture it really dictates a shift in how we verify 242 00:11:27.159 --> 00:11:30.720 our code. Because when you write a massive, monolithic script, 243 00:11:31.039 --> 00:11:33.320 the only way to test it is to load the browser, 244 00:11:33.840 --> 00:11:37.000 stare at the screen and ask does this chart look correct? 245 00:11:37.159 --> 00:11:40.840 Which is subjective, entirely manual, and honestly impossible to scale 246 00:11:40.879 --> 00:11:41.519 across a team. 247 00:11:41.720 --> 00:11:45.840 Right, But building these isolated modules unlocks the ability to 248 00:11:45.879 --> 00:11:50.120 do test driven development or TDD. Yes, the authors utilize 249 00:11:50.120 --> 00:11:53.480 a behavior driven framework called Jasmine to build a test suite, 250 00:11:53.919 --> 00:11:57.159 and they make a crucial point here. A comprehensive test 251 00:11:57.200 --> 00:12:00.000 suite is actually the best documentation a code based on 252 00:12:00.080 --> 00:12:00.320 can have. 253 00:12:00.639 --> 00:12:03.759 I completely agree with that. It physically verifies the contract 254 00:12:03.799 --> 00:12:06.480 of your public API. Yeah, a written read me file 255 00:12:06.519 --> 00:12:08.919 will inevitably fall out a date someone will forget to 256 00:12:08.960 --> 00:12:11.799 update it, but a test week it will fail the 257 00:12:11.840 --> 00:12:14.120 moment the code diverges from its promised behavior. 258 00:12:14.320 --> 00:12:17.559 But testing a visual medium like an SPG chart requires 259 00:12:17.559 --> 00:12:19.559 a shift in perspective. Right, I mean you cannot write 260 00:12:19.559 --> 00:12:22.919 a unit test to check if a visualization is esthetically pleasing. 261 00:12:23.120 --> 00:12:26.240 No, you're not testing for pretty. You are testing structural mechanisms. 262 00:12:26.519 --> 00:12:29.519 Right. You write an automated test to see if the 263 00:12:29.559 --> 00:12:33.399 module successfully appends SVG groups to the page. You write 264 00:12:33.440 --> 00:12:35.600 a test to assert that when you call the dot 265 00:12:35.639 --> 00:12:39.279 with five hundred setter, the internal variable actually updates to 266 00:12:39.320 --> 00:12:40.519 five hundred, And. 267 00:12:40.519 --> 00:12:44.000 Most crucially, you test the data binding. The DOM binding 268 00:12:44.080 --> 00:12:45.279 is the core mechanic of. 269 00:12:45.279 --> 00:12:48.679 D three, So you aren't just testing if a rectangle exists. 270 00:12:49.080 --> 00:12:53.440 No, you physically query the document object model, locate the rectangle, 271 00:12:53.440 --> 00:12:55.519 and inspect its hidden data property. 272 00:12:55.240 --> 00:12:58.360 The underscore underscore data field exactly. 273 00:12:58.440 --> 00:13:01.000 You check that feel to guarantee that the JSON or 274 00:13:01.039 --> 00:13:04.840 CSV row was successfully attached to that specific DOM node 275 00:13:05.120 --> 00:13:05.919 by the module. 276 00:13:06.000 --> 00:13:09.279 But visualizations are highly interactive. How do you test something 277 00:13:09.320 --> 00:13:11.799 like a user hovering their mouse over a specific bar 278 00:13:11.879 --> 00:13:15.720 to trigger a tooltip the text mentions using spies in Jasmine. 279 00:13:15.799 --> 00:13:18.200 But how does a test suite actually fake a mouse click? 280 00:13:18.360 --> 00:13:21.240 So a spy is basically a function proxy. If your 281 00:13:21.279 --> 00:13:23.960 module have a custom function that triggers a complex tooltip 282 00:13:23.960 --> 00:13:27.240 animation on hover, running that full animation during a unit 283 00:13:27.279 --> 00:13:29.200 test would slow down your test suite. 284 00:13:29.000 --> 00:13:32.080 It need a full browser rendering engine, right, exactly. 285 00:13:31.720 --> 00:13:35.759 So a Jasmine spy intercepts the call. It essentially replaces 286 00:13:35.799 --> 00:13:39.360 your tooltip function with a dummy wrapper. Okay, when the 287 00:13:39.399 --> 00:13:43.320 test simulates the hover event, the spy catches the execution 288 00:13:43.759 --> 00:13:47.159 records yes, I was invoked, and here are the coordinates 289 00:13:47.159 --> 00:13:50.159 I was passed, and then it stops the execution before 290 00:13:50.159 --> 00:13:51.720 the heavy animation code even runs. 291 00:13:51.879 --> 00:13:55.240 It intercepts the command. It's like, okay, it's like evaluating 292 00:13:55.240 --> 00:13:59.200 the assisiency of a restaurant kitchen. How so, Well, you 293 00:13:59.279 --> 00:14:01.600 don't verify the system by sitting in the dining room 294 00:14:01.639 --> 00:14:04.399 and tasting the final plated meal every five minutes. 295 00:14:04.320 --> 00:14:05.360 Right, That would take forever. 296 00:14:05.799 --> 00:14:09.279 You unit test the kitchen infrastructure. You check that the 297 00:14:09.320 --> 00:14:12.240 ticketing machine successfully routes a piece of paper to the 298 00:14:12.279 --> 00:14:15.000 grill station. You don't need to actually cook the steak 299 00:14:15.039 --> 00:14:16.720 to know the communication system worked. 300 00:14:17.080 --> 00:14:21.360 That clarifies the distinction perfectly. You are verifying the communication 301 00:14:21.440 --> 00:14:25.120 pathways and the state changes, not the final visual output. 302 00:14:25.679 --> 00:14:29.799 And once you have established that bulletproof testable infrastructure, you 303 00:14:29.879 --> 00:14:33.360 are finally equipped to handle a real world scenario where 304 00:14:33.399 --> 00:14:36.559 the data is actively working against you, which brings. 305 00:14:36.360 --> 00:14:38.440 Us to the core case study of the book, the 306 00:14:38.639 --> 00:14:39.840 Urban Data Challenge. 307 00:14:40.200 --> 00:14:41.480 This is where it gets fun right. 308 00:14:41.799 --> 00:14:45.320 The task was to visualize complex public transit networks for 309 00:14:45.440 --> 00:14:50.440 three major cities, Zurich, Geneva, and San Francisco. They had 310 00:14:50.440 --> 00:14:53.120 their testable building blocks, but now they had to feed 311 00:14:53.240 --> 00:14:54.759 real world data into them. 312 00:14:54.720 --> 00:14:58.000 And real world data is rarely clean. Public transit data 313 00:14:58.120 --> 00:15:02.279 especially is notoriously fragmed. The raw source files for this 314 00:15:02.360 --> 00:15:06.159 project amounted to roughly one hundred and eighty megabytes of 315 00:15:06.360 --> 00:15:08.000 dirty csvs one. 316 00:15:08.000 --> 00:15:11.919 Hundred eighty megabytes of text data for San Francisco alone. 317 00:15:11.960 --> 00:15:15.759 The scheduled bus timetables and the real time arrival coordinates 318 00:15:15.799 --> 00:15:19.320 were siloed in entirely different files that needed a relational join. 319 00:15:19.519 --> 00:15:22.159 Yeah, and the data set for Geneva was completely missing. 320 00:15:22.200 --> 00:15:24.759 Its latitude and longitude columns just gone. 321 00:15:24.639 --> 00:15:25.799 Which is kind of important for. 322 00:15:25.759 --> 00:15:28.039 Our map just a bit. Yeah, and if a developer 323 00:15:28.039 --> 00:15:30.360 attempted the load one hundred and eighty megabytes of raw 324 00:15:30.480 --> 00:15:34.480 unjoined text files directly into the browser's memory via AJAX. 325 00:15:34.080 --> 00:15:36.519 Request, oh, the application would freeze right immediately. 326 00:15:36.679 --> 00:15:39.519 The browser tab would likely crash. The client side environment 327 00:15:39.679 --> 00:15:42.039 just cannot handle that volume of raw computation. 328 00:15:42.399 --> 00:15:44.639 So the authors knew the browser couldn't handle it, so 329 00:15:44.639 --> 00:15:48.279 they stepped outside of D three entirely. They used data Mirror, 330 00:15:48.279 --> 00:15:53.440 which is an enterprise big data analytics tool, to ingest, clean, join, 331 00:15:53.600 --> 00:15:57.039 and aggregate the raw files. They managed to reduce the 332 00:15:57.039 --> 00:15:59.440 one hundred and eighty megabytes of noise down to a 333 00:15:59.519 --> 00:16:03.879 highly optimized sixty megabyte zipped payload, which is still a 334 00:16:03.879 --> 00:16:06.200 massive amount of data, but it crosses the threshold of 335 00:16:06.200 --> 00:16:08.600 being manageable for a modern browser exactly. 336 00:16:09.120 --> 00:16:11.960 But to orchestrate the intake of that sixty megabyte file, 337 00:16:12.159 --> 00:16:15.639 the author's engineered a dedicated data manager API. 338 00:16:15.679 --> 00:16:17.960 The Data Manager API, and this is a D three 339 00:16:18.000 --> 00:16:21.559 module built using the exact closure pattern we discussed, but 340 00:16:21.639 --> 00:16:25.480 it doesn't render a single pixel to the screen, none. 341 00:16:24.840 --> 00:16:28.759 Not one pixel. Its sole responsibility is to asecrenously fetch 342 00:16:28.759 --> 00:16:32.320 the compressed CSV files, route the data through specific cleaning 343 00:16:32.320 --> 00:16:35.440 functions depending on whether it's processing Zerich or Geneva, and 344 00:16:35.519 --> 00:16:37.279 manage the state of data load. 345 00:16:37.679 --> 00:16:39.919 I understand the need to load data, but why go 346 00:16:39.960 --> 00:16:42.879 to the trouble of building a complex data manager API 347 00:16:43.000 --> 00:16:44.879 at all? I mean, D three has a built in 348 00:16:44.960 --> 00:16:47.919 D three dot CSV function that loads files perfectly well 349 00:16:48.039 --> 00:16:51.240 it does. Why separate the loading logic from the chart code? 350 00:16:51.360 --> 00:16:54.440 It roots back to the model view controller architecture or MVC. 351 00:16:55.279 --> 00:16:58.200 If you embid a D three dot CSV call directly 352 00:16:58.200 --> 00:17:01.120 inside your bar chart module, the bar chart is permanently 353 00:17:01.120 --> 00:17:02.440 coupled to the network. 354 00:17:02.159 --> 00:17:04.160 OH, meaning it can't do anything else. 355 00:17:04.160 --> 00:17:06.359 Right, it cannot draw itself until it fetches a file. 356 00:17:07.119 --> 00:17:10.039 By separating the data manager, which is the model, from 357 00:17:10.079 --> 00:17:13.839 the visual modules, the views, the charts remain completely agnostic 358 00:17:13.839 --> 00:17:15.200 about where the data comes from. 359 00:17:15.480 --> 00:17:16.519 How do they communicate? 360 00:17:16.599 --> 00:17:16.759 Then? 361 00:17:17.039 --> 00:17:19.920 If the chart doesn't know about the data manager, how 362 00:17:19.960 --> 00:17:21.759 does it know when to actually draw the data? 363 00:17:22.079 --> 00:17:25.240 Ah. The text details their use of D three dot 364 00:17:25.319 --> 00:17:28.119 dispatch dispatch o YE. D three dot dispatch operates on 365 00:17:28.160 --> 00:17:32.440 a publisher subscriber mechanism. The decoupled chart modules load onto 366 00:17:32.480 --> 00:17:35.119 the page and they simply listen. They just listen for 367 00:17:35.160 --> 00:17:36.039 a specific signal. 368 00:17:36.119 --> 00:17:37.680 Meanwhile, the data manager is working in the. 369 00:17:37.640 --> 00:17:41.480 Background exactly crunching the sixty megabytes. When it finishes, the 370 00:17:41.559 --> 00:17:44.720 data manager uses D three Dot dispatch to broadcast a 371 00:17:44.759 --> 00:17:49.000 custom event. It's essentially shouting data ready into the application's 372 00:17:49.000 --> 00:17:52.119 global name space along with the cleaned array of transit stops. 373 00:17:52.359 --> 00:17:54.960 The charts hear the shout, they catch the array, and 374 00:17:55.000 --> 00:17:59.359 they render. The separation of concerns is absolute. It's beautiful, 375 00:17:59.400 --> 00:18:02.400 and once the data is finally loaded and clean, we 376 00:18:02.519 --> 00:18:05.880 arrive at the visual challenge drawing millions of individual transit 377 00:18:05.920 --> 00:18:06.599 stops right. 378 00:18:06.960 --> 00:18:11.079 They utilize D three's geographic capabilities for this, specifically the 379 00:18:11.160 --> 00:18:14.200 D three dot geo dot Mercator projection. 380 00:18:14.000 --> 00:18:14.880 The map API. 381 00:18:15.119 --> 00:18:19.519 Yes, this map API dynamically calculates the mathematical center, the 382 00:18:19.559 --> 00:18:22.359 optimal zoom scale, and the pixel translation for each of 383 00:18:22.400 --> 00:18:25.440 the three city layouts. It maps the raw latitude and 384 00:18:25.440 --> 00:18:29.000 longitude coordinates into x and y pixel positions, and it 385 00:18:29.119 --> 00:18:31.920 pends an SVG circle for every single transit stop. 386 00:18:32.240 --> 00:18:35.039 Rendering the map is a processing challenge for sure, but 387 00:18:35.200 --> 00:18:37.960 the true computational hurdle is interactivity. 388 00:18:38.119 --> 00:18:39.240 Always they needed the. 389 00:18:39.279 --> 00:18:42.039 User to be able to instantly filter millions of transit 390 00:18:42.079 --> 00:18:45.400 coordinates without triggering a massive rerender that would crash the 391 00:18:45.400 --> 00:18:48.160 frame rate. And to solve this, they implemented a library 392 00:18:48.200 --> 00:18:50.240 called crossfilter dot js, which. 393 00:18:50.359 --> 00:18:53.559 Not coincidentally was also developed by Mike Bostock, and it 394 00:18:53.640 --> 00:18:57.359 is specifically engineered to handle lightning fast multidimensional filtering in 395 00:18:57.359 --> 00:18:57.839 the browser. 396 00:18:58.039 --> 00:19:01.440 The underlying mechanic of cross filter in fol defining dimensions. 397 00:19:02.079 --> 00:19:04.799 You instruct cross filter to iterate through the massive data 398 00:19:04.799 --> 00:19:08.759 set once and build a highly optimized index based on 399 00:19:08.759 --> 00:19:10.119 a specific trait like. 400 00:19:10.079 --> 00:19:13.599 A location dimension, utilizing those transit coordinates. 401 00:19:13.160 --> 00:19:16.519