WEBVTT 1 00:00:00.160 --> 00:00:04.000 Is keeping your network secure feeling like, uh, just this 2 00:00:04.200 --> 00:00:07.120 constant battle against a flood of data. 3 00:00:07.000 --> 00:00:08.320 It really can be overwhelming. 4 00:00:08.480 --> 00:00:10.880 Yeah, and are you confident you've plugged every single hole 5 00:00:10.919 --> 00:00:14.160 against attacks? Is your sound system, you know, really seeing 6 00:00:14.160 --> 00:00:15.080 what it needs to see? 7 00:00:15.519 --> 00:00:18.559 Or maybe you're just starting out right and wondering where 8 00:00:18.640 --> 00:00:21.600 do I even begin with intrusion detection exactly? 9 00:00:21.760 --> 00:00:24.879 If any of that sounds familiar, well, you're definitely in 10 00:00:24.879 --> 00:00:27.519 the right place today. Our source material kicks off with 11 00:00:27.559 --> 00:00:29.399 those exact questions. 12 00:00:29.039 --> 00:00:32.479 Finding those critical signals, the important bits amidst all that 13 00:00:32.600 --> 00:00:33.439 network noise. 14 00:00:34.000 --> 00:00:35.560 That's the core challenge. 15 00:00:35.200 --> 00:00:37.799 Really absolutely, and that's what we do here on the 16 00:00:37.799 --> 00:00:41.280 Deep Dive. We take dense information like this fantastic guide 17 00:00:41.280 --> 00:00:43.359 we're looking at and try to pull out the key 18 00:00:43.359 --> 00:00:46.320 insights for you, you know, those aha moments. 19 00:00:46.439 --> 00:00:48.679 So our mission today is pretty focused. We want to 20 00:00:48.759 --> 00:00:52.320 understand how you can build a robust intrusion detection system 21 00:00:52.719 --> 00:00:56.439 and IDs and also a SIME environment, and. 22 00:00:56.320 --> 00:00:59.640 Crucially doing it with open source tools that are freely available. 23 00:00:59.799 --> 00:01:00.479 That's right. 24 00:01:00.640 --> 00:01:03.920 Our guide for this is Richard Medlins A Network Defender's 25 00:01:03.960 --> 00:01:09.280 Guide to Threat Detection using zeke Elastic Search log Stash, Cubana, 26 00:01:09.400 --> 00:01:10.519 tor and more. 27 00:01:10.920 --> 00:01:13.840 Quite a title promises a lot, and we're hoping to 28 00:01:13.840 --> 00:01:15.120 make that journey a bit smoother. 29 00:01:15.640 --> 00:01:18.319 What's interesting is a guide itself points out a big hurdle. 30 00:01:18.439 --> 00:01:21.840 You can find info on installing these tools one by one, sure, 31 00:01:22.400 --> 00:01:25.840 but getting them all configured to work together effectively clear 32 00:01:26.079 --> 00:01:28.400 up to date guides are surprisingly hard to come by. 33 00:01:28.760 --> 00:01:31.799 Okay, so let's unpack the toolkit we've got. Zeke used 34 00:01:31.840 --> 00:01:34.480 to be called BRO. Some might remember that name. The 35 00:01:34.560 --> 00:01:38.319 elastic search, Logstash, and Cubana. That's the ELK. 36 00:01:38.079 --> 00:01:39.920 Stack, the classic ELK stack. 37 00:01:40.000 --> 00:01:44.359 Yeah, finally, tour the Nanimity network. The source says getting 38 00:01:44.400 --> 00:01:46.280 these to play nice is tricky, so let's try and 39 00:01:46.319 --> 00:01:47.079 clear that path. 40 00:01:47.280 --> 00:01:50.319 Okay, let's start with the foundation Zeke. Our guide rightly 41 00:01:50.319 --> 00:01:52.400 calls it a cornerstone for a solid IDs. 42 00:01:52.480 --> 00:01:57.159 So Zeke formerly BRO maybe a nod to Orwell's nineteen 43 00:01:57.239 --> 00:02:00.760 eighty four seems kind of fitting for network monitoring. Maybe 44 00:02:00.799 --> 00:02:02.200 a bit ominous, huh? 45 00:02:02.280 --> 00:02:06.480 Maybe it's open source Network analysis software functions as a 46 00:02:06.519 --> 00:02:10.400 network intrusion detection system, and an IDs basically gives you 47 00:02:10.439 --> 00:02:12.439 a live view of network happenings. 48 00:02:12.520 --> 00:02:15.120 What makes it special though. It's not just basic pattern matching, 49 00:02:15.199 --> 00:02:15.400 is it. 50 00:02:15.759 --> 00:02:20.039 No, that's the key thing. It's smarter. It analyzes live traffic, yes, 51 00:02:20.240 --> 00:02:24.479 but also files moving across the network, even old recorded 52 00:02:24.520 --> 00:02:29.879 traffic and pccapy files, and it generates these neutral events 53 00:02:29.960 --> 00:02:32.479 based not just on known attack signatures, but also on 54 00:02:32.520 --> 00:02:34.199 network behavior that seems unusual. 55 00:02:34.400 --> 00:02:37.120 Right, So it's like noticing someone acting suspiciously near the building, 56 00:02:37.199 --> 00:02:38.759 not just spotting a crowbar. 57 00:02:38.960 --> 00:02:39.719 Exactly like that. 58 00:02:39.919 --> 00:02:44.919 Yeah, and it has built in analyzers for common protocols, HTTP, FTPS, 59 00:02:45.039 --> 00:02:47.960 mtp DNS, the usual suspects. 60 00:02:47.479 --> 00:02:48.599 And you can add more if needed. 61 00:02:48.719 --> 00:02:49.120 You can. 62 00:02:49.520 --> 00:02:52.360 Plus, it integrates with other tools like snort and crucially 63 00:02:52.400 --> 00:02:54.479 for US today, with elastic search. 64 00:02:54.759 --> 00:02:58.159 The guide mentioned some basic dependencies libcap for packet capture, 65 00:02:58.439 --> 00:03:01.520 open SSL for crypto, usual build tools like make c 66 00:03:01.599 --> 00:03:02.280 plus plus RAG. 67 00:03:02.400 --> 00:03:04.520 Yeah, the standard stuff for building from sores. But it 68 00:03:04.599 --> 00:03:08.680 quickly gets practical focusing on prepping Aubuntu for good packet capture. 69 00:03:09.120 --> 00:03:11.759 Right, you need to see the traffic clearly first, So 70 00:03:11.800 --> 00:03:16.199 step one, according to the guide, disable network manager. Why 71 00:03:16.280 --> 00:03:16.919 turn that off? 72 00:03:17.240 --> 00:03:20.919 Well, think of network manager as like the automatic pilot 73 00:03:20.960 --> 00:03:21.919 for your network connection. 74 00:03:22.400 --> 00:03:24.560 It tries to be helpful manage things. 75 00:03:24.360 --> 00:03:25.199 Which is usually good. 76 00:03:25.360 --> 00:03:29.080 Usually yeah, But when we're doing passive sniffing just listening in, 77 00:03:29.240 --> 00:03:32.840 it's active management can actually interfere, maybe drop packets or 78 00:03:32.919 --> 00:03:33.800 change how we see them. 79 00:03:34.000 --> 00:03:34.560 So for a. 80 00:03:34.479 --> 00:03:37.240 Dedicated sensor, you want it off. The guide gives the 81 00:03:37.280 --> 00:03:38.840 system trick to ural commands. 82 00:03:39.120 --> 00:03:43.400 Okay, that makes sense, less interference. Next step disabling NIC 83 00:03:43.680 --> 00:03:47.039 offloading functions sounds technical? What's the risk here? 84 00:03:47.240 --> 00:03:49.919 Modern network cards and icees they do tricks and hardware 85 00:03:49.960 --> 00:03:51.919 to speed things up right, take some load off the 86 00:03:51.919 --> 00:03:53.039 main CPU. 87 00:03:52.840 --> 00:03:56.159 Like handling checksums or segmenting large data packets. 88 00:03:55.800 --> 00:04:01.560 Exactly, things like TCP segmentation offloading TSO, Generic segmentation offloading GSO, 89 00:04:01.840 --> 00:04:06.000 Large receive offloading LRO all designed for performance, but they 90 00:04:06.039 --> 00:04:09.080 can bundle packets together or change packet timing in ways 91 00:04:09.080 --> 00:04:11.759 that distort the picture for our analysis tool. For ZEKE, 92 00:04:12.080 --> 00:04:15.759 you might even see packet sizes reported that are technically impossible. 93 00:04:16.160 --> 00:04:20.360 Ah, so we need the raw, unaltered stream. We need 94 00:04:20.399 --> 00:04:23.720 to turn those hardware shortcuts off on our analysis machine precisely. 95 00:04:24.079 --> 00:04:26.279 The guide shows using the f tool command with a 96 00:04:26.279 --> 00:04:30.439 bunch of flags rx tx, TSO, UPHO, GSO, grow l, row, 97 00:04:30.519 --> 00:04:32.399 et cetera, setting them all. 98 00:04:32.240 --> 00:04:35.360 To off, and the guide stresses you probably need to 99 00:04:35.399 --> 00:04:37.720 run this f tool command every time you start zeke. 100 00:04:37.879 --> 00:04:41.040 Yeah, that's a key point because those settings might not 101 00:04:41.199 --> 00:04:45.000 persist across reboots. You use f toolpinae to check their off. 102 00:04:45.319 --> 00:04:47.240 Some might say they can't be changed, but you verify 103 00:04:47.279 --> 00:04:48.279 the important ones are off. 104 00:04:48.319 --> 00:04:52.240 Okay, Then it mentions ensuring the DNS network service is 105 00:04:52.319 --> 00:04:55.639 running but not really changing its can fig Yeah. Then 106 00:04:55.839 --> 00:04:59.680 the big one, promiscuous mode sounds sneaky. 107 00:04:59.399 --> 00:05:03.079 Hu Well, Normally your network card only listens to traffic 108 00:05:03.160 --> 00:05:06.199 specifically addressed to it, like a polite receptionist only taking 109 00:05:06.199 --> 00:05:08.959 calls for their extension. Right, Promiscuous mode tells the card 110 00:05:09.040 --> 00:05:11.759 listen to everything, all traffic hitting the wire on that 111 00:05:11.759 --> 00:05:12.680 network segment. 112 00:05:12.439 --> 00:05:14.959 Doesn't matter who it's for, which is absolutely essential for 113 00:05:15.000 --> 00:05:17.319 an IDs. Otherwise you'd miss most of the conversations. 114 00:05:17.519 --> 00:05:20.720 Totally essential. If it's not promiscuous, it only sees traffic 115 00:05:20.759 --> 00:05:24.160 to the analysis box itself, useless for monitoring others. 116 00:05:24.279 --> 00:05:26.839 The guide shows if canfig interface promises to turn it 117 00:05:26.879 --> 00:05:29.959 on and ip A show interface rep I promised to 118 00:05:30.040 --> 00:05:31.680 check handy check YEP. 119 00:05:31.680 --> 00:05:34.839 Then it lists dependencies core ones we mentioned, but also 120 00:05:35.000 --> 00:05:37.519 optional stuff like GUOIP support. 121 00:05:37.240 --> 00:05:40.639 Ah geolocation so zeke can tell you where traffic is 122 00:05:40.680 --> 00:05:44.360 coming from or going to geographically based on IP address exactly. 123 00:05:44.800 --> 00:05:47.680 Using the limex mindb library and the free geolite two 124 00:05:47.759 --> 00:05:52.600 database super useful for spotting, say, unexpected international connections. Has 125 00:05:52.639 --> 00:05:55.360 that set up, The guide walks through app to get 126 00:05:55.360 --> 00:05:58.920 installed lib max mindyb dev, then using curl to download 127 00:05:58.920 --> 00:06:02.000 the geolite two city database, putting it in our share 128 00:06:02.040 --> 00:06:05.680 gip and it wisely says check for updated download links. 129 00:06:06.000 --> 00:06:10.680 Things change good tip Okay, next up. Pfriing sounds fast. 130 00:06:10.800 --> 00:06:11.040 It is. 131 00:06:11.079 --> 00:06:13.920 It's all about boosting packet capture speed, getting higher data 132 00:06:14.000 --> 00:06:15.600 rates than standard Linux methods. 133 00:06:15.720 --> 00:06:16.439 How does it work? 134 00:06:16.480 --> 00:06:21.560 Basically, it uses a Linux kernel mechanism called NAPI new Api. Essentially, 135 00:06:21.600 --> 00:06:25.000 it creates a more direct, efficient path for packets from 136 00:06:25.040 --> 00:06:28.160 the NIC into memory buffers that Z can access quickly 137 00:06:28.680 --> 00:06:31.319 reduces CPU overhead. Think of it like an express lane 138 00:06:31.319 --> 00:06:31.959 for packets. 139 00:06:32.000 --> 00:06:35.360 Gotcha, and installation looks like compiling from source again. 140 00:06:35.439 --> 00:06:38.120 Yep, get the code from GitHub, compile it with Make 141 00:06:38.399 --> 00:06:41.079 install the kunnel module the guide puts it in opt 142 00:06:41.079 --> 00:06:42.800 which is common for optional software. 143 00:06:42.839 --> 00:06:46.600 So the steps are installed. Build tools like bison flex right. 144 00:06:46.560 --> 00:06:50.800 Get, clone the pfring rebo cd into the kernel directory, 145 00:06:51.000 --> 00:06:54.240 Make Pseudo, make install, then Pseudo in smithmad to load 146 00:06:54.279 --> 00:06:57.839 the kernel module. It even covers building userspace libraries and 147 00:06:58.000 --> 00:07:00.319 TCP dump with pfr ing support. 148 00:07:00.399 --> 00:07:04.360 Okay, lots of prep work. But finally installing zke itself. 149 00:07:04.519 --> 00:07:05.399 Yes, clone the. 150 00:07:05.399 --> 00:07:08.399 Z cre go from GitHub, use the recursive flag that's important. 151 00:07:08.439 --> 00:07:11.160 Gets all the submodules right. Then the classic dot configure, 152 00:07:11.360 --> 00:07:14.399 make make install. The configure step is key. You tell 153 00:07:14.399 --> 00:07:17.360 it where PFI an G is, where guyp is. The 154 00:07:17.399 --> 00:07:18.879 guys suggests installing. 155 00:07:18.519 --> 00:07:21.480 To opsec compilation can take a while, I bet can do. 156 00:07:21.839 --> 00:07:24.399 Then a reboot is needed. Update your system's path so 157 00:07:24.439 --> 00:07:26.199 you can just type z commands easily. 158 00:07:26.639 --> 00:07:27.319 Almost there. 159 00:07:27.480 --> 00:07:31.560 Okay, installed. How do we tell zeke what to monitor? 160 00:07:32.079 --> 00:07:33.560 Basic config that's next. 161 00:07:33.839 --> 00:07:36.199 Two main files and ops can secut dot no dot 162 00:07:36.240 --> 00:07:38.120 cfg and networks dot cfg. 163 00:07:38.399 --> 00:07:40.000 No. Dot cfg is four. 164 00:07:40.319 --> 00:07:43.519 Defining your zke setup for a simple single sensor. The 165 00:07:43.560 --> 00:07:46.160 main thing is telling it which network interface to listen 166 00:07:46.199 --> 00:07:48.600 on the one we set to permiscuous mode earlier. 167 00:07:48.680 --> 00:07:50.120 And networks dot cfg. 168 00:07:50.279 --> 00:07:53.480 That's where you define your local IP address ranges, you know, 169 00:07:53.600 --> 00:07:56.439 using cid R notation like one ninety two point one 170 00:07:56.519 --> 00:07:58.959 sixty eight point one point zero two four or ten 171 00:07:59.000 --> 00:08:01.519 point zero point zero zero point zero eight. This helps 172 00:08:01.600 --> 00:08:04.199 zeke know what's internal versus external traffic. 173 00:08:04.319 --> 00:08:06.279 Makes sense. It also mentioned local dot Zeke. 174 00:08:06.439 --> 00:08:07.079 Yeah, that's like. 175 00:08:07.040 --> 00:08:10.079 The main control panel for enabling or disabling different Zke 176 00:08:10.120 --> 00:08:10.879 analysis scripts. 177 00:08:10.920 --> 00:08:11.839 You load the scripts you. 178 00:08:11.759 --> 00:08:14.920 Want to run there installed configured basics. Time to fire 179 00:08:14.959 --> 00:08:15.519 it up right. 180 00:08:15.759 --> 00:08:17.920 First, you need to give the zeke binari's permission to 181 00:08:17.920 --> 00:08:20.759 actually capture packets using the set cap command. It's a 182 00:08:20.759 --> 00:08:23.439 security thing, okay. Then you use zke toil the command 183 00:08:23.519 --> 00:08:26.879 line controller first time ever. You run zeke toil install. 184 00:08:27.079 --> 00:08:29.120 After that to start it with your config you use 185 00:08:29.199 --> 00:08:32.519 ziketil deploy and check if it's running zeke til status 186 00:08:33.039 --> 00:08:35.159 and you can watch the logs live like the connection 187 00:08:35.279 --> 00:08:39.120 log with tail opt sk logs, current con dot log. 188 00:08:39.600 --> 00:08:41.519 See the traffic rolling in phew. 189 00:08:41.679 --> 00:08:45.360 Okay, solid foundation with zeke running. Now the guide switches 190 00:08:45.399 --> 00:08:49.799 gears to tour. Why bring an anonymity tool into an 191 00:08:49.840 --> 00:08:52.519 intrusion detection discussion seems backwards. 192 00:08:52.639 --> 00:08:55.159 That's a great question, and the guide tackles it directly, 193 00:08:55.600 --> 00:08:58.399 understanding tour, how it works, what its traffic looks like. 194 00:08:59.120 --> 00:09:02.559 It's actually idle for defense how so, well, partly it's 195 00:09:02.600 --> 00:09:05.600 no your enemy right, understand the tool attackers might use, 196 00:09:05.799 --> 00:09:09.919 but more practically, seeing tour traffic inside a typical corporate network, 197 00:09:09.960 --> 00:09:11.440 that's usually a big red flag. 198 00:09:11.200 --> 00:09:14.360 Because employees generally shouldn't be hiding their activity using TOR 199 00:09:14.440 --> 00:09:15.840 for work stuff exactly. 200 00:09:15.879 --> 00:09:19.159 There's usually no legitimate business need, so detecting it could 201 00:09:19.200 --> 00:09:23.519 indicate policy violations or even malicious activity like data exfiltration 202 00:09:23.720 --> 00:09:24.960 or command and control channel. 203 00:09:25.080 --> 00:09:27.240 Okay, that makes sense, understand it to detect it. 204 00:09:27.639 --> 00:09:31.159 The guide explains onion routing briefly, how tour encrypts traffic 205 00:09:31.240 --> 00:09:33.919 and bounces it through multiple random relays to hide the 206 00:09:33.919 --> 00:09:35.639 source and destination, and. 207 00:09:35.639 --> 00:09:37.559 It warns about running an exit relay. 208 00:09:37.720 --> 00:09:41.080 Definitely, big legal disclaimer there. Exit relays are where the 209 00:09:41.120 --> 00:09:43.799 tour traffic pops out onto the regular internet, so you 210 00:09:43.799 --> 00:09:46.759 could be blamed for what others do through your exit node. 211 00:09:46.960 --> 00:09:50.320 Good warning. It also mentions law enforcement uses TOR too. 212 00:09:50.159 --> 00:09:53.600 Sometimes yeah, for investigations anonymous tip lines things like that, 213 00:09:53.679 --> 00:09:58.000 and critically it advises don't do anything illegal and don't 214 00:09:58.000 --> 00:10:02.440 log into personal accounts if you're using tour for anonymity. 215 00:10:02.480 --> 00:10:03.480 Common sense, but. 216 00:10:03.519 --> 00:10:07.080 Important, absolutely so setting up Tour on our analysis system. 217 00:10:07.240 --> 00:10:10.200 The guide covers installing Tour itself, plus a helper tool 218 00:10:10.240 --> 00:10:13.320 called privoxy sox it's a web proxy here. It's used 219 00:10:13.320 --> 00:10:16.080 as a middleman to easily forward web browser traffic into 220 00:10:16.120 --> 00:10:19.080 the tour network, which listens on a soocks port. 221 00:10:19.399 --> 00:10:20.320 How's install work? 222 00:10:20.440 --> 00:10:24.080 Add the Tour project's repository ear system, import their signing key, 223 00:10:24.440 --> 00:10:30.440 update packages, then APT install tor, guipdb, torsosdeb, dot Tour project, 224 00:10:30.480 --> 00:10:33.879 dot org, desh gearing pretty standard package. Install a provoxy 225 00:10:34.000 --> 00:10:36.320 just APT install privoxy simple enough. 226 00:10:36.360 --> 00:10:38.440 How do they connect? How do you tell provoxy to 227 00:10:38.519 --> 00:10:38.919 use Tor? 228 00:10:39.279 --> 00:10:42.480 You edit provoxies, canfig file, et cetera. Provoxes can fig 229 00:10:42.799 --> 00:10:45.639 add one line at the end, forward socks five local 230 00:10:45.639 --> 00:10:48.919 host point nine zero five zero zero that tells it 231 00:10:48.960 --> 00:10:53.440 to send traffic to tours default soocks port ninety fifty 232 00:10:53.759 --> 00:10:57.080 running on the local machine. The guide also suggests commenting 233 00:10:57.159 --> 00:10:59.960 out the log file line, then restart the provoxy. 234 00:11:00.679 --> 00:11:03.639 Okay, installed, configured, How do you actually use it? The 235 00:11:03.679 --> 00:11:04.919 guide mentioned torsocks. 236 00:11:05.159 --> 00:11:06.240 Yeah, torsocks is neat. 237 00:11:06.279 --> 00:11:08.799 You just run torsocks command and it tries to force 238 00:11:08.840 --> 00:11:12.159 that command's network traffic through tour. The guide shows an 239 00:11:12.200 --> 00:11:12.720 example with. 240 00:11:12.720 --> 00:11:15.360 Curl, but not everything works with torsocks. 241 00:11:14.960 --> 00:11:18.240 Right, especially graphical things like web browsers. For those, you 242 00:11:18.279 --> 00:11:20.320 often need to go into the browser's network settings and 243 00:11:20.399 --> 00:11:23.799 manually configure the SCKS five proxy point it to local 244 00:11:23.799 --> 00:11:25.240 host port ninety fifty. 245 00:11:25.360 --> 00:11:28.200 The guide then shows a script tour switch to toggle 246 00:11:28.200 --> 00:11:31.200 tour on and off easily. That sounds useful, super useful. 247 00:11:31.320 --> 00:11:33.679 It uses x settings to change the system wide proxy 248 00:11:33.679 --> 00:11:37.200 settings for Genome desktops and system cordal to start stop 249 00:11:37.240 --> 00:11:38.720 the tour in provoxy services. 250 00:11:38.799 --> 00:11:39.919 Saves a lot of typing, and. 251 00:11:39.919 --> 00:11:42.399 It provides the script code how to make it executable 252 00:11:42.480 --> 00:11:46.399 with chew mode. Even handles potential ownership issues YEP, and it. 253 00:11:46.360 --> 00:11:49.879 Covers letting a regular user run this script without needing 254 00:11:49.919 --> 00:11:51.480 the pseudo password every time. 255 00:11:51.879 --> 00:11:55.159 How editing the sudoer's file gotta be careful. 256 00:11:55.159 --> 00:11:58.639 They're very careful use with pseudo always. It checks syntax 257 00:11:58.679 --> 00:12:01.919 before saving. The guy gives example lines to add allowing 258 00:12:01.960 --> 00:12:06.519 specific commands system sort, telecorner, stop, status, et cetera for 259 00:12:06.600 --> 00:12:08.720 a user without a password. 260 00:12:08.399 --> 00:12:11.039 Prompt Finally installing the Tour browser itself. 261 00:12:11.120 --> 00:12:14.799 Yeah, that's the prepackaged browser designed for Tour. Simple install 262 00:12:15.240 --> 00:12:18.759 apt get install Tour Browser launcher. Then you run Tour 263 00:12:18.799 --> 00:12:21.159 Browser Launcher as a normal user, not root. 264 00:12:21.360 --> 00:12:23.360 So now we can generate Tour traffic on our. 265 00:12:23.240 --> 00:12:27.039 Network exactly, which means we can test the Zeke signatures 266 00:12:27.080 --> 00:12:28.960 designed to detect it, which we'll get too later. 267 00:12:29.120 --> 00:12:30.559 Helps you understand it and spot it. 268 00:12:30.639 --> 00:12:34.120 Okay, Zeke is capturing We understand Tour now making sense 269 00:12:34.120 --> 00:12:37.120 of all that data. Enter the ELK stack, Elastic Search, 270 00:12:37.240 --> 00:12:38.960 log Stash, Cabana. 271 00:12:38.720 --> 00:12:42.440 Right, the powerhouse trio for log management and analysis, open source, 272 00:12:42.639 --> 00:12:43.960 very flexible, scalable. 273 00:12:44.080 --> 00:12:46.399 Let's break them down. Elastic Search is the search engine 274 00:12:46.399 --> 00:12:47.759 the database sort of. 275 00:12:47.840 --> 00:12:51.440 Yeah, It's a search and analytics engine based on Lucine, 276 00:12:51.840 --> 00:12:55.679 highly scalable, distributed. Its stores and indexes all the data 277 00:12:55.720 --> 00:12:57.559 so you can search it incredibly fast. 278 00:12:57.919 --> 00:12:58.759 It's the heart of. 279 00:12:58.799 --> 00:13:03.879 ELK and Stash the data processor exactly. Think of log 280 00:13:03.919 --> 00:13:06.559 Stash as the data pipeline or like the prep station. 281 00:13:07.039 --> 00:13:10.519 It ingests data from many sources filebeat for instance, then 282 00:13:10.559 --> 00:13:11.720 it can parse it clean. 283 00:13:11.480 --> 00:13:12.200 It up, enrich it. 284 00:13:12.600 --> 00:13:15.080 Enrich it how like adding GOIP info based on an 285 00:13:15.080 --> 00:13:19.559 IP address or translating cryptic log codes into human readable text. 286 00:13:19.960 --> 00:13:22.960 It uses things called grock patterns to understand structured or 287 00:13:23.000 --> 00:13:26.960 even unstructured logs. Then it sends the processed data to 288 00:13:27.120 --> 00:13:27.879 elastic Search. 289 00:13:27.960 --> 00:13:30.799 And Cabana is the shiny front end the dashboard. 290 00:13:30.879 --> 00:13:31.240 That's it. 291 00:13:31.440 --> 00:13:36.440 Cabana talks to elastic Search and lets you build visualizations charts, graphs, maps, tables, 292 00:13:36.799 --> 00:13:39.120 lets you explore the data interactively. It's how you see 293 00:13:39.120 --> 00:13:40.279 the patterns and anomalies. 294 00:13:40.480 --> 00:13:42.919 The guide also brings in Filebeat. What's its role? 295 00:13:43.200 --> 00:13:45.960 Filebeat is a lightweight agent, a shipper. You install it 296 00:13:46.000 --> 00:13:48.360 on servers where logs are generated. Its job is just 297 00:13:48.399 --> 00:13:50.840 to collect those logs and send them efficiently, either straight 298 00:13:50.879 --> 00:13:53.679 to elastic search or to log stash for that extra processing. 299 00:13:53.960 --> 00:13:57.240 Why use filebeat instead of just having logstash collect directly. 300 00:13:57.399 --> 00:14:01.000 Filebeat is much lighter, uses fewer resources on the source machine. 301 00:14:01.120 --> 00:14:03.960 It also has pre built modules for common log types 302 00:14:04.080 --> 00:14:07.639 like zeke logs, which simplifies setup. And it has this 303 00:14:08.080 --> 00:14:11.279 back pressure sensitive thing, meaning it won't flood log Stash 304 00:14:11.360 --> 00:14:13.879 or elastic Search if they get temporarily overloaded. 305 00:14:14.000 --> 00:14:17.519 Okay, so the flow is typically zeke writes logs, Filebeat 306 00:14:17.519 --> 00:14:21.519 reads logs sends to either log stash to elastic Search 307 00:14:21.799 --> 00:14:25.799 or elastic Search directly, then Kibana reads from elastic Search 308 00:14:25.840 --> 00:14:26.559 for visualization. 309 00:14:26.759 --> 00:14:29.639 That's the common pattern. The guide then walks through installing 310 00:14:29.679 --> 00:14:33.480 ELK plus filebeat, big warning, use the exact same version 311 00:14:33.519 --> 00:14:35.600 for all of them, avoids headaches. 312 00:14:35.360 --> 00:14:38.639 Right for elastic Search, add the elastic key in repo 313 00:14:39.240 --> 00:14:43.360 app to install elastic Search, then enable journal qualcogging. Why 314 00:14:43.399 --> 00:14:43.960 is that needed? 315 00:14:44.159 --> 00:14:44.840 By default? 316 00:14:44.879 --> 00:14:47.559 Elastic Search is a bit quiet in the main system logs. 317 00:14:47.600 --> 00:14:50.720 The guide shows modifying its service file elastic Search stop 318 00:14:50.759 --> 00:14:54.320 service to remove the quiet flag. Makes troubleshooting way easier 319 00:14:54.399 --> 00:14:56.919 using journal calls. Then you check it's running with corol 320 00:14:56.960 --> 00:14:59.240 local hosts point nine to two zero zero, and we. 321 00:14:59.200 --> 00:15:01.159 Need Java, open jdk. 322 00:15:01.159 --> 00:15:04.080 YEP, elastic Search and logs Dash run on Java. The 323 00:15:04.120 --> 00:15:07.200 guide recommends open jdk eight for the versions it discusses. 324 00:15:07.600 --> 00:15:09.679 Simple app to install open Jdk eight. 325 00:15:09.759 --> 00:15:13.840 Jdk log Stash install is similar, but it doesn't start automatically. 326 00:15:13.320 --> 00:15:15.919 Correct install the package, but you need to explicitly start 327 00:15:15.960 --> 00:15:18.399 the service. The guide shows the commands and where the 328 00:15:18.440 --> 00:15:19.559 config files live. 329 00:15:19.840 --> 00:15:24.440 Kubana install dot enable and start the service. Configure Kobana 330 00:15:24.519 --> 00:15:26.600 dot AML. What are the key settings there? 331 00:15:26.679 --> 00:15:29.279 Server dot port usually five six oh one, serve it 332 00:15:29.279 --> 00:15:33.120 dof host often local host or zero point zero zero 333 00:15:33.600 --> 00:15:37.519 zero to listen on all interfaces, and crucially elastic search 334 00:15:37.519 --> 00:15:40.240 dot hosts, which tells Kubana where to find elastic Search 335 00:15:40.480 --> 00:15:42.639 usually http dot local host, dot nine two Z or 336 00:15:42.639 --> 00:15:43.240 A zero and. 337 00:15:43.200 --> 00:15:46.480 Filebeat install enable it to start on boot okay stack 338 00:15:46.519 --> 00:15:48.399 installed now making it work with Zeke. 339 00:15:48.639 --> 00:15:51.480 The guide shows two main ways. First the simpler one 340 00:15:51.559 --> 00:15:54.240 filebeat send z clogs directly to elastic search. 341 00:15:54.360 --> 00:15:55.200 How is that configured? 342 00:15:55.360 --> 00:15:59.200 You edit filebeat dot AML, tell it where z clogs are, opseclogs, 343 00:15:59.240 --> 00:16:02.720 current dot log usually enable to built in filebeat z module. 344 00:16:02.960 --> 00:16:05.480 Then configure the output dot elastic search section with your 345 00:16:05.519 --> 00:16:06.960 elastic search address. 346 00:16:06.759 --> 00:16:09.799 And what changes on the zke side. For this direct method. 347 00:16:09.679 --> 00:16:11.799 You need to tell Zeke to output logs in JSON 348 00:16:11.879 --> 00:16:15.960 format and importantly use eph time stamps that's the number 349 00:16:15.960 --> 00:16:19.720 of seconds since nineteen seventy. You modify aii dot z 350 00:16:20.039 --> 00:16:22.600 and local dot z to set this up. The Kabana 351 00:16:22.720 --> 00:16:25.000 zke module expects that format. 352 00:16:24.799 --> 00:16:27.639 Then restart everything, go into gabona, create a filebeat index 353 00:16:27.679 --> 00:16:28.240 pattern and. 354 00:16:28.200 --> 00:16:31.440 Boom, you should see the pre built zke overview dashboard 355 00:16:31.519 --> 00:16:34.240 populated with your data instant visibility. 356 00:16:34.399 --> 00:16:37.799 Cool. But there's a second, more powerful method using logsdash. 357 00:16:37.919 --> 00:16:41.080 Yes, filebeat tas logsdash elastic search. This lets you do 358 00:16:41.200 --> 00:16:43.240 more sophisticated processing in logsdash. 359 00:16:43.279 --> 00:16:45.759 What changes for zke in this case timestams right. 360 00:16:45.879 --> 00:16:46.840 For the logstash route. 361 00:16:46.840 --> 00:16:48.639 You can figure a zeke jets an output again, but 362 00:16:48.679 --> 00:16:51.039 this time use ISOSO eight six to oh one time stamps. 363 00:16:51.080 --> 00:16:53.679 The more standard yy y y m M d d 364 00:16:53.840 --> 00:16:56.600 t h m s SC's format set that in. 365 00:16:56.600 --> 00:16:59.159 Local dot z and logstash needs configuring to receive from 366 00:16:59.200 --> 00:16:59.960 filebeats exactly. 367 00:17:00.120 --> 00:17:02.840 You edit pipelines dot mL to define the pipeline maybe 368 00:17:02.840 --> 00:17:04.160 tweet logstash dot mL. 369 00:17:04.240 --> 00:17:06.359 The core is a new config file, say zke dot com. 370 00:17:06.400 --> 00:17:08.160 Well goes in zeke dot com. 371 00:17:07.880 --> 00:17:10.599 An input section using the beats plugin listening on a 372 00:17:10.640 --> 00:17:15.240 port like five thousand and one. Then filter plugins, remove headers, 373 00:17:15.440 --> 00:17:19.240 parse json, use the date filter for the isotimestamp. Crucially, 374 00:17:19.240 --> 00:17:23.359 add the geop filter for location data. Maybe translate constate codes. 375 00:17:23.440 --> 00:17:25.839 You take field types, rename fields. 376 00:17:25.599 --> 00:17:26.160 Lots of power. 377 00:17:26.200 --> 00:17:30.039 There finally an output section pointing to elastic search. 378 00:17:29.920 --> 00:17:32.880 So filebeat needs to send to logstash port five thousand 379 00:17:32.880 --> 00:17:34.400 and one instead of elastic Search. 380 00:17:34.519 --> 00:17:37.559 Correct change the output section and filebeat dot aml from 381 00:17:37.559 --> 00:17:42.119 elastic search to logstash, specifying hosts Localhost dot five zero 382 00:17:42.240 --> 00:17:44.880 zero one, test the config, restart everything. 383 00:17:44.960 --> 00:17:47.680 Then in cabana, create a logfash index pattern yep. 384 00:17:48.039 --> 00:17:51.440 And now, because logstash added the goip data, you can 385 00:17:51.440 --> 00:17:55.400 build visualizations like coordinate maps showing traffic origins. The guide 386 00:17:55.480 --> 00:17:59.000 also adds a good troubleshooting tip. If data isn't showing upright, 387 00:17:59.240 --> 00:18:02.160 sometimes delete it and recreating the index pattern in kipana 388 00:18:02.240 --> 00:18:02.720 fixes it. 389 00:18:02.799 --> 00:18:06.279 Okay, Zeke is logging, elk is processing and visualizing, but 390 00:18:06.400 --> 00:18:09.039 Zeke isn't really alerting yet. Is it just logging connections 391 00:18:09.039 --> 00:18:12.119 each TTP, et cetera. We need signatures for active. 392 00:18:11.839 --> 00:18:15.839 Detection, exactly signatures. Tell Zeke hey, if you see the 393 00:18:15.880 --> 00:18:20.039 specific pattern, generate a notice. It moves Zeke from passive 394 00:18:20.039 --> 00:18:22.160 observation to active alerting. 395 00:18:22.319 --> 00:18:24.960 The guide points to installing signatures from GitHub. 396 00:18:25.079 --> 00:18:28.279 Yes, there's a repository linked you get clone it, then 397 00:18:28.519 --> 00:18:32.279 run and install dot Sher script provides this copies the 398 00:18:32.319 --> 00:18:35.279 signature files into the right place in your Zeke installation 399 00:18:36.000 --> 00:18:37.480 op zeek share Zeke site. 400 00:18:37.559 --> 00:18:40.559 Usually important warning from the guide here back up your 401 00:18:40.599 --> 00:18:42.000