By Taisa

If you’d like to solve the Web Access Log challenge in a way that falls somewhere between CryptoKait’s Excel method and Paul Buonopane’s Python method, command-line tools are that happy middle ground between the ease of a GUI and the power of scripting.

In this post, we’ll untangle the Web Access Log Log Analysis challenge using only command-line tools (plus a little OSINT). This post is the third in a series. Unlike Part 2, however, starting with Question #2 you’ll be challenged to use suggested tools to arrive at answers on your own! (But hints and solutions are available if you need them!)

---

Prerequisites

• Command-line Log Analysis FOR THE WIN (1/3): How to Approach a Wild Log – Explains the methodologies and tools used in this walkthrough.

• Command-line Log Analysis FOR THE WIN (2/3): Walking through “Leaping” – Offers detailed demonstrations of the cat, grep, cut, sort, uniq, tr, paste, bc, wc, and head utilities, along with a little regex.

• Web Access Log – A sample challenge representative of an Easy Log Analysis challenge within the National Cyber League (NCL) Games.

---

Related Posts

• Summer Camp 2020 – Log Analysis – Paul Buonopane’s write-up for the Web Access Log challenge, using Python scripting as an advanced solution.

• Log Analysis Tips for People Who Aren’t Ready for Command Line – CryptoKait’s tutorial on the other end of the Log Analysis tool spectrum, demonstrating how to tackle a log using Excel.

---

Always Make a Table

We constructed this table in Part 1, but it’s replicated here for convenience. It contains information about web access log formats that we uncovered via OSINT in this manner:

For the Web Access Log challenge, the scenario tells us this is an Nginx log. What happens when you Google web access log format Nginx? Is that information a little too technical? What if you make the search less refined, and just Google web access log format without “Nginx“?

Command-line Log Analysis FOR THE WIN (1/3): How to Approach a Wild Log

---

Log Format Table for “Web Access Log”

Sample Log Entry:
35.187.132.245 - - [02/May/2020:00:05:32 +0000] "GET / HTTP/1.1" 200 18 "http://54.236.11.149" "Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)"

Field Description	Example
IP of client / remote host	35.187.132.245
identd (identity of client)	-
userid (remote user)	-
Timestamp in format [day/month/year:hour:minute:second zone]	[02/May/2020:00:05:32 +0000]
Request line in format “METHOD RESOURCE PROTOCOL”	"GET / HTTP/1.1"
Status code	200
Size in bytes of the object returned to the client	18
Referer	"http://54.236.11.149"
User-agent	"Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)"

---

Web Access Log #1

Tools We’ll Use

Utility	Switch	Description
cat		outputs file content, e.g. cat leaping.log
|		“pipe” – takes the output of the command on the left and uses it as input for the command on the right
wc		“word count” – counts the number of lines, words, and characters
	-l	displays the line count only

Approach

We know from our OSINT research into this log format that every entry in this type of log represents a request made to the web server. In other words, this question is asking, “How many entries are in this log?“

STEP 1 – CHECK FOR SURPRISES

Usually, the number of log entries will be equal to the total number of new lines in the file.

That’s not always the case, though!

It’s worth skimming through the log visually to ensure there is no header at the top and that log entries appear as a single line. In some logs, you’ll find that a single log entry is comprised of multiple new lines.

Do you see anything in this log that might affect the way entries are counted?

In this challenge, as with most of the Easy challenges, our log follows the standard one-entry-per-line-and-no-surprises format.

If that hadn’t been the case, grep‘s -A and -B switches would have been invaluable for parsing log entries spanning multiple new lines.

STEP 2 – COUNT THE NUMBER OF NEW LINES

If this challenge had been presented to us on the Cyber Skyline platform, the embedded text viewer would show us line numbers. The line number at the bottom would reveal how many new lines are in the file.

Absent that option, we have the wc (“word count”) utility, which comes with a -l (lowercase “L”) switch for counting lines. Its use looks like this:

cat access.log | wc -l

To learn more about the wc utility and its options, run this command:

wc --help

---

Web Access Log #2

Recommended Tools

Utility	Switch	Description
cat		outputs file content, e.g. cat leaping.log
|		“pipe” – takes the output of the command on the left and uses it as input for the command on the right
cut		extracts fields (columns that you specify) from lines
	-d	specifies the use of a delimiter other than the default tab; it can only be one character
	-f	denotes which field(s) to display, indicated by their numerical order of appearance
sort		alphabetizes input, e.g. a, b, c, 1, 101, 2
uniq		deduplicates – detects repeated lines if they are adjacent and removes the duplicates (deduplication)
grep		conducts a case-sensitive search
	-v	excludes lines that match the search term
wc		“word count” – counts the number of lines, words, and characters
	-l	displays the line count only

Approach

Aren’t you glad we did that OSINT?? 😀 If we didn’t already know what status codes generally look like, our OSINT revealed exactly which field to look in! Refer back to the Log Format Table to find the status code field.

This question is similar to one we already tackled about unique IP addresses in Part 2. Our approach to this question will be almost the same:

STEP 1 – ISOLATE THE STATUS CODES – using cut

STEP 2 – SORT THE STATUS CODES – using sort

STEP 3 – REMOVE DUPLICATE STATUS CODES – using uniq

BUT DON’T COUNT THE RESULTS YET!

After cutting, sorting, and removing duplicates, can you see why using wc -l to count the number of lines would have caused an incorrect result?

Highlight this box to see the command that cuts, sorts, and removes the duplicate status codes, then try it to see what problem it presents:

cat access.log | cut -d " " -f 9 | sort | uniq

When you run that command, "-" appears as a placeholder in the status code field when there is no status code! But the placeholder is not itself a unique status code.

There are few enough results that we could count them by hand, or just minus one from the total count returned by wc -l, but that won’t help us learn our tools!

Here are two ways to clean up the results using command-line tools that are already familiar to us from Part 2—grep, grep -v, and character classes. Highlight the boxes to see the full commands:

cat access.log | cut -d " " -f 9 | sort | uniq | grep [[:digit:]]

cat access.log | cut -d " " -f 9 | sort | uniq | grep -v "-"

STEP 4 – COUNT THE STATUS CODES

With the results cleaned up, what utility would you append to count those results for you? Hint: We just demonstrated it in Web Access Log #1!

Highlight the box below to check your solution:

cat access.log | cut -d " " -f 9 | sort | uniq | grep [[:digit:]] | wc -l

---

Web Access Log #3

Recommended Tools

Utility	Switch	Description
cat		outputs file content, e.g. cat leaping.log
|		“pipe” – takes the output of the command on the left and uses it as input for the command on the right
cut		extracts fields (columns that you specify) from lines
	-d	specifies the use of a delimiter other than the default tab; it can only be one character
	-f	denotes which field(s) to display, indicated by their numerical order of appearance
sort		alphabetizes input, e.g. a, b, c, 1, 101, 2
	-n	sorts input numerically instead of alphabetically (so instead of 1, 101, 2 you get 1, 2, 101)
	-r	sorts in reverse order (handy for showing largest numbers first)
head		displays the first 10 lines of input; use head -[number] to change the default number

Approach

STEP 1 – ISOLATE THE BYTES

According to our OSINT, what field were bytes located in? Hint: Refer back to the Log Format Table.

Can you use the cut utility to isolate that field? Using a space as the delimiter, what’s the numerical position of the field?

1              2 3 4                     5      6    7 8         9   
35.187.132.245 - - [02/May/2020:00:05:32 +0000] "GET / HTTP/1.1" 200 

10 11                     12           13        14 15   16   17      
18 "http://54.236.11.149" "Mozilla/5.0 (Windows; U; MSIE 9.0; Windows 

18 19   20     21                22                                  
NT 9.0; en-US) AppEngine-Google; (+http://code.google.com/appengine; 

23     24
appid: s~virustotalcloud)"

STEP 2 – SORT NUMERICALLY WITH LARGEST VALUES FIRST

Remember the switches for sort that will show the numerically largest amount?

Highlight this box to check your solution:

cat access.log | cut -d " " -f 10 | sort -nr | head -1

---

Web Access Log #4

Recommended Tools

Utility	Switch	Description
cat		outputs file content, e.g. cat leaping.log
|		“pipe” – takes the output of the command on the left and uses it as input for the command on the right
cut		extracts fields (columns that you specify) from lines
	-d	specifies the use of a delimiter other than the default tab; it can only be one character
	-f	denotes which field(s) to display, indicated by their numerical order of appearance
grep		conducts a case-sensitive search
	-E	adds support for a more extensive regular expression language (regex); we’ll use it to unlock match repetition (“{ }“)
	-v	excludes lines that match the search term
sort		alphabetizes input, e.g. a, b, c, 1, 101, 2
uniq		deduplicates – detects repeated lines if they are adjacent and removes the duplicates (deduplication)
	-c	prefixes each line with the number of adjacent occurrences, so you know how many were found in that position before the duplicates were removed

Approach

STEP 1 – OSINT

What does an HTTP tunneling attempt look like? Try Googling:
web access log “HTTP tunneling”

A helpful Wikipedia page quickly points us to a certain method. If we’re not sure what a “method” is, that same Wikipedia page has a handy hyperlink which explains it. An HTTP request method is indicated by a word describing an action performed on a resource, like GET or POST—but those aren’t the only possible methods!

What methods appear in this log? Here’s a command to help you see what shows up in the field associated with methods:

cat access.log | cut -d " " -f 6 | sort | uniq

STEP 2 – ISOLATE AND COUNT THE METHODS

Once you know which method indicates an HTTP tunneling attempt, what switch can you use with uniq to count the number of occurrences?

Highlight this box to check your solution:

cat access.log | cut -d " " -f 6 | sort | uniq -c

As a training exercise, what are some ways you could use grep -v to clean up your results? Are there certain characters, character classes, or patterns you could exclude?

Highlight these boxes for some ideas:

cat access.log | cut -d " " -f 6 | grep -v "\x" | sort | uniq -c

cat access.log | cut -d " " -f 6 | grep -v [[:digit:]] | sort | uniq -c

cat access.log | cut -d " " -f 6 | grep -Ev ".{10,}" | sort | uniq -c

---

Web Access Log #5

Recommended Tools

Utility	Switch	Description
cat		outputs file content, e.g. cat leaping.log
|		“pipe” – takes the output of the command on the left and uses it as input for the command on the right
\|		the “or” operator, being escaped by a backslash so that the system doesn’t try to interpret it as pipe
grep		conducts a case-sensitive search
	-v	excludes lines that match the search term
cut		extracts fields (columns that you specify) from lines
	-d	specifies the use of a delimiter other than the default tab; it can only be one character
	-f	denotes which field(s) to display, indicated by their numerical order of appearance
wc		“word count” – counts the number of lines, words, and characters
	-l	displays the line count only

Approach

I’ll be the first to confess that this question is confusing to a neophyte such as myself. Doesn’t binary data look like strings of 1s and 0s? There are no such strings in this log. This is where NCL comes in and teaches me things!

STEP 1 – OSINT AND INTUITION

Just by the process of eliminating what we know to be good baseline data—because it matches the log format we OSINTed (Is that a verb? It is now!)—we can tell which requests seem valid vs. invalid.

For example, compare these two log entries and see if you can distinguish, just by guessing, the valid request from the invalid request:

44.224.22.196 - - [02/May/2020:00:56:41 +0000] "GET http://example.com/ HTTP/1.1" 200 18 "-" "AWS Security Scanner"

44.224.22.196 - - [02/May/2020:00:56:44 +0000] "\x16\x03\x01\x00\xD2\x01\x00\x00\xCE\x03\x03\xE6\xE7\xA4\x1B(AU\x87Ij\x8D\x10\xD0\xF3M<\xB2G:\xC2<x\xC3\xB2\xD2\xB0b\xC8\x14P\xDE\xB2\x00\x00b\xC00\xC0,\xC0/\xC0+\x00\x9F\x00\x9E\xC02\xC0.\xC01\xC0-\x00\xA5\x00\xA1\x00\xA4\x00\xA0\xC0(\xC0$\xC0\x14\xC0" 400 163 "-" "-"

The question suggests another distinction might exist, though, between:

• invalid request lines which do contain raw binary data and
• invalid request lines which do not contain raw binary data.

During a competition, time is of the essence, and I’m not finding a magic Google search that will help me make quick sense of these escape sequences. Do we dare to assume, then, that all these escape sequences represent “raw binary data“?

This challenge has been categorized as an Easy one. On Easy challenges, I dare make such assumptions, and it hasn’t bit me in the arse just yet!

We mustn’t get complacent, though. When a question in NCL implies a distinction, it’s generally for a reason—some distinction does exist. We’ll learn more as we proceed.

STEP 2 – ISOLATE THE LINES CONTAINING RAW BINARY DATA

For now, our job is to isolate those funny-looking lines! This log is small enough that we could do it by hand, but that wouldn’t prepare us for harder challenges later.

What are some elements we can filter by?

Option A: Filter out What We Don’t Want
We know from Web Access Log #4 that the invalid requests with raw binary data don’t contain any of the standard HTTP methods. Try using grep -v to eliminate entries that contain the CONNECT, HEAD, GET, or POST methods, to see what’s left behind.

Highlight the box below to check your solution:

cat access.log | grep -v "CONNECT\|HEAD\|GET\|POST"

Append a | wc -l to the end of that to count the number of invalid request lines. How many are there?

Option B: Search for What We Do Want
If you looked up the status codes that we saw in Web Access Log #2, you know that “400” is the status code for a bad (invalid) request. Looking at a sample line containing escape sequences, you see that its status code is 400:

44.224.22.196 - - [02/May/2020:00:56:44 +0000] "\x16\x03\x01\x00\xD2\x01\x00\x00\xCE\x03\x03\xE6\xE7\xA4\x1B(AU\x87Ij\x8D\x10\xD0\xF3M<\xB2G:\xC2<x\xC3\xB2\xD2\xB0b\xC8\x14P\xDE\xB2\x00\x00b\xC00\xC0,\xC0/\xC0+\x00\x9F\x00\x9E\xC02\xC0.\xC01\xC0-\x00\xA5\x00\xA1\x00\xA4\x00\xA0\xC0(\xC0$\xC0\x14\xC0" 400 163 "-" "-"

Does every line containing status code 400 also contain this weird escaped data? Use this command to check:

cat access.log | grep " 400 "

The answer is no! Now we see where the distinction is drawn:

• Invalid request lines all have a status code of 400
• All lines containing raw binary data have a status code of 400
• Not all invalid request lines contain raw binary data, so there will be lines with status code 400 that do not contain raw binary data

Let’s intentionally make a mistake now to highlight one of the ways in which logs can be tricky to analyze via command-line, because issues like this do come up frequently.

Examine the log entries containing raw binary data. If you were to cut the fields using a space as the delimiter, what field number does the status code appear in? It’s not the same field number as the normal log entries:

1             2 3 4                     5      
44.224.22.196 - - [02/May/2020:00:56:41 +0000] 
"GET http://example.com/ HTTP/1.1" 200 18 "-" "AWS Security Scanner"
6    7                   8         9   10 11  12   13       14

1             2 3 4                     5      
44.224.22.196 - - [02/May/2020:00:56:44 +0000] 
"\x16\x03\x01\x00\xD2\x01\x00\x00\xCE\x03\x03\xE6\xE7\xA4\x1B(AU\x87Ij\x8D\x10\xD0\xF3M<\xB2G:\xC2<x\xC3\xB2\xD2\xB0b\xC8\x14P\xDE\xB2\x00\x00b\xC00\xC0,\xC0/\xC0+\x00\x9F\x00\x9E\xC02\xC0.\xC01\xC0-
\x00\xA5\x00\xA1\x00\xA4\x00\xA0\xC0(\xC0$\xC0\x14\xC0" 400 163 "-" "-"
6                                                       7   8   9   10

So would that mean that we could count the number of these entries by checking the number of times 400 appears in the 7th field?

Try it out:

cat access.log | cut -d " " -f 7 | grep 400 | wc -l

What is the result? Why is it one less than the result we got with Option A? What’s missing?

There’s one log entry that includes a space among the escape sequences, throwing off the field count on just that one line:

1             2 3 4                     5
44.224.22.196 - - [02/May/2020:12:31:11 +0000] 

6
"\x16\x03\x01\x00\xD2\x01\x00\x00\xCE\x03\x03\xE5\x904d\x22y\xB2 

7         8   9   10  11
\x7F\xDD" 400 163 "-" "-"

Always try to verify your answers by more than one method! NCL isn’t shy about testing you with anomalies that your eyes or methods may initially miss. You may find rogue punctuation, failures to remove duplicates due to case sensitivity, and all manner of other real-world issues.

STEP 3 – COUNT THE LINES CONTAINING RAW BINARY DATA

What utility can you append to the option that worked successfully in Step 2, above, to count the number of lines containing raw binary data?

Highlight the box below to check your solution:

cat access.log | grep -v "CONNECT\|HEAD\|GET\|POST" | wc -l

---

Web Access Log #6

Recommended Tools

Utility	Switch	Description
cat		outputs file content, e.g. cat leaping.log
|		“pipe” – takes the output of the command on the left and uses it as input for the command on the right
cut		extracts fields (columns that you specify) from lines
	-d	specifies the use of a delimiter other than the default tab; it can only be one character
	-f	denotes which field(s) to display, indicated by their numerical order of appearance
grep		conducts a case-sensitive search
wc		“word count” – counts the number of lines, words, and characters
	-l	displays the line count only

Approach

The question is specific: we’re only to consider “those invalid entries” uncovered in the last question—the ones containing escape sequences. But how can we tell which of those are trying to establish SSL or TLS connections? The sequences don’t decode into anything that’s human-readable.

STEP 1 – OSINT

Googling is our only way out of this one. I started by throwing as many words as I had into the search box:
web access log invalid request attempt to establish ssl or tls connection “\x”

Try to ignore that the first search result is for Paul Buonopane’s write-up on CryptoKait.com. 😉

I check the first few matches. Google says there are no great matches, and I agree, so I assume I’m being a little too specific or not using the best terminology for my search.

After playing with my search string, this ended up being the one that hit upon it for me(*):
web log 400 attempt to establish ssl or tls connection “\x”

STEP 2 – ISOLATE RELEVANT LOG ENTRIES

What do the invalid log entries that are trying to establish SSL or TLS connections all have in common? What do their requests all begin with? Is it something we can isolate and count?

Again, we run into a real-world issue, rather unintentionally!

If you read Paul Buonopane’s advanced write-up for this challenge, the sequence to look for is identified as \x16\x03\x01\x00. But our Googled source cites a sequence of \x16\x03\x02\x00 and explains the difference if you read a little further down:

\x03\x02 – the SSL/TLS version. In this case “cutting edge” TLS 1.1 (0x03 0x01 is TLS 1.0).

https://isc.sans.edu/forums/diary/SSL+Requests+to+nonSSL+HTTP+Servers/21551/

Sometimes, the version of the information that you find doesn’t match the version of the information you’re looking for.

At least between TLS 1.0 and 1.1, the first two bytes don’t change, so let’s begin our search there. Using the ^ character, there’s a way to tell grep to only look for lines that start with the string you identify. Here’s how that looks (highlight to reveal the full command):

cat access.log | cut -d '"' -f 2 | grep '^\\x16\\x03'

(The backslash needs to be escaped to prevent the system from trying to interpret it as anything other than a literal backslash, and that’s why you see double backslashes in the command.)

STEP 3 – COUNT THE RELEVANT LOG ENTRIES

After isolating the relevant log entries, what utility can we append to count them?

Highlight this box to check your solution:

cat access.log | cut -d '"' -f 2 | grep '^\\x16\\x03' | wc -l

---

Web Access Log #7

Recommended Tools

Utility	Switch	Description
cat		outputs file content, e.g. cat leaping.log
|		“pipe” – takes the output of the command on the left and uses it as input for the command on the right
cut		extracts fields (columns that you specify) from lines
	-d	specifies the use of a delimiter other than the default tab; it can only be one character
	-f	denotes which field(s) to display, indicated by their numerical order of appearance
grep		conducts a case-sensitive search
	-v	excludes lines that match the search term
sort		alphabetizes input, e.g. a, b, c, 1, 101, 2
uniq		deduplicates – detects repeated lines if they are adjacent and removes the duplicates (deduplication)
wc		“word count” – counts the number of lines, words, and characters
	-l	displays the line count only
head		displays the first 10 lines of input; use head -[number] to change the default number

Approach

Thanks to our initial OSINT into the log format, even if we didn’t know before what a user agent string looks like, we do know exactly which field to find it in!

STEP 1 – ISOLATE THE USER AGENT STRINGS

Web Access Log #5 showed us that the space is an unreliable delimiter for parsing out fields in this log. What delimiter in this case is a reliable one? How can you use it to craft a command that will isolate the user agent strings?

Run this command so you can stare at a few entries while considering the best way to get at the user agent strings using the cut utility:

cat access.log | head

Highlight the box below to check your solution:

cat access.log | cut -d '"' -f 6

STEP 2 – SORT AND REMOVE DUPLICATES

With the user agents isolated, how can sort and uniq be used next to show only those user agent strings which are unique?

Highlight the box below to check your solution:

cat access.log | cut -d '"' -f 6 | sort | uniq

STEP 3 – COUNT THE UNIQUE USER AGENT STRINGS

Before we can achieve an accurate count, we need to be sure we’re not including any user agents which are empty or missing. The question cautioned us about these.

When a field is empty or missing in this type of log, we know from Web Access Log #2 that a dash ( - ) acts as a placeholder. Even if we didn’t know that already, though, even without Googling it we could deduce this based on:

• The way the question is worded, which tells us some user agents are empty or missing, and

• The way - looks in comparison to the other user agent strings. Every other string contains descriptive information, while - implies information is missing.

However, if we use grep -v to remove every line that contains a dash somewhere in it, we’ll lose a lot of lines that we intended to keep. Compare the output of these two commands:

cat access.log | cut -d '"' -f 6 | sort | uniq | wc -l

cat access.log | cut -d '"' -f 6 | sort | uniq | grep -v "-" | wc -l

We only wanted to lose the one line with - in the user agent field, but we lost 12 lines! Any user agent that contained a dash has been excluded, e.g. redhat-linux-gnu, en-US, and Nimbostratus-Bot, etc.

We could take the count from the first command above and just minus one to get our answer, but let’s take this opportunity to learn more about grep!

Can we tell grep -v to only remove the user agents that begin and end with the - symbol?

In Web Access Log #6, we were able to use the ^ symbol to tell grep to only look for lines that start with the string we specified. The $ symbol tells grep to match lines that end with the string we specify. We can use both of these symbols together to tell grep the exact match we want to remove. Try it out!

Highlight this box to check your solution:

cat access.log | cut -d '"' -f 6 | sort | uniq | grep -v "^-$" | wc -l

---

Web Access Log #8

Recommended Tools

Utility	Switch	Description
cat		outputs file content, e.g. cat leaping.log
|		“pipe” – takes the output of the command on the left and uses it as input for the command on the right
grep		conducts a case-sensitive search
wc		“word count” – counts the number of lines, words, and characters
	-l	displays the line count only

Approach

This friendly question takes us back to the basics!

1. How would you isolate log entries that specify Firefox?
   Hint: Use grep.
2. How would you count the number of lines you’ve isolated?
   Hint: Use wc.

Highlight the box below to check your solution:

cat access.log | grep Firefox | wc -l

---

Web Access Log #9

Recommended Tools

Utility	Switch	Description
cat		outputs file content, e.g. cat leaping.log
|		“pipe” – takes the output of the command on the left and uses it as input for the command on the right
grep		conducts a case-sensitive search
	-i	makes the search case-insensitive
wc		“word count” – counts the number of lines, words, and characters
	-l	displays the line count only

Approach

STEP 1 – OSINT

First, we’ll need to Google CVE-2020-8515.

Looks like it targets a specific URI… Hmm!

STEP 2 – ISOLATE RELEVANT LOG ENTRIES

Does that URI appear anywhere in the log? How can we find out?

Highlight the box below to check your solution:

cat access.log | grep -i "cgi-bin/mainfunction.cgi"

STEP 3 – COUNT THE RELEVANT LOG ENTRIES

The URI appears infrequently enough that we can count the instances by hand, but is there a utility you could append to the end of the command to count the number of instances automatically?

Highlight the box below to check your solution:

cat access.log | grep -i "cgi-bin/mainfunction.cgi" | wc -l

---

What’s Next?

---

Your feedback matters! Share your thoughts about this series below!