curl, wget, python simple http server, Apache2

Objective: Practice using wget and curl to copy files from web servers. Use Apache2 and Python's Simple HTTP servers.

Discussion: Many Linux systems include wget and curl. You can read about the differences between the two tools here, but for your initial introduction, realize that the two tools are similar. Both are command-line tools (you know, no GUI). Both read content from web servers, but neither renders that content as a page like you would be used to with a traditional browser. Rendering content requires the tool (i.e., the browser) to parse HTML and present the page with text, colors, menus, bars, buttons, pictures and all kinds of actions (made possible with scripting languages and engines). Neither wget or curl can parse html or run client-side scripts. One difference between wget and curl is how the tools handle data. Wget writes data to a file, while curl by default writes data to stdout. The term stdout might be new. In short, stdout refers to output or response from some command being presented in the terminal or shell. For example, running the command whoami returns the current user's username. Stdout writes that output (the username) to the terminal. We could redirect stdout to a file or to a null page, depending on what was needed, but by default stdout would go to the terminal.

In this exercise, you will use wget and curl to transfer text and binary files. Unlike netcat, both wget and curl require a http(s) server. Recall, we used netcat to transfer files through a raw tcp socket using a netcat listener. Netcat did not need or send http requests. The tools wget and curl depend on http, and therefore require content to be served on a http(s) server. There are lots of http servers you could choose from. The two most popular Linux http servers are Apache2 and Nginx. Installing Apache2 and Nginx on Linux servers are very similar, although configuring the two servers can be quite different. We will practice with Apache2.

Both Apache2 and Nginx serve pages from a webroot. The webroot is the directory where the server will look for requested resources. A common webroot in Linux systems is /var/www/html. The URL http://localhost/index.html would look for the resource index.html in the webroot (/var/www/html/index.html). The URL http://localhost/dev/index.html would look for the resouce /dev/index.html, also in the webroot (/var/www/html/dev/index.html). Python has a simple http server that can be used when you need a temporary http server for security testing or system administration needs. Although there are simple http servers for both Python2 and Python3, we will specifically use Python3's simple http server. Python's simple http server uses the working directory as the webroot. The webroot is created dynamically when the Python simple http server is started while the webroot for more robust http servers such as Apache2 and Nginx are statically configured in system files. Python simple http servers can conveniently meet many security testing needs, so you will also practice with Python3's simple http server.

Before we get started, spin up an Ubuntu container.

sudo docker run -it -d -p 8081:80 --name web ubuntu

Apache2

Install apache2, python3 and net-tools in the Ubuntu container (you already know how to get into the container).

apt update && apt install apache2 python3 net-tools -y

Examine the webroot, directory /var/www/html. Note that only index.html is currently in the webroot. Let's overwrite the default index.html with our own content.

echo "My fancy schmancy web page" > /var/www/html/index.html

Now, let's create a new page in the webroot (type the following if cut & paste mangles the commands).

echo "echo 'The user is:'" >> /var/www/html/dev-page.html

echo whoami >> /var/www/html/dev-page.html

Start apache2

service apache2 start

Use netstat to verify that apache2 is listening on TCP 80. Also, note the IP address for your Ubuntu container.

The index.html and dev-page.html are set up, and we know apache2 is listening on TCP 80. Exit the Ubuntu container.

curl

Curl your web server. Use your correct IP.

curl http://172.17.0.X

Take a screenshot.

Curl /dev-page.html. Again, user your correct IP.

curl http://172.17.0.X/dev-page.html

Take a screenshot.

So far, all we have done is read the files. Now, we will copy those files to the /tmp directory.

curl http://172.17.0.X -o /tmp/index2.html

curl http://172.17.0.X/dev-page.html -o /tmp/dev-page2.html

Read out the file /tmp/index2.html and /tmp/dev-page2.html using cat. Take a screenshot of the contents.

Notice that /tmp/dev-page.html looks like bash commands (echo and whoami). The file is a bash script. Let's execute it by reading the file with cat and piping the output to bash. Take a screenshot with the results.

cat /tmp/dev-page2.html | bash

That's interesting. Try to read /dev-page.html from the Ubuntu server using curl again, but this time pipe the output to bash. Take a screenshot of the results..

curl http://172.17.0.X/dev-page.html | bash

wget

Change directory to /tmp.

Use the wget command to copy index.html and dev-page.html from the web server.

wget http://172.17.0.X

wget http://172.17.0.X/dev-page.html

List the contents of /tmp showing index.htmp and dev-page.html. Take a screenshot.

Python3 Simple HTTP Server

The following sections requires two terminals. One terminal will be used to start the Python3 Simple HTTP Server in the Ubuntu container, the other terminal will be used to run commands from the GCP instance.

In the Ubuntu containter, change directory to /tmp. Add the following content to /tmp/python.sh. The following is the content of /tmp/python.sh, not commands to run. Use examples above to figure out how to add this content to /tmp/python.sh.

echo 'my python3 version is'

python3 -V

In the Ubuntu container, change directory to /tmp.

In the Ubuntu container, start the Python3 Simple HTTP Server. The default port would by TCP 8000 but we will start the server on TCP 9000 by specifying a port.

python3 -m http.server 9000

On the GCP instance, grab a copy of /python.sh and save it /tmp on the GCP instance. Note that :9000 specifies TCP 9000. Apache2 is on TCP 80. Attempting to read /python.sh on TCP 80 would result in a 404 error (page not found). The Python3 HTTP server is on TCP 9000. List the /tmp directory and take a screenshot. Also, examine the Ubuntu server terminal and note the http request and response.

curl http://172.17.0.X:9000/python.sh -o /tmp/python.sh

Finally, use earlier steps to curl /python.sh on the Python3 http server and pipe it into bash. Verify that the bash script in the file executed and take a screenshot of the results.

Submission

Screenshot of curl http://172.17.0.X .

Screenshot of curl http://172.17.0.X/dev-page.html.

Screenshot of /tmp/index2.html and /tmp/dev-page2.html contents.

Screenshot of cat /tmp/dev-page2.html | bash.

Screenshot of curl http://172.17.0.X/dev-page.html | bash.

Screenshot of /tmp showing index.htmp and dev-page.html.

Screenshot of /tmp show python.sh.

Screenshot of executing python.sh by piping the output of curl to bash.

Apache2 is published to published to TCP 8081 on the GCP instance. Provide the command to grab /dev-page.html from your GCP instance. Choose whichever tool you like.