The HTTP protocol is used by web browsers to obtain documents hosted by servers.
The browser connects to a server via TCP, on port 80 by default. The minimal request contains a command (here GET), a URL (/hello.txt), an empty line.

GET /hello.txt

The answer contains the requested file.

Hello world

$ printf 'GET /hello.txt \n\r\n' | nc localhost 8001

HTTP protocol version 1.1 is optimized for complex HTML pages transferring, and allows negotiation of language and encoding formats.
The minimal request contains a command (GET), a url (/hello.txt), the version (HTTP/1.1), the Host field, an empty line.

GET /hello.txt HTTP/1.1
Host: 10.10.10.11 80

The answer contains a header, composed of many fields (server, date,...), the length of the content (here 13), and the content as text.

HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/2.7.15+
Date: Thu, 26 Dec 2019 17:06:12 GMT
Content-type: text/html; charset=UTF-8
Content-Length: 13

Hello world

The headers of the answer contains lots of usefull information about the server, its version...
Here is an Apache server in version 0.8.4 that runs scripts with a php interpreter in version 7.3.13.

HTTP/1.1 200 OK
Server: Apache/0.8.4
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Powered-By: PHP/7.3.13

$ printf 'GET / HTTP/1.1\r\nHost: localhost 8001\r\n\r\n' | nc localhost 8001

HTTP response codes

    1xx informational response – the request was received, continuing process
    2xx successful – the request was successfully received, understood, and accepted
        200 OK
    3xx redirection – further action needs to be taken in order to complete the request
        301 Moved Permanently => Redirection
        304 Not Modified
    4xx client error – the request contains bad syntax or cannot be fulfilled
        400 Bad request
        401 Unauthorized
        403 Forbidden
        404 Not found
    5xx server error – the server failed to fulfil an apparently valid request
        500 Internal Server Error
        502 Bad Gateway - The server was acting as a gateway or proxy and received an invalid response from the upstream server
        503 Service Unavailable - The server cannot handle the request (because it is overloaded or down for maintenance)
        504 Gateway Timeout - The server was acting as a gateway or proxy and did not receive a timely response from the upstream server.
    https://en.wikipedia.org/wiki/List_of_HTTP_status_codes

Download a file :

$ wget http://localhost:8001/

Print the content of the file :

$ curl http://localhost:8001/

Print the HTTP headers and the content of the file :

$ curl -v http://localhost:8001/

HTTP headers are standardized and contain usefull informations.

• Host: the target of the request.
• Referer: Address of the web page that generated the query.
  
• User-Agent: Browser used to log in.
  

Custom headers can be freely added. Here 'X-MyHeader: value':

GET / HTTP/1.1
Host: localhost:8001
User-Agent: curl/7.58.0
Accept: */*
X-MyHeader: value

Add a custom Header with curl : -H 'header: value'

$ curl -H 'X-MyHeader: yoloforpresident' http://localhost:8001

The header of HTTP requests contains a User-Agent field.
It is used by browsers to identify themselves and give information about their version.
The server can adapt the page to an older Internet Explorer or a newer Firefox browser.

Curl v7.68.0 uses 'curl/7.68.0' as the User-Agent. It is possible to set the User-Agent value with the -A option.

curl -A 'MyAgent' http://localhost:8001

An URI (Universal Resource Locator) specify an internet resource, followed by optional parameters.
The first parameter is identified by a ?, the following ones by a &.
. If the parameters contain a ? or a &, they are encoded as %3F and %26. This is called Percent (%) encoding.
.https://fr.wikipedia.org/wiki/Percent-encoding

Exemple:

$ curl 'http://localhost:8001/register.php?name=jean&lastname=bon&age=42&admin=true'

As unix shell uses the & to launch background tasks, it is imperative to put the URL between quotes '.

HTTP has basic authentication feature, based on a field containing a username and a password in clear text.
login:password is base64 encoded then added in the request header.
Authorization: Basic bG9naW46cGFzc3dvcmQ=

Exemple:

GET /hello.txt HTTP/1.1
Host: localhost:8001
Authorization: Basic bG9naW46cGFzc3dvcmQ=
User-Agent: curl/7.58.0
Accept: */*

Basic auth with curl:

$ curl -u login:password http://localhost:8001/hello.txt

Base64 encode login:password in shell

$ printf 'login:password' | base64
bG9naW46cGFzc3dvcmQ=

Base64 decode

$ printf 'bG9naW46cGFzc3dvcmQ=' | base64 -d
login:password

Bruteforce Basic auth with curl and rockyou password list:

for i in `cat rockyou.txt`; do printf \n$i:; curl  -u admin:$i http://12.10.1.11/training-http-auth-simple.php; done

Web pages send Form fields either using GET method in the URL parameters, or POST method in the request body.

Parameter of GET requests are saved in logs files, and have limited length.
Parameter of POST requests do not appear in the logs, and are not limited in length. It is thus possible to upload large files.
These parameters are encoded with 'Percent encoding'.
Example:

POST /login.php HTTP/1.1
Host: 10.10.1.11
Content-Type: application/x-wwww-form-urlencoded
Content-Length: 24

login=James&password=007

Post a form with curl:

$ curl -X POST -F 'username=admin' -F 'password=megapassword'  http://localhost:8001/

Forms are HTML base objects, enclosed by <form> and </form> tags.
<input name='xxx'>: Text input fields
<form action='xxx': URL to send the Form to. If empty send to the current URL.
Exemple:

<form action="/action_page.php"> 
  First name: <input type="text" name="firstname" value="Mickey">
  Last name:  <input type="text" name="lastname" value="Mouse">
  <input type="submit" value="Submit"> 
</form> 

Post a Form thanks curl:

$ curl -X POST -F 'firstname=Mickey' -F 'lastname=Mouse'  http://12.10.1.11/action_page.php

HTML Forms most often use the POST method, sometime also the GET method is used.
Paramètres are sent as request URI arguments.
When using curl in shell, dont forget to use quotes '
Exemple:

$ curl 'http://localhost:8001/register.php?name=jean&lastname=bon&age=42&admin=true'

It's common to procces form fields values in JavaScript before sending them to the server.
JavaScript natively makes use of JSon format to exchange structured datas.
Exemple: {key1:value1, key2:value2}.
The Content-Type header is then set to json: Content-Type: application/json

Exemple:

POST / HTTP/1.1
Host: localhost:8001
User-Agent: curl/7.58.0
Accept: */*
Content-Type: application/json
Content-Length: 34

{"key1":"value1", "key2":"value2"}

Curl:

$ curl --header "Content-Type: application/json" -X POST --data  '{"key1":"value1", "key2":"value2"}' http://10.10.1.11/

Forms are often used to register, login, and upload files.
Lots of sanity checks can be done in JavaScript in the browser before the upload or on the server after the upload.
Filename and file extension, are often checked, sometime even the file header is chekced to verify wether it's an image or a php file.
A full chapter is dedicated to file upload and filter bypass.

A File field can be identified by the following HTML code

<input type=file name=fileToUpload>:

Curl command

curl -X POST -F 'fileToUpload=@./picture.jpg' http://10.10.1.11/upload

Cookies are used to store data on the browser, that will be reused at the next session.
They can contain everything: choices of language, colors, and sometimes password...

Read cookies in the server response and store them in a cookie jar.

$ curl  -c cookies.txt http://10.10.1.11/cookies.php

Send a stored cookie, and update its value with the response

$ curl -b cookies.txt -c cookies.txt http://10.10.1.11/cookies.php

Send a manually crafted cookie

$ curl -b 'code=1447' http://10.10.1.11/cookies.php