In order to form a HTTP request, a TCP request is made to a server with a given IP address (usually resolved by looking up the host through DNS). The server can run multiple hosts (e.g. my domain.com and myotherdomain.com). A host can have multiple paths (e.g. /, /index.html, /article). A path can have multiple methods. Methods are generally standardized for GET, POST, PUT, DELETE, OPTIONS, HEAD, PATCH, CONNECT, TRACE, and more as the standard and HTTP version evolves. We refer to an endpoint as the host+path+method. If you are the developer of the server, you can pick which methods you support and even create custom ones to interoperate with your bespoke clients. Depending on the path and the method, you may require authentication.
A basic HTTP requests appears as follows:
GET /software/rust-http-server/ HTTP/1.0 Host: www.chriswirz.com:443 Content-Length: 0
and the server may respond with something like
HTTP/1.0 200 OK Date: Tue, 06 Mar 2018 16:34:55 GMT Content-Type: text/json Content-Length: 4 Body
Within the server, there are several steps to get to a response. For a basic server, the the sequence is as follows:
-
Receive TCP request
and
Handle request in background thread - Check cache for handle
-
If not in cache, Check disk
(store in cache if found on disk) - Return 200 or 40X response
It is very important that HTTP servers have a cache for small files/blobs. Disk operations take far longer than operations to memory, and disks are far less durable. Overall, the server will perform better and last longer if it efficiently uses a cache. One should not fill the cache with large and rarely-used files - so understanding the content you are serving is important in writing a custom server.
This example is a good "getting started" project for any language, so let's explore Rust.
To begin, let's install Rust and use cargo to initialize a project.
cargo new rust_http
With the project created, we first add our dependencies.
In this case, we are going to add lazy_static which will help us make a thread-agnostic cache.
cargo add lazy_static
Now, we can head to the src folder and design the cache.
For this example, I call the cache a blob_map.
After all, it's just going to be a vector of bytes - ready to send as a HTTP response.
The key is the handle/uri and the value is the "blob".
use std::collections::HashMap;
use std::sync::Mutex;
#[derive(Clone)]
pub struct BlobItem {
mime_type: String,
size: usize,
bytes : Vec<u8>,
}
lazy_static! {
static ref BLOB_MAP: Mutex<HashMap<String, BlobItem>> = {
let mut m = HashMap::new();
m.insert("null".to_string(), BlobItem {
mime_type: "text/html".to_string(),
size: 0,
bytes : Vec::new(),
});
return Mutex::new(m);
};
}
pub fn blob_map_contains_key(key: &String) -> bool {
match BLOB_MAP.lock() {
Ok(unrapped_blob_map) => unrapped_blob_map.contains_key(key),
Err(_) => false,
}
}
pub fn blob_map_get(key: &String) -> Option<BlobItem> {
match BLOB_MAP.lock() {
Ok(unrapped_blob_map) => unrapped_blob_map.get(key)
.map(|i|
return BlobItem {
mime_type: i.mime_type.clone(),
size: i.size,
bytes: i.bytes.clone(),
}
),
Err(_) => None,
}
}
pub fn blob_map_insert(key: &String, mime_type: &String, bytes : &Vec<u8>) -> Option<BlobItem> {
match BLOB_MAP.lock() {
Ok(mut unrapped_blob_map) =>
unrapped_blob_map.insert(
key.to_owned(),
BlobItem {
mime_type: mime_type.to_owned(),
size: bytes.len(),
bytes: bytes.to_owned()
}
),
Err(_) => None,
}
}
This gives us the methods blob_map_contains_key,blob_map_get, and blob_map_insert.
Because the browser is going tbe be encoding portions of the URI so it won't match what we see on disk, we need a decode trait.
trait HttpDecode {
fn uri_decode(self) -> String;
}
impl HttpDecode for &str {
fn uri_decode(self) -> String {
self.replace("%20", " ")
.replace("%2E", ".")
}
}
mod blob_map;
/* Simple HTTP Server */
use std::env;
use std::fs;
use std::io::prelude::*;
use std::net::TcpListener;
use std::net::TcpStream;
use std::borrow::BorrowMut;
use std::borrow::Borrow;
use std::path::Path;
use std::thread;
#[macro_use]
extern crate lazy_static;
include!("mime_types.rs");
include!("http_decode.rs");
include!("blob_map.rs");
#[derive(Clone)]
pub struct AppArguments {
host: String,
port: String,
base_directory: String,
}
impl AppArguments {
pub fn host(&self) -> &str {
self.host.as_ref()
}
pub fn set_host(&mut self, host: String) {
self.host = host;
}
pub fn port(&self) -> &str {
self.port.as_ref()
}
pub fn set_port(&mut self, port: String) {
self.port = port;
}
pub fn base_directory(&self) -> &str {
self.base_directory.as_ref()
}
pub fn set_base_directory(&mut self, base_directory: &String) {
self.base_directory = base_directory.to_owned();
}
pub fn debug(&mut self) {
println!("--host {} --port {} --base-directory {}", self.host(), self.port(), self.base_directory());
}
}
lazy_static! {
static ref MIME_TYPES: HashMap<&'static str, &'static str> = HashMap::from([
("html", "text/html")
// Other mime types from
])
}
fn write_and_flush(mut stream: &TcpStream, response_contents: &BlobItem) {
stream
.write(
format!(
"{}\r\n{}\r\n{}\r\n\r\n",
"HTTP/1.1 200 OK",
format!("Content-Type: {}", &response_contents.mime_type),
format!("Content-Length: {}", &response_contents.size)
)
.as_bytes(),
)
.unwrap();
stream.write(&response_contents.bytes).unwrap();
stream.flush();
}
fn handle_buffer_and_connection(buffer: [u8; 1024], stream: TcpStream, args: &AppArguments) {
let s: std::borrow::Cow<str> = String::from_utf8_lossy(&buffer[..]);
let mut line_number = 0;
let mut resource_path = std::path::Path::new(args.base_directory()).join("index.html");
for line in s.split("\n") {
let trimmed = line.trim();
if trimmed.eq("") {
// headers are delimited by an empty line
break;
}
if line_number == 0 {
let parts = trimmed.split(" ");
let mut part_number = 0;
for part in parts {
if part_number == 1 { // Format is 'GET resource_path HTTP/1.0'
if part.starts_with("/") {
resource_path = std::path::Path::new(args.base_directory()).join(part.to_string()[1..].split("?").next().unwrap().uri_decode());
} else {
resource_path = std::path::Path::new(args.base_directory()).join(part.split("?").next().unwrap().uri_decode())
}
break;
}
part_number = part_number + 1;
}
break;
}
line_number = line_number + 1;
}
println!("Attempting to load: {}", resource_path.display());
// Get the request path before '?'
let request_path: String = String::from(resource_path.as_os_str().to_str().unwrap());
// Check the cache
if blob_map_contains_key(&request_path) {
println!("Found {} in cache", request_path);
write_and_flush(stream.borrow(), &blob_map_get(&request_path.to_string()).unwrap());
return;
}
else if resource_path.exists() && !resource_path.is_dir() {
// Check the disk
println!("Resource path {} exists and is not a directory", resource_path.display());
let bytes = fs::read(&request_path).unwrap();
if bytes.len() < 16384 {
println!("Caching {}", resource_path.display());
// Add to cache if reasonably small
blob_map_insert(
&request_path,
&get_mime_type_for_path(&resource_path).to_string(),
&bytes
);
write_and_flush(stream.borrow(), &blob_map_get(&request_path.to_string()).unwrap());
return;
}
println!("Responding with {} bytes", bytes.len());
write_and_flush(stream.borrow(), BlobItem {
mime_type: get_mime_type_for_path(&resource_path).to_string(),
size: bytes.len(),
bytes: bytes.to_owned()
}.borrow());
return;
} else if resource_path.exists() && resource_path.is_dir() {
match fs::read_dir(resource_path) {
Ok(paths) => {
let mut response_html = "<html>\r\n\t<body>".to_owned();
for path in paths {
match path {
Ok(entry) => {
let obj_path = &entry.path().display().to_string()[args.base_directory().len()..];
response_html.push_str(&format!(
"<a href=\"{}\">{}</a><br>",
obj_path,
obj_path
))
},
Err(e) => {
println!("Error resolving path: {e:?}")
}
}
}
blob_map_insert(&request_path, &"text/html".to_string(), &Vec::from(response_html.as_bytes()));
write_and_flush(stream.borrow(), &blob_map_get(&request_path.to_string()).unwrap());
}
Err(e) => {
println!("error reading file directory: {e:?}")
}
};
} else {
match fs::read_dir(args.base_directory()) {
Ok(paths) => {
let mut response_html = "<html>\r\n\t<body>".to_owned();
for path in paths {
match path {
Ok(entry) => {
let obj_path = &entry.path().display().to_string()[args.base_directory().len()..];
response_html.push_str(&format!(
"<a href=\"{}\">{}</a><br>",
obj_path,
obj_path
))
},
Err(e) => {
println!("Error resolving path: {e:?}")
}
}
}
response_html.push_str("\r\n\t</body>\r\n</html>");
}
Err(e) => {
println!("error reading file directory: {e:?}")
}
};
}
write_and_flush(stream.borrow(), BlobItem {
mime_type: "text/html".to_string(),
size: 0,
bytes : Vec::new(),
}.borrow());
}
// handle a connection as it comes in
fn handle_connection(mut stream: TcpStream, args: &mut AppArguments) {
let mut buffer: [u8; 1024] = [0; 1024];
match stream.read(&mut buffer) {
Err(e) => {
println!("error reading TCP stream to buffer: {e:?}")
}
Ok(_) => {
handle_buffer_and_connection(buffer, stream, args);
}
}
}
fn main() {
// Define the host and port
let mut args = AppArguments {
host: "0.0.0.0".to_string(),
port: "8675".to_string(),
base_directory: ".".to_string()
};
let passed_args: Vec<String> = env::args().collect();
println!("");
println!("running rust_http with arguments:");
let arg_max_index = passed_args.len() - 1;
for i in 0..arg_max_index {
if passed_args[i].eq("--port") && i < arg_max_index {
args.set_port(passed_args[i + 1].to_owned());
}
if passed_args[i].eq("--host") && i < arg_max_index {
args.set_host(passed_args[i + 1].to_owned());
}
if passed_args[i].eq("--base-directory") && i < arg_max_index {
args.set_base_directory(&passed_args[i + 1]);
}
println!("\t{}: {}", i, passed_args[i]);
}
if args.base_directory().eq(".") {
match env::current_dir() {
Err(e) => {
println!("error reading current directory: {e:?}")
}
Ok(dir) => {
args.set_base_directory(&dir.as_path().display().to_string());
println!("");
println!("Current directory: {}", args.base_directory())
}
}
}
println!("Arguments:");
args.debug();
println!("");
// Create TCP Listener
match TcpListener::bind(format!("{}:{}", args.host(), args.port())) {
Err(e) => {
println!("error in TcpListener::bind: {e:?}")
}
Ok(tcp_listener) => {
println!("Web server is listening at port {}", args.port());
// Connect to any incoming connections
for stream in tcp_listener.incoming() {
// Handle each connection in background thread
thread::scope(|s| {
match stream {
Ok(tcp_stream) => {
s.spawn(||{
handle_connection(tcp_stream, args.borrow_mut());
});
},
Err(e) => {
println!("error in TcpListener::incoming: {e:?}")
},
}
});
}
}
};
}
Now let's test it! In action. We build and run it.
cargo build
cargo run
Now we go to http://localhost:8675/ (where 8675 is the port) in the browser and see responses at terminal.
Attempting to load: C:\Users\admin\Desktop\rust_http\sample_response.txt
Resource path C:\Users\admin\Desktop\rust_http\sample_response.txt exists and is not a directory
Caching C:\Users\admin\Desktop\rust_http\sample_response.txt
Attempting to load: C:\Users\admin\Desktop\rust_http\sample_response.txt
Found C:\Users\admin\Desktop\rust_http\sample_response.txt in cache
Attempting to load: C:\Users\admin\Desktop\rust_http\favicon.ico
Resource path C:\Users\admin\Desktop\rust_http\favicon.ico exists and is not a directory
Caching C:\Users\admin\Desktop\rust_http\favicon.ico
Attempting to load: C:\Users\admin\Desktop\rust_http\sample_response.txt
Found C:\Users\admin\Desktop\rust_http\sample_response.txt in cache
Attempting to load: C:\Users\admin\Desktop\rust_http\favicon.ico
Found C:\Users\admin\Desktop\rust_http\favicon.ico in cache
As shown, in just a few hundred lines of code, one can create a reasonable HTTP server, in Rust, from scratch. You can grab these files if you like, and give it a try yourself.