Range Requests

Efficiently transferring large files over the web has always been a challenge, but thanks to the clever use of HTTP headers, we have a powerful tool at our disposal: the Range header. Imagine being able to download just a specific portion of a massive file instead of waiting for the entire content to arrive. This is where the Range header comes into play. In this article, we’ll delve into the world of HTTP’s Range header, exploring its purpose and practical applications. Whether you’re a developer looking to optimize file transfers or simply curious about the technology that makes streaming media seamless, let’s uncover how the Range header revolutionizes the way we interact with online content.

At its core, the Range header is a crucial feature of the HTTP that allows clients, such as web browsers or download managers, to request specific portions of a resource from a server rather than the entire content. This mechanism is especially valuable when dealing with large files, such as videos, and audio clips, where waiting for a complete download could be time-consuming and bandwidth-intensive. By sending a Range header along with a GET request, the client can specify a byte range within the resource it wants to retrieve. The server then responds with the requested range of bytes typically accompanied by a 206 Partial Content status code, indicating that the response contains only a portion of the resource. This elegant approach optimizes data transfer by fetching only the needed data and facilitates smoother streaming experiences, quicker downloads, and reduced strain on client and server resources.

Assume you’re downloading a file and abruptly your internet blinks out, leaving you with only the first 2000 bytes. Instead of re-downloading from scratch, the browser will ask for just the missing parts. Imagine you’ve got the first slice of a file and need the rest. With the Range header, you send a request starting from where you left off – say, from byte 2000 onward. The server nods appreciatively and sends what you need. Your browser might shoot this request with a header like this:

GET /large-document.pdf HTTP/1.1
Host: www.example.com
Range: bytes=2000-
User-Agent: MyDownloadManager/2.0

Most of the servers accommodate range requests. Servers can signal their availability to accept range requests by adding the “Accept-Ranges” header within their responses:

HTTP/1.1 200 OK
Date: Thu, 13 Aug 2023 16:45:30 GMT
Server: Nginx/2.1.8
Accept-Ranges: bytes

Range headers play a big role in widely-used file-sharing apps like torrents. They help these apps download various sections of videos or music at the same time but from different users. This speeds up the process and makes it more efficient.

Let’s write a download manager

You might remember using download manager software such as IDM or Free Download Manager, where you’d see a window like this:

By looking at that picture, we can say how to create a simple download manager:

1. Get the web address (URL).
2. Ask for the size of the file with a request.
3. Plan the sections based on the file size.
4. Request each section separately and begin downloading at the same time.
5. Put together the downloaded pieces to complete the file.

I found a simple code in Github and I tweaked the code to enhance its clarity. You can find the source code in the resources section.

Here’s the modified version with around 100 lines of code:

package main

import (
	"errors"
	"fmt"
	"io/ioutil"
	"net/http"
	"os"
	"strconv"
	"sync"
	"time"
)

type DownloadManager struct {
	SourceUrl     string
	TargetPath    string
	TotalSections int
}

func main() {
	startTime := time.Now()
	dm := DownloadManager{
		SourceUrl:     "", // Provide URL to download
		TargetPath:    "", // Provide target file path
		TotalSections: 10,
	}
	_ := dm.Do()
	fmt.Printf("Download took %v seconds\n", time.Now().Sub(startTime).Seconds())
}

func (dm DownloadManager) Do() error {
	r, err := dm.sendRequest("HEAD")
	resp, err := http.DefaultClient.Do(r)
	if resp.StatusCode > 299 {
		return errors.New(fmt.Sprintf("Can't process, status: %v", resp.StatusCode))
	}

	size, err := strconv.Atoi(resp.Header.Get("Content-Length"))
	fmt.Printf("Size: %v bytes\n", size)

	var sections = make([][2]int, dm.TotalSections)
	eachSize := size / dm.TotalSections
	fmt.Printf("Each section: %v bytes\n", eachSize)

	for i := range sections {
		sections[i][0] = i * eachSize
		sections[i][1] = sections[i][0] + eachSize
		if i == dm.TotalSections-1 {
			sections[i][1] = size - 1
		}
	}

	var wg sync.WaitGroup
	for i, section := range sections {
		wg.Add(1)
		go func(i int, sec [2]int) {
			defer wg.Done()
			err = dm.downloadSection(i, sec)
			if err != nil {
				panic(err)
			}
		}(i, section)
	}
	wg.Wait()

	return dm.mergeFiles(sections)
}

func (dm DownloadManager) downloadSection(i int, c [2]int) error {
	r, _ := dm.sendRequest("GET")
	r.Header.Set("Range", fmt.Sprintf("bytes=%v-%v", c[0], c[1]))
	resp, _ := http.DefaultClient.Do(r)
	if resp.StatusCode > 299 {
		return errors.New(fmt.Sprintf("Can't process, status: %v", resp.StatusCode))
	}
	fmt.Printf("Downloaded %v bytes for section %v\n", resp.Header.Get("Content-Length"), i)
	bytes, _ := ioutil.ReadAll(resp.Body)
	_ = ioutil.WriteFile(fmt.Sprintf("section-%v.tmp", i), bytes, os.ModePerm)
	return nil
}

func (dm DownloadManager) sendRequest(method string) (*http.Request, error) {
	r, _ := http.NewRequest(method, dm.SourceUrl, nil)
	r.Header.Set("User-Agent", "Silly Download Manager v001")
	return r, nil
}

func (dm DownloadManager) mergeFiles(sections [][2]int) error {
	f, err := os.OpenFile(dm.TargetPath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, os.ModePerm)
	if err != nil {
		return err
	}
	defer f.Close()
	for i := range sections {
		tmpFileName := fmt.Sprintf("section-%v.tmp", i)
		b, err := ioutil.ReadFile(tmpFileName)
		if err != nil {
			return err
		}
		n, err := f.Write(b)
		if err != nil {
			return err
		}
		err = os.Remove(tmpFileName)
		if err != nil {
			return err
		}
		fmt.Printf("%v bytes merged\n", n)
	}
	return nil
}

Resources:
What Are HTTP Range Requests?
go-download-manager – Github