A Mirror Cache

I regularly rebuild machines in my home lab. To cut down on the delay and bandwidth required to download the same package multiple times a forward proxy can be employed to cache and serve files.

Squid Configuration

cache_dir ufs /var/squid/cache 67275 16 256
refresh_pattern -i .rpm$ 129600 100% 129600 refresh-ims override-expire
maximum_object_size 4096 MB
cache_replacement_policy heap LFUDA

Keeping a maximum of 67GB of data, spread out over the 16 directories and 256 subdirectories
Since this cache is used for packages there is no need to HTTP cache control headers. Instead raise up the expiration times (in minutes) to provide a lifetime of 90 days
By default squid does not cache objects over 4MB, crank this up to 4GB
Enable dynamic aging

Finally, initialize the cache directory

squid -z

Manual Verification

A cache hit is indicated by the X-Cache header

$ src=http://eradman.com/entrproject/code/entr-5.2.tar.gz
$ curl --proxy http://192.168.3.2:3128 -D - -s -o /tmp/entr-5.2.tgz $src
HTTP/1.1 200 OK
Content-Length: 25834
Content-Type: application/octet-stream
Date: Fri, 19 Aug 2022 05:23:53 GMT
Last-Modified: Thu, 05 May 2022 15:19:44 GMT
Server: OpenBSD httpd
X-Cache: MISS from xa10.eradman.com
Via: 1.1 xa10.eradman.com (squid/5.5)
Connection: keep-alive

$ sleep 170

$ curl --proxy http://192.168.3.2:3128 -D - -s -o /tmp/entr-5.2.tgz $src
HTTP/1.1 200 OK
Content-Length: 25834
Content-Type: application/octet-stream
Date: Fri, 19 Aug 2022 05:23:53 GMT
Last-Modified: Thu, 05 May 2022 15:19:44 GMT
Server: OpenBSD httpd
Age: 175
X-Cache: HIT from xa10.eradman.com
Via: 1.1 xa10.eradman.com (squid/5.5)
Connection: keep-alive

Client Configuration

On RPM-based systems yum will use this cache after setting the proxy option in /etc/dnf/dnf.conf

proxy=http://192.168.3.2:3128

Or for Ubuntu a configuration file in /etc/apt/apt.conf.d/

Acquire::http::Proxy "http://192.168.3.2:3128";
Acquire::https::Proxy "http://192.168.3.2:3128";

Failover

On OpenBSD and FreeBSD, Common Address Redundancy Protocol provides a relatively simple way to set up an IP that will failover if one server is offline. In this example an IP alias is defined on an interface that will float between systems

# /boot/loader.conf
carp_load="YES"

# /etc/rc.conf
ifconfig_em0_alias0="inet vhid 1 advskew 100 pass ****** alias 192.168.3.2/32"

Docker Proxy

Proxy settings for Docker has two aspects: access to repository images and the environment inside a running image.

Adjust the docker service a systemd environment variable to /etc/systemd/system/docker.service.d/

[Service]
Environment="HTTP_PROXY=http://192.168.3.2:3128"
Environment="HTTPS_PROXY=http://192.168.3.2:3128"
Environment="NO_PROXY=localhost,127.0.0.0/8"

To inject proxy settings into the docker runtime, create $HOME/.docker/config.json

{
  "proxies": {
    "default": {
      "httpProxy": "http://10.10.10.1:3129",
      "httpsProxy": "http://10.10.10.1:3129",
      "noProxy": "localhost,127.0.0.0/8"
    }
  }
}

Python

Pip can be configured to use a proxy using .config/pip/pip.conf

[global]
proxy = http://192.168.3.2:3128

Docker Proxy-Proxy

The default networking for docker is bridge mode, which does not mean containers are bridged with the host, but only with each other. This scheme breaks path MTU discovery, so it is sometimes useful to set up another proxy as a workaround

To set the docker brige IP create /etc/docker/daemon.json

{
  "bip": "10.10.10.1/24"
}

Then configure Tinyproxy to forward requests to Squid. Binding on the IP of the docker0 inerface allows docker images a known endpoint for HTTP requests

User tinyproxy
Group tinyproxy
Port 3129
Listen 10.10.10.1
Timeout 600
DefaultErrorFile "/usr/share/tinyproxy/default.html"
StatFile "/usr/share/tinyproxy/stats.html"
LogFile "/var/log/tinyproxy/tinyproxy.log"
PidFile "/run/tinyproxy/tinyproxy.pid"
Upstream http 192.168.3.2:3128
MaxClients 50
Allow 0.0.0.0/0
StartServers 2

Eric Radman : a Journal