Linux Proxy Acceleration: Solutions for apt and Docker

On Windows and mobile devices, implementing a proxy is straightforward. You can simply open a software and it will start running and have some effect. However, on Linux, there is no user interface or run button, making it difficult to get started.

Tired of the slow speed and frequent disconnections when pulling images from domestic machines, and waiting half an hour for an apt update, AhFei has also been tormented.

The content of this article should be able to solve 99% of the proxy acceleration needs. How do you download GitHub projects on domestic machines? How do you use apt to quickly update software and use Docker to pull images on domestic machines?

Applicable systems: Debian-based distributions, including Ubuntu and Armbian

I might have made some mistakes, please let me know if I’ve gotten anything wrong!

Why not use image acceleration?

When AhFei first started using Linux, he saw a lot of methods for configuring mirrors online. However, these mirror sources didn't seem to have any noticeable acceleration effect on his machine. So his impression of mirror acceleration was that it was garbage.
Proxy acceleration is a powerful method. It works well and is widely applicable.

Next, how to install the magic software, edit the configuration, and obtain the SOCKS5 and HTTP proxy addresses, please refer to the official tutorials of the respective projects on how to install clients on Linux. This article is only about how to use acceleration, for technical exchange, and does not involve any magic methods.

Assuming the SOCKS proxy address is 127.0.0.1:1080 and the HTTP proxy address is 127.0.0.1:8118.

Familiarize yourself with how to verify if the proxy is working properly: (cip.cc returns your machine's IP)

# SOCKS proxy
curl -x socks5://127.0.0.1:1080 cip.cc

# HTTP proxy
curl -x 127.0.0.1:10809 cip.cc

For proxies with passwords:

curl -x socks5://Username:[email protected]:10808 cip.cc

SOCKS to HTTP Proxy Conversion#

The reason for this section is that some magic methods do not provide HTTP proxies. They require converting SOCKS to HTTP in order to be used by apt, Docker, and other tools.

If you already have an HTTP proxy, you can skip this section.

privoxy#

Official website: Privoxy - Home Page

Installation:

sudo apt install -y privoxy

Configuration:

sudo vim /etc/privoxy/config

Change the listen-address and forward-socks5t settings, with specific explanations:

listen-address is the local HTTP proxy listening port, which is set to listen on 127.0.0.1:8118 by default for domestic machines.

# For domestic machines, it should be set to only allow local access
listen-address  127.0.0.1:8118
# For home servers, it can be opened up to facilitate use by other machines on the local network
listen-address  0.0.0.0:8118

The port 8118 can be changed freely as long as it is not occupied by other programs.

forward-socks5t is the SOCKS proxy that privoxy listens to. It acts as a bridge, forwarding the requests received on the HTTP port to this SOCKS proxy. Do not forget the . at the end.

forward-socks5t  /  127.0.0.1:1080 .
# With password
forward-socks5  /  Username:[email protected]:1080 .

Restart for the changes to take effect:

sudo systemctl restart privoxy.service
sleep 3
sudo systemctl status privoxy.service

forward-socks5t and forward-socks5 are two different proxy protocols.

forward-socks5t: This is a SOCKS5 proxy protocol that supports transparent transmission. When using this protocol, the proxy server forwards the TCP packets sent by the client to the target server as is, and the responses from the target server are also sent directly back to the client. This achieves transparent proxying, making the connection between the client and the target server completely transparent and imperceptible to both parties.
forward-socks5: This is a regular SOCKS5 proxy protocol. When using this protocol, after the client establishes a connection with the proxy server, the client needs to send specific command requests to the proxy server to establish a connection with the target server. The proxy server relays the data requested by the client to the target server and sends back the response from the target server to the client. In this case, the client is aware that it is communicating through a proxy, and the target server also knows that the request is coming from a proxy server.

Therefore, the main difference lies in the difference between transparent transmission and regular proxying. When it is necessary to hide the presence of the proxy between the client and the target server and make it transparent to both parties, forward-socks5t can be used. If the client and the target server are aware of the presence of the proxy and have a specific interaction protocol for communication, forward-socks5 can be used.

Setting Environment Variables#

First, the principle behind this method is that certain software that supports proxies will check if the proxy is set in the environment variables at runtime. If it is, the software will use the proxy.

In other words, this method does not change the network structure of the machine itself. Whether it works or not depends on whether the software supports it or not.

For software that supports it, there should generally be an option to manually specify the proxy, which is more controllable. For example:

pip3 (not needed if the http_proxy environment variable is set)

pip3 install --proxy="http://127.0.0.1:8118" packageName

Only set two environment variables: HTTP_PROXY and HTTPS_PROXY:

export http_proxy="http://127.0.0.1:8118"   # Proxy for HTTP traffic
export https_proxy="http://127.0.0.1:8118"
export no_proxy=127.0.0.1,localhost,192.168.*   # Networks that do not require proxy

To disable the proxy:

unset http_proxy https_proxy no_proxy

Test if the proxy is working:

curl cip.cc

For proxies with authentication:

export http_proxy="Username:Password@proxy-server-ip:8118"
export https_proxy="Username:Password@proxy-server-ip:8118"

The above commands are not very convenient to use, so you can use aliases to simplify the commands.

vim ~/.bashrc

Add the following (remember to modify the proxy address):

alias ftz='export http_proxy="127.0.0.1:8118" https_proxy="127.0.0.1:8118" no_proxy=127.0.0.1,localhost,192.168.*'
alias ctz='unset http_proxy https_proxy no_proxy'

Make the configuration take effect immediately:

source ~/.bashrc

After connecting to the server, in the terminal:

Enter ftz (enable proxy) to set the proxy for the terminal
Enter ctz (disable proxy) to disable the proxy for the terminal

In addition, after using ftz, the proxy will only take effect in the current terminal. If you exit and log in again, there will be no proxy. This provides better control.

Apt Proxy#

Apt does not use the HTTP Proxy environment variable. It uses a separate configuration file.

Create a new configuration file in /etc/apt/apt.conf.d/:

sudo vim /etc/apt/apt.conf.d/proxy.conf

Acquire {
  HTTP::proxy "http://127.0.0.1:8118";
  HTTPS::proxy "http://127.0.0.1:8118";
}

Verify:

sudo apt update

Docker Image Proxy#

Because the pulling and management of images are handled by the Docker daemon, which is managed by systemd, the Docker daemon needs to be configured through systemd to set up the proxy.

Official documentation: Configure the daemon with systemd | Docker Docs

Create the systemd directory for dockerd. The configurations in this .d directory will override the default configurations.

sudo mkdir -p /etc/systemd/system/docker.service.d

Create the configuration file http-proxy.conf

sudo vim /etc/systemd/system/docker.service.d/proxy.conf

Add the configuration, similar to the previous configurations

[Service]
Environment="HTTP_PROXY=http://127.0.0.1:8118/"
Environment="HTTPS_PROXY=http://127.0.0.1:8118/"
Environment="NO_PROXY=127.0.0.1,localhost,192.168.*,*.example.com"
# If `NO_PROXY=*`, then all requests will not go through the proxy server

Reload the configuration file and restart Dockerd for the changes to take effect

sudo systemctl daemon-reload
sudo systemctl restart docker

Check to confirm that the environment variables are correctly configured:

sudo systemctl show --property=Environment docker

You can now try pulling images to see if the acceleration effect is achieved.

Docker Container Proxy#

During the container runtime, if you need to access the internet through a proxy, you just need to add the environment variables. For example, if you are using docker-compose, you can add the following three parts to the environment variables in the configuration file.

    environment:
        - http_proxy="192.168.1.11:10809"
        - https_proxy="192.168.1.11:10809"
        - no_proxy="localhost,127.0.0.1,.example.com"

The principle behind this is the so-called "setting environment variables" mentioned earlier. Therefore, whether it works or not depends on whether the services running inside the container actively fetch the environment variables.

This method doesn't seem to be very useful, as it can be used to solve the problem of Emby not being able to scrape in China, but this global variable may affect other functions.

If the container already uses a proxy by default, you can also configure ~/.docker/config.json.

{
 "proxies":
 {
   "default":
   {
     "httpProxy": "http://proxy.example.com:8080",
     "httpsProxy": "http://proxy.example.com:8080",
     "noProxy": "localhost,127.0.0.1,.example.com"
   }
 }
}

This is a user-level configuration file that includes proxies as well as docker login and other related information. It can also configure the format of information display, plugin parameters, etc.

Note: Whether it is docker run or docker build, the default is network isolation. If the proxy used is localhost:3128 or similar, it will not work. This type of proxy is limited to local use and must be used with --network host to function properly.

Docker Build Proxy#

Although the essence of docker build is to start a container, the environment is slightly different and user-level configurations do not work. During the build process, http_proxy and other parameters need to be injected.

docker build . \
    --build-arg "HTTP_PROXY=http://proxy.example.com:8080/" \
    --build-arg "HTTPS_PROXY=http://proxy.example.com:8080/" \
    --build-arg "NO_PROXY=localhost,127.0.0.1,.example.com" \
    -t your/image:tag

Original article: https://technique.vfly2.com/2023/11/linux-proxy-acceleration-for-apt-docker/

Copyright Notice: All articles on this blog, unless otherwise noted, are original works by AhFei and are licensed under the CC BY-NC-SA 4.0 license. Please indicate the source as Technique (technique.vfly2.com).

Stay updated ٩(•̤̀ᵕ•̤́๑)ᵒᵏᵎᵎᵎᵎ with clear and beneficial practical skills. Feel free to subscribe using RSS, or follow @[email protected] on platforms that support ActivityPub to receive push notifications for new articles.