Understanding Docker's -net=host Option

February 27, 2020

Introduction

Containers are now first class citizens in any development cycle. It is essential for us to understand how container networking works. This is not only important from the perspective of service communication but also forms an important aspect of infrastructure security.

In this post we will learn briefly about various networking modes available for Docker containers and deep dive into Host Mode networking.

Overview of Networking Modes

None

None is straightforward in that the container receives a network stack, but lacks an external network interface. It does, however, receive a loopback interface. Both the rkt and Docker container projects provide similar behavior when None or Null networking is used. This mode of container networking has a number of uses including testing containers, staging a container for a later network connection, and being assigned to containers with no need for external communication.

Bridge Mode

A Linux bridge provides a host internal network in which containers on the same host may communicate, but the IP addresses assigned to each container are not accessible from outside the host. Bridge networking leverages iptables for NAT and port-mapping, which provide single-host networking. Bridge networking is the default Docker network type (i.e., docker0), where one end of a virtual network interface pair is connected between the bridge and the container.

Here’s an example of the creation flow:

  1. A bridge is provisioned on the host.
  2. A namespace for each container is provisioned inside that bridge.
  3. Containers’ ethX are mapped to private bridge interfaces.
  4. Iptables with NAT are used to map between each private container and the host’s public interface.

NAT is used to provide communication beyond the host. While bridged networks solve port-conflict problems and provide network isolation to containers running on one host, there’s a performance cost related to using NAT.

Host Mode

In this approach, a newly created container shares its network namespace with the host, providing higher performance — near metal speed — and eliminating the need for NAT; however, it does suffer port conflicts. While the container has access to all of the host’s network interfaces, unless deployed in privilege mode, the container may not reconfigure the host’s network stack.

Host networking is the default type used within Mesos. In other words, if the framework does not specify a network type, a new network namespace will not be associated with the container, but with the host network. Sometimes referred to as native networking, host networking is conceptually simple, making it easier to understand, troubleshoot and use.

Overlay

Overlays use networking tunnels to deliver communication across hosts. This allows containers to behave as if they are on the same machine by tunneling network subnets from one host to the next; in essence, spanning one network across multiple hosts. Many tunneling technologies exist, such as virtual extensible local area network (VXLAN).

VXLAN has been the tunneling technology of choice for Docker libnetwork, whose multi-host networking entered as a native capability in the 1.9 release. With the introduction of this capability, Docker chose to leverage HashiCorp’s Serf as the gossip protocol, selected for its efficiency in neighbor table exchange and convergence times.

For those needing support for other tunneling technologies, Flannel may be the way to go. It supports udp, vxlan, host-gw, aws-vpc or gce. Each of the cloud provider tunnel types creates routes in the provider’s routing tables, just for your account or virtual private cloud (VPC). The support for public clouds is particularly key for overlay drivers given that among others, overlays best address hybrid cloud use cases and provide scaling and redundancy without having to open public ports.

Multi-host networking requires additional parameters when launching the Docker daemon, as well as a key-value store. Some overlays rely on a distributed key-value store. If you’re doing container orchestration, you’ll already have a distributed key-value store lying around.

Overlays focus on the cross-host communication challenge. Containers on the same host that are connected to two different overlay networks are not able to communicate with each other via the local bridge — they are segmented from one another.

Underlay

Underlay network drivers expose host interfaces (i.e., the physical network interface at eth0) directly to containers or VMs running on the host. Two such underlay drivers are media access control virtual local area network (MACvlan) and internet protocol VLAN (IPvlan). The operation of and the behavior of MACvlan and IPvlan drivers are very familiar to network engineers. Both network drivers are conceptually simpler than bridge networking, remove the need for port-mapping and are more efficient. Moreover, IPvlan has an L3 mode that resonates well with many network engineers. Given the restrictions — or lack of capabilities — in most public clouds, underlays are particularly useful when you have on-premises workloads, security concerns, traffic priorities or compliance to deal with, making them ideal for brownfield use. Instead of needing one bridge per VLAN, underlay networking allows for one VLAN per subinterface.

Now let’s focus on host mode networking.

Understanding Host Mode Networking in Detail

Host mode seems pretty straight forward but there are some items you need to keep in mind when deploying it. Let’s look at an example so you can see what we are talking about. First, let’s start a basic web container on a host.

docker run -d --name=web1 --net=host vaibhavthakur/docker:webinstance1

Note that I’m passing the ‘–net=host’ flag in the docker run command. Also note that I’m not specifying any port mappings. Once the image is downloaded docker will run the image as a container called ‘web’. Since we told docker to run this container as a daemon let’s connect to a bash shell on the container.

docker exec -it web1 /bin/bash

This should drop you in a shell inside the container and now you should check all the network adapters available to the container. You can do so as shown below.

<p>CODE:https://gist.github.com/denshirenji/0de291836a3ef4d01507703afca7507a.js</p>

Now, you should exit the container and run the same command on the host.

<p>CODE:https://gist.github.com/denshirenji/4f45a912538c3e553abefb1c3a723f56.js</p>

It can be seen that the output of the command is exactly the same. This means you should be able to access the container using host ip. Please keep in mind that a firewall rule to accept on port 80 is enabled on the machine if you are using a cloud provider.

Let’s try to access the service on the instance’s public ip.

We can see the response from service running inside web1 container.


You should note that we did not provide any port mapping while running the container and explicitly specified host node networking and thus can access the apache server running inside the container on port 80.

Now let’s try another thing. We will try to run another container of the same service using host mode networking on this host and see what happens.

docker run -it --name web2 --net=host vaibhavthakur/docker:webinstance2

If you notice carefully, the container did not even start. Let’s look at the logs.

<p>CODE:https://gist.github.com/denshirenji/54b48d284362cec2b21bf5aefe47916e.js</p>

So, this clearly shows that the container could not come up since the entrypoint process died quickly. Let's dig a little deeper and get inside the container.

<p>CODE:https://gist.github.com/denshirenji/0007652bd6f66db00673a81bb947029e.js</p>

We first checked the status of httpd service and it showed dead. Then we tried to start it but it failed again because it could not bind to port 80. Let’s do one more test and check the status of port 80.

<p>CODE:https://gist.github.com/denshirenji/167b259b12674149d2a30703a1b044cc.js</p>

It can be clearly seen that port 80 is occupied by some process, and this process is our container named web1. Therefore, it is safe to conclude that web2 container tried to bind to port 80 on the same host on which web1 is running and thus it failed. Now, let’s kill web1 container and try to start the httpd service in web2 container.

After killing web1, we tried to start httpd on web2 and it was successful.

<p>CODE:https://gist.github.com/denshirenji/af4e339c1e570e65f7f47c1db4abb3d9.js</p>

One final test, let’s access the instance on port 80.

We can see the response from service running inside web2 container.

Conclusion

It is important to understand that host mode networking provides better network performance for containers since the network traffic need not traverse docker0 bridge and iptables port mappings. I hope this article clearly explains what host mode networking is and how it is different from other networking options. Feel free to reach out should you have any questions around it.

If you're looking to monitor your containers, it can be done with Hosted Prometheus through MetricFire. Sign up for a free trial here, or book a demo and talk to us directly. If you're looking to learn more check out our three-part series on Docker Networking, featuring a deep dive into many kinds of docker networking.

Related Posts

GET FREE MONITORING FOR 14 DAYS