Detailed explanation of Docker container cross-host multi-network segment communication solution

1. MacVlan

There are many solutions to achieve cross-host network communication of Docker, such as the deployment of Consul service to achieve cross-host communication of Docker containers in the previous blog post.

Macvlan working principle:

Macvlan is a network interface supported by the Linux kernel. Required Linux builds are v3.9–3.19 and 4.0+;
By creating a Macvlan sub-interface for a physical network card, a physical network card is allowed to have multiple independent MAC addresses and IP addresses. The virtualized sub-interface will be directly exposed to the adjacent physical network. From the outside, it looks like the network cable is divided into multiple strands and connected to different hosts respectively;
After the physical network card receives the packet, it will determine whether the packet needs to be handed over to the virtual network card based on the destination MAC address of the received packet.

When the container needs to be directly connected to the physical network, Macvlan can be used. Macvlan itself does not create a network. In essence, it first makes the host physical network card work in 'promiscuous mode', so that the MAC address of the physical network card will be invalid, and the physical network card can receive all traffic in the layer 2 network. The next step is to create a virtual network card on this physical network card and assign a MAC address to the virtual network card to achieve multiple uses of one card. From the perspective of the physical network, each virtual network card is a separate interface.

Note when using Macvlan:

The container is directly connected to the physical network, which is responsible for allocating IP addresses. This may result in the exhaustion of physical network IP addresses. Another consequence is network performance issues. As more hosts are connected to the physical network, the proportion of broadcast packets increases rapidly, causing network performance to degrade.
A network on the host needs to work in 'promiscuous mode';
As mentioned above, the MAC address of a physical network card working in promiscuous mode will become invalid. Therefore, containers running in this mode cannot communicate with the external network, but it will not affect the communication between the host and the external network.
In the long run, bridge networks and overlay networks are better choices because virtual networks should be isolated from physical networks rather than shared.

Working diagram:

2. Configuration Example

Example 1 (implementing single-segment cross-host communication in containers based on macvlan)

Result:

Two centos 7.3 machines, each running a docker service;
The two Docker servers create the same MacVlan network, so that the containers on the Docker servers can communicate across hosts.

Start configuration

1. The first docker server is configured as follows

[root@docker01 ~]# ip link set ens33 promisc on # Enable promiscuous mode for ens33 network card [root@docker01 ~]# ip link show ens33 # Make sure the viewed information contains the following words in red2: ens33: <BROADCAST,MULTICAST,'PROMISC',UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
  link/ether 00:0c:29:9f:33:9f brd ff:ff:ff:ff:ff:ff
[root@docker01 ~]# docker network create -d macvlan --subnet 172.22.16.0/24 --gateway 172.22.16.1 -o pa
rent=ens33 mac_net1
#Create a macvlan network, specify the gateway, network segment and other information, "-o" specifies which network card to bind to [root@docker01 ~]# docker run -itd --name test1 --ip 172.22.16.10 --network mac_net1 busybox # Run a container based on the newly created macvlan network and specify its IP

Confirm the IP address of the running container

[root@docker01 ~]# docker exec test1 ip a # Check the IP and make sure the following red ones are the same as configured: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
  link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  inet 127.0.0.1/8 scope host lo
    valid_lft forever preferred_lft forever
6: eth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
  link/ether 02:42:ac:16:10:0a brd ff:ff:ff:ff:ff:ff
  inet '172.22.16.10/24' brd 172.22.16.255 scope global eth0
    valid_lft forever preferred_lft forever

2. The second docker server is configured as follows (basically similar to the first docker server)

[root@docker02 ~]# ip link set ens33 promisc on # Enable promiscuous mode [root@docker02 ~]# ip link show ens33 
2: ens33: <BROADCAST,MULTICAST,'PROMISC',UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
  link/ether 00:0c:29:b5:bc:ed brd ff:ff:ff:ff:ff:ff
[root@docker02 ~]# docker network create -d macvlan --subnet 172.22.16.0/24 --gateway=172.22.16.1 -o parent=ens33 mac_net1
#Create a macvlan with the same network segment and gateway as the first docker server. And bind it to the physical network card.
#In order to visually see that the macvlan on other docker servers is in the same network segment as this one. Therefore, it is recommended to set the network name to be the same.
[root@docker02 ~]# docker run -itd --name test2 --ip 172.22.16.11 --network mac_net1 busybox
#Run a container and specify it is based on the macvlan network #Note that its IP address should not conflict with the container IP address on other docker servers

Confirm the IP address of the running container

[root@docker02 ~]# docker exec test2 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
  link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  inet 127.0.0.1/8 scope host lo
    valid_lft forever preferred_lft forever
6: eth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
  link/ether 02:42:ac:16:10:0b brd ff:ff:ff:ff:ff:ff
  inet '172.22.16.11/24' brd 172.22.16.255 scope global eth0
    valid_lft forever preferred_lft forever

Use container test2 on the second docker server to ping container test1 on the first docker server

OK, cross-host container communication is achieved through macvlan. Since using promiscuous mode will cause the MAC address of the physical network card to become invalid, the container cannot communicate with the external network through this mode.

Example 2 (Solution for multi-segment cross-host network based on macvlan)

The achieved effects are as follows:

Two centos 7.3 machines, each running a docker service;
Each host creates two MacVlan segments for containers to use (172.10.16.0/24 and 172.20.16.0/24);
The first docker server runs containers test1 and test2, and the second docker server runs containers test3 and test4.
Finally, containers in the same network segment across hosts can communicate with each other.

Start configuration:

1. The first docker server is configured as follows

[root@docker01 ~]# ip link set ens33 promisc on # Enable promiscuous mode of ens33 network card #That is, enable multiple virtual interfaces of the network card 
[root@docker01 ~]# ip link show ens33 # Make sure the information you view contains the following words in red2: ens33: <BROADCAST,MULTICAST,'PROMISC',UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
  link/ether 00:0c:29:9f:33:9f brd ff:ff:ff:ff:ff:ff
[root@docker01 ~]# modinfo 8021q
 # Check whether the 8021q module is loaded. If there is information returned, it means that the module has been loaded

The information returned by the modinfo 8021q command is as follows

[root@docker01 ~]# modprobe 8021q #If the 8021q module is not loaded, execute this command [root@docker01 ~]# cd /etc/sysconfig/network-scripts/
[root@docker01 network-scripts]# vim ifcfg-ens33 
           ................
BOOTPROTO=manual # Change this configuration item to "manual", which also means manual [root@docker01 network-scripts]# cp -p ifcfg-ens33 ifcfg-ens33.10 # Copy a network card configuration file, -p retains the properties of the original file [root@docker01 network-scripts]# vim ifcfg-ens33.10 
BOOTPROTO=none 
NAME=ens33.10 #Note the name change DEVICE=ens33.10 #Note the name change ONBOOT=yes
IPADDR=192.168.10.11 # Set an IP for the virtual network card
PREFIX=24
GATEWAY=192.168.10.2
VLAN=yes
[root@docker01 network-scripts]# cp ifcfg-ens33.10 ifcfg-ens33.20 
[root@docker01 network-scripts]# vim ifcfg-ens33.20 
BOOTPROTO=none
NAME=ens33.20
DEVICE=ens33.20
ONBOOT=yes
IPADDR=192.168.20.10 # Note that the IP here is not in the same network segment as ens33.10 PREFIX=24
GATEWAY=192.168.20.2
VLAN=yes
[root@docker01 network-scripts]# ifdown ens33;ifup ens33 #Restart the network card to make the changes take effect[root@docker01 network-scripts]# ifup ens33.10 # Start the network card[root@docker01 network-scripts]# ifup ens33.20 # Start[root@docker01 ~]# docker network create -d macvlan --subnet 172.10.16.0/24 --gateway 172.10.16.1 -o parent=ens33.10 mac_net10
#Create a macvlan network, define a network segment, gateway and bind it to ens33.10
[root@docker01 ~]# docker network create -d macvlan --subnet 172.20.16.0/24 --gateway 172.20.16.1 -o parent=ens33.20 mac_net20
#Create a macvlan network, define a network segment, gateway and bind it to ens33.20
#Next, run a container based on the macvlan network just created

2. The second docker server is configured as follows (basically similar to the first operation, pay attention to the IP not to conflict)

[root@docker02 ~]# ip link set ens33 promisc on # Enable promiscuous mode [root@docker02 ~]# ip link show ens33 
2: ens33: <BROADCAST,MULTICAST,'PROMISC',UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
  link/ether 00:0c:29:b5:bc:ed brd ff:ff:ff:ff:ff:ff
[root@docker02 ~]# modinfo 8021q
Please refer to Figure 1 for the returned information

[root@docker02 ~]# modprobe 8021q #If the 8021q module is not loaded, execute this command [root@docker02 ~]# cd /etc/sysconfig/network-scripts/
[root@docker02 network-scripts]# vim ifcfg-ens33 
      ...............#Omit some content BOOTPROTO=manual    
[root@docker02 network-scripts]# scp [email protected]:/etc/sysconfig/network-scripts/ifcfg-ens33.* . # Pay attention to the trailing “.”
ifcfg-ens33.10 100% 128 83.4KB/s 00:00  
ifcfg-ens33.20 100% 124 75.0KB/s 00:00 
[root@docker02 network-scripts]# vim ifcfg-ens33.10 
BOOTPROTO=none
NAME=ens33.10
DEVICE=ens33.10
ONBOOT=yes
IPADDR=192.168.10.11 # Change the IP to avoid conflict with the first one PREFIX=24
GATEWAY=192.168.10.2
VLAN=yes
[root@docker02 network-scripts]# vim ifcfg-ens33.20 

BOOTPROTO=none
NAME=ens33.20
DEVICE=ens33.20
ONBOOT=yes
IPADDR=192.168.20.11
PREFIX=24
GATEWAY=192.168.20.2
VLAN=yes
[root@docker02 network-scripts]# ifdown ens33;ifup ens33 # Restart the network card to make the configuration effective [root@docker02 network-scripts]# ifup ens33.10 # Start the network card [root@docker02 network-scripts]# ifup ens33.20
#Next, create a macvlan network, the same as the network created by the first docker server [root@docker02 ~]# docker network create -d macvlan --subnet 172.10.16.0/24 --gateway 172.10.16.1 -o parent=ens33.10 mac_net10
[root@docker02 ~]# docker network create -d macvlan --subnet 172.20.16.0/24 --gateway 172.20.16.1 -o parent=ens33.20 mac_net20 
[root@docker02 ~]# docker run -itd --name test3 --network mac_net10 --ip 172.10.16.11 busybox
[root@docker02 ~]# docker run -itd --name test4 --network mac_net20 --ip 172.20.16.21 busybox

After this configuration, you can perform a ping test. If the configuration is correct, test3 should be able to communicate with test1 (because they are both based on the mac_net10 network); test4 should be able to communicate with test2 (similarly).

But test3 and test1 cannot communicate with test4 and test2 (because they are not based on the same virtual LAN).

Container test3 pings container test1 test (Note: If you use a vmware virtual machine for testing, due to the characteristics of vmware, you need to change its network adapter to "bridge mode" instead of NAT mode, etc. Otherwise, communication will not be possible)

Container test4 pings container test2 test:

At this point, cross-host network multi-segment has been implemented. Similarly, each container cannot communicate with the external network. If you have patience, it is recommended to read the official Docker documentation

This is the end of this article about the Docker container cross-host multi-segment communication solution. For more relevant Docker container cross-host multi-segment communication content, please search 123WORDPRESS.COM's previous articles or continue to browse the following related articles. I hope everyone will support 123WORDPRESS.COM in the future!

You may also be interested in:

Operations of communication between Docker containers and external network communication
Implementation of Docker container connection and communication
Detailed explanation of direct routing in cross-host communication of Docker containers
Detailed explanation of how Docker containers communicate across hosts
Detailed explanation of a method of communication between docker containers
How to implement communication between Docker containers

<<: Solution to MySQL root password error number 1045

>>: Encapsulate the navigation bar component with Vue

Solution to ERROR 1366 when entering Chinese in MySQL

The National Day is coming. Use JS to implement a small tool to generate a National Day style avatar. Detailed explanation of the implementation process

Table of contents 1. Page Layout 2. Image upload ...

Implementing a table scrolling carousel effect through CSS animation

An application of CSS animation, with the same co...

Detailed explanation of Docker container cross-host multi-network segment communication solution

Solution to ERROR 1366 when entering Chinese in MySQL

Docker network mode and configuration method

Solve the problem of secure_file_priv null

js dynamically implements table addition and deletion operations

Nginx/Httpd reverse proxy tomcat configuration tutorial

Teach you how to deploy zabbix service on saltstack

Website Design Experience Summary of Common Mistakes in Website Construction

Detailed use cases of vue3 teleport

Pure CSS implementation (no script) Html command-style tooltip text prompt effect

Two types of tab applications in web design

Recommend

Common array operations in JavaScript

Correct modification steps for Docker's default network segment

Webpack loads css files and its configuration method

Docker file storage path, get container startup command operation

CSS border adds four corners implementation code

A small problem about null values in MySQL

How to install suPHP for PHP5 on CentOS 7 (Peng Ge)

Java+Tomcat environment deployment and installation process diagram

The specific use and difference between attribute and property in Vue

Install Kafka in Linux

Div exceeds hidden text and hides the CSS code beyond the div part

Complete steps to use mock.js in Vue project

Practical example of Vue virtual list

The National Day is coming. Use JS to implement a small tool to generate a National Day style avatar. Detailed explanation of the implementation process

Implementing a table scrolling carousel effect through CSS animation