搜云库技术团队
前文我们聊了下haproxy的访问控制ACL的配置，回顾请参考https://tech.souyunku.com/qiuhom-1874/p/12817773.html；今天我们来聊一聊haproxy的https的配置、基于tcp四层负载均衡以及访问控制的配置说明；
首先我们来看一下haproxy的https的配置；https是什么我这里就不过多阐述了，有关证书的申请相关说明请参考https://tech.souyunku.com/qiuhom-1874/p/12237944.html；在haproxy的配置文件中，我们要明确的声明监听某个端口，该端口需要ssl协议来访问，类似nginx里的listen 443 ssl配置；除此之外我们还需要用crt来指定证书；不同于nginx里的配置是haproxy的证书内容包含私钥信息；所以在我们申请好证书后，还需要把证书文件内容同私钥文件做合并；
示例：配置haproxy监听443端口，并支持https
提示：红框中的内容表示监听443端口，并明确指定使用ssl协议访问；证书文件是/etc/haproxy/ssl/haproxy.pem
创建ssl目录和证书文件
[root@docker_node1 ~]# mkdir -p /etc/haproxy/ssl
[root@docker_node1 ~]# cd /etc/haproxy/ssl
[root@docker_node1 ssl]# ls
[root@docker_node1 ssl]# (umask 066;openssl genrsa -out haproxy.key 2048)
Generating RSA private key, 2048 bit long modulus
.......+++
...+++
e is 65537 (0x10001)
[root@docker_node1 ssl]# ls
haproxy.key
[root@docker_node1 ssl]# openssl req -new -x509 -key haproxy.key -out haproxy.crt -subj "/CN=www.test.com"
[root@docker_node1 ssl]# ls
haproxy.crt haproxy.key
[root@docker_node1 ssl]# cat haproxy.crt haproxy.key > haproxy.pem
[root@docker_node1 ssl]# ls
haproxy.crt haproxy.key haproxy.pem
[root@docker_node1 ssl]# openssl x509 -in haproxy.pem -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
e6:c5:30:f9:10:e5:da:cf
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=www.test.com
Validity
Not Before: May 2 14:41:50 2020 GMT
Not After : Jun 1 14:41:50 2020 GMT
Subject: CN=www.test.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:e6:bd:db:2d:37:1e:b3:26:21:45:37:c3:bc:c2:
f8:f0:35:31:7c:af:f4:22:be:bc:f2:01:a1:ce:31:
32:91:37:06:14:a8:eb:ec:99:30:ae:8f:66:6b:51:
03:94:bb:d1:ae:7d:15:23:ea:9a:83:74:6d:d0:be:
30:5d:bf:4c:dd:79:c6:8d:51:01:ea:d7:a8:e5:93:
f9:11:d9:75:cc:2c:65:d6:31:db:15:20:7f:5e:9f:
56:15:3e:17:b9:82:a8:25:ba:40:17:1c:ef:f2:fc:
11:cb:72:ce:07:5b:57:5d:c3:f1:f6:42:1f:02:63:
b0:33:de:87:a3:43:c7:a0:1b:03:c4:ab:09:f4:67:
ce:ec:ef:1f:88:7a:c2:aa:ca:41:ed:1d:78:60:b1:
41:6d:b9:46:67:1e:c9:ce:43:d8:d6:36:be:ba:ed:
d9:2b:01:9a:0c:8e:64:59:07:3a:c2:2c:55:2b:9b:
06:e5:5b:c7:ba:e2:9f:ea:be:71:9f:76:da:79:c6:
f0:b8:94:70:c0:4f:7b:57:53:5c:30:71:c5:82:d0:
7a:a8:d0:71:32:2c:f0:38:61:69:9b:9e:ac:da:45:
74:63:25:76:78:91:c3:be:6c:7c:72:1c:54:4a:ac:
10:24:45:9c:89:6a:1f:5b:00:22:3a:b6:fe:3f:b8:
73:7b
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:
B2:EC:53:AE:68:E0:65:1D:E6:DB:C8:16:E3:BA:D3:70:BC:E7:79:81
X509v3 Authority Key Identifier:
keyid:B2:EC:53:AE:68:E0:65:1D:E6:DB:C8:16:E3:BA:D3:70:BC:E7:79:81
X509v3 Basic Constraints:
CA:TRUE
Signature Algorithm: sha256WithRSAEncryption
91:a2:36:9b:b8:d3:fd:89:ea:6c:b4:b9:3c:b6:a1:f3:86:87:
71:13:8b:10:a0:73:e9:96:b7:1b:dd:7e:91:64:41:a4:c3:80:
7c:b6:f2:ce:a7:77:28:74:51:da:a6:52:98:a7:88:48:41:2f:
ad:3d:cb:36:bd:94:f9:27:72:f1:e8:ae:f8:bd:38:2e:d6:ea:
0b:26:2d:8b:49:3e:bc:4e:58:9e:bf:79:99:5b:88:23:da:98:
e4:45:79:9c:d2:c9:05:05:9f:23:8a:0f:38:db:9a:c4:5c:e9:
a0:f5:e5:aa:02:bc:df:a7:5d:11:cd:35:08:cc:41:08:40:b6:
0b:e4:ad:79:df:9a:1d:7b:70:8b:65:e5:dc:85:34:55:5c:f2:
94:dc:07:91:43:d0:bb:b4:fb:31:b9:74:19:7a:69:43:11:70:
41:41:86:93:ad:83:42:62:e2:67:69:38:cd:18:c7:e9:f7:f4:
be:78:22:ea:ee:20:db:27:1f:06:87:4c:51:67:19:0a:64:97:
3a:e6:2c:32:bd:84:91:88:96:d2:01:e5:c3:62:59:11:c8:20:
7d:a1:c8:5a:3d:8a:fc:f5:14:fe:41:15:97:ee:47:ec:e5:19:
49:0b:c1:8a:c9:3f:10:4d:66:bf:d5:01:21:2d:fd:8b:a7:95:
17:08:7e:46
[root@docker_node1 ssl]#
提示：我这里就直接用-subj来指定证书/CN来生成自签名证书；haproxy的证书要求是把证书信息和私钥信息放在一起，所以我们需要把证书文件信息和私钥新信息通过重定向的方式合并在一起；
测试：重启haproxy,用浏览器访问
提示：可以看到我们用https访问是可以正常的访问，只是证书浏览器不信任，所以会提示我们不安全；
把80端口的请求重向定443
提示：红框中的配置表示把http请求重定向为https，这个类似nginx里的重写url；
测试：用浏览器访问http://192.168.0.22看看是否会把http重写为https?
提示：可以看到我们访问用http去访问，在响应报文中会用location告诉浏览器去访问https://192.168.0.22；这样一来就实现了全站https；
向后端传递用户请求的协议和端口（frontend或backend）
提示：红框中的内容就表示向后端请求报文中设置X-Forwarded-port首部的值为dst_port变量的值；添加X-Forwared-Proto首部，其值为https，在请求协议是https的请求时；通常这两个设置是后端应用server要求使用https访问时需要把前端https请求通过X-Forwared-Proto传递给后端；
测试：在后端server上配置日志格式，分别将这两个首部记录到日志，然后通过访问，查看日志中记录这两个首部的值是否是我们访问的443端口和https协议？
提示：以上定义httpd的日志格式，分别记录{X-Forwarded-Port和X-Forwared-Proto首部的值
用浏览器访问测试，看看日志中记录的信息
提示：可以看到日志中把对应访问端口和协议都传给了后端server；
haproxy基于4层代理做tcp负载均衡
示例：基于tcp对mysql做负载均衡
提示：以上配置表示用tcp协议来代理后端server；
后端server环境准备
[root@docker_node1 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
wordpress latest c3fa1c8546fb 44 hours ago 540MB
mysql 5.7 f965319e89de 4 days ago 448MB
httpd 2.4.37-alpine dfd436f9a5d8 16 months ago 91.8MB
[root@docker_node1 ~]# docker run --name db1 -d --net bridge -e MYSQL_ROOT_PASSWORD=admin mysql:5.7
8cade03e4f28f40e7d7e970355735c8b881892754b18a637fef0d956f3f88877
[root@docker_node1 ~]# docker run --name db2 -d --net bridge -e MYSQL_ROOT_PASSWORD=admin mysql:5.7
1bff841c226abe88441103764655dc86257d5b99079eaa113384fd04b62cf0f8
[root@docker_node1 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1bff841c226a mysql:5.7 "docker-entrypoint.s…" 7 seconds ago Up 6 seconds 3306/tcp, 33060/tcp db2
8cade03e4f28 mysql:5.7 "docker-entrypoint.s…" 12 seconds ago Up 12 seconds 3306/tcp, 33060/tcp db1
3572c621f827 wordpress "docker-entrypoint.s…" 9 hours ago Up 9 hours 80/tcp wordpress
f07288d607e5 httpd:2.4.37-alpine "httpd-foreground" 7 days ago Up 13 hours 80/tcp web3
971595b7f409 httpd:2.4.37-alpine "httpd-foreground" 7 days ago Up 13 hours 80/tcp web2
5c74f3be1868 httpd:2.4.37-alpine "httpd-foreground" 7 days ago Up 13 hours 80/tcp web1
[root@docker_node1 ~]# docker exec -it db1 /bin/sh
# mysql -padmin
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.30 MySQL Community Server (GPL)
Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> grant all on *.* to "myuser"@'172.17.0.%' identified by 'admin';
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> \q
Bye
# exit
[root@docker_node1 ~]# docker exec -it db2 /bin/sh
# mysql -padmin
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.30 MySQL Community Server (GPL)
Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> grant all on *.* to "myuser"@'172.17.0.%' identified by 'admin';
Query OK, 0 rows affected, 1 warning (0.01 sec)
mysql> \q
Bye
# exit
[root@docker_node1 ~]#
提示：启动mysql容器时，我们需要对root设置密码；新建用户时，我们指定的主机IP地址应该是docker0桥的地址，虽然是基于4层tcp代理，但它还是要更改源ip；
测试：重启haproxy，用mysql客户端访问haproxy所监听的3306端口，看看是否能够正常的连接到后端mysql容器里？
提示：可以看到用别的主机上的mysql客户端是可以正常访问haproxy监听的3306端口；
haproxy基于4层tcp做访问控制
提示：tcp-request connection 表示根据第4层条件对传入连接执行操作；以上配置表示拒绝源地址为192.168.0.21的连接
测试：重启haproxy，用192.168.0.21上的mysql客户端连接192.168.0.22：3306看看是否能够正常连接？
提示：可以看到在192.168.0.21上用mysql工具就不能够连接192.168.0.22上的mysql了；
示例：仅开放源地址为192.168.0.21的主机访问，拒绝其他主机的访问
提示：tcp-request connection 允许访问用accept来标识，拒绝用reject；如果是联合使用后面范围大的可以不用写条件，表示除上面匹配到的ACL条目以外的所有条目；同http-request 类似，不同的是http-request允许是用allow标识，拒绝用deny标识；
测试：分别用源地址为192.168.0.21的主机上的mysql客户端去连接192.168.0.22上的mysql和源地址非192.168.0.21上的mysql客户端连接192.168.0.22上的mysql看看是否都可以连接？
提示：可以看到在源地址非192.168.0.21上使用mysql客户端是不能够连接到后端mysql容器的；这是因为它没有被我们定义ACL匹配，所以就会被tcp-request connect reject匹配；从而拒绝连接；
最后一个话题，在前边的博客中我们聊的了haproxy的状态页的配置；其中我们演示的是通过页面展示给我们，通过鼠标点点点去管理后端server；这样一来在对于我们想要监控haproxy本身以及后端server就是一个难题；聪明的你一定注意到haproxy的全局配置段中有一个stat socket /var/lib/haproxy/stats这样的配置；我们接下来说说这个配置是干嘛用的；
提示：以上配置是表示将监控页面信息绑定到一个socket文件上；我们可以通过对该socket文件发送特定指令，实现操作haproxy的目的
示例：通过socat向/var/lib/haproxy/stats传递help信息，让其打印帮助页面
[root@docker_node1 ~]# echo "help"|socat stdio /var/lib/haproxy/stats
Unknown command. Please enter one of the following commands only :
clear counters : clear max statistics counters (add 'all' for all counters)
clear table : remove an entry from a table
help : this message
prompt : toggle interactive mode with prompt
quit : disconnect
show info : report information about the running process
show pools : report information about the memory pools usage
show stat : report counters for each proxy and server
show errors : report last request and response errors for each proxy
show sess [id] : report the list of current sessions or dump this session
show table [id]: report table usage stats or dump this table's contents
get weight : report a server's current weight
set weight : change a server's weight
set server : change a server's state or weight
set table [id] : update or create a table entry's data
set timeout : change a timeout setting
set maxconn : change a maxconn setting
set rate-limit : change a rate limiting value
disable : put a server or frontend in maintenance mode
enable : re-enable a server or frontend which is in maintenance mode
shutdown : kill a session or a frontend (eg:to release listening ports)
show acl [id] : report available acls or dump an acl's contents
get acl : reports the patterns matching a sample for an ACL
add acl : add acl entry
del acl : delete acl entry
clear acl <id> : clear the content of this acl
show map [id] : report available maps or dump a map's contents
get map : reports the keys and values matching a sample for a map
set map : modify map entry
add map : add map entry
del map : delete map entry
clear map <id> : clear the content of this map
set ssl <stmt> : set statement for ssl
[root@docker_node1 ~]#
提示：从上面的帮助信息，我们可以了解到我们可以通过管道把对应指令传给/var/lib/haproxy/stats文件，从而实现管理后端server的目的；
示例：通过socat命令把web1标记为disable
提示：我们执行把web1标记为disable，系统提示我们没有权限；接下来我们来配置权限即可
提示：以上配置表示该socket文件具有admin权限，接下来在来把web1标记为disable（修改haproxy配置文件，需要重启haproxy ）
提示：我们修改了权限以后，现在执行以上命令就没有报什么权限之类的错误了，接下来我们打开状态页，看看web1的状态是什么
提示：可以看到web1变成了维护的模式；
示例：列出监控页面指标数据的信息
[root@docker_node1 ~]# echo "show info " |socat stdio /var/lib/haproxy/stats
Name: HAProxy
Version: 1.5.18
Release_date: 2016/05/10
Nbproc: 4
Process_num: 3
Pid: 6374
Uptime: 0d 0h05m04s
Uptime_sec: 304
Memmax_MB: 0
Ulimit-n: 8039
Maxsock: 8039
Maxconn: 4000
Hard_maxconn: 4000
CurrConns: 0
CumConns: 3
CumReq: 10
MaxSslConns: 0
CurrSslConns: 0
CumSslConns: 0
Maxpipes: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 0
ConnRateLimit: 0
MaxConnRate: 1
SessRate: 0
SessRateLimit: 0
MaxSessRate: 1
SslRate: 0
SslRateLimit: 0
MaxSslRate: 0
SslFrontendKeyRate: 0
SslFrontendMaxKeyRate: 0
SslFrontendSessionReuse_pct: 0
SslBackendKeyRate: 0
SslBackendMaxKeyRate: 0
SslCacheLookups: 0
SslCacheMisses: 0
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
ZlibMemUsage: 0
MaxZlibMemUsage: 0
Tasks: 14
Run_queue: 1
Idle_pct: 100
node: docker_node1
description:
[root@docker_node1 ~]#
提示：以上信息就是监控页面中的指标数据；
示例：上线web1
在监控页面看web1的状态
提示：可以看到web1已经正常上线了；聪明的你一定想到了通过这样的方式动态的去管理后端server；这样就可以实现很多功能，比如通过zabbix去监控haproxy里的状态指标；通过脚本动态的上线和下线server、动态的修改后端server的权重（这个需要根据算法来，如果调度算法是动态的就支持，静态的就不支持）；
