系统运维-21-4-http工具
curl命令
基于URL语法在命令行方式下工作的文件传输工具
支持FTP FTPS HTTP HTTPS GOPHER TELNET DICT FILE 及LDAP等协议
支持HTTPS认证
支持HTTP的POST PUT等方法
支持FTP上传
支持KERBEROS认证
支持HTTP上传
支持代理服务器，COOKIES，用户名/密码认证，下载文件断点续传，上载文件断点续传，HTTP代理服务器管道（PROXY TUNNELING）
支持IPV6，SOCKS5代理服务器，通过HTTP代理服务器上传文件到FTP服务器等
curl常用选项
-A/--user-agent 设置用户代理发送给服务器
-basic 使用HTTP基本认证
--tcp-nodelay 使用TCP_NODELAY选项
-e/--referer 来源网址
--cacert CA证书 SSL
--compressed 要求返回是压缩的格式
-H/--header 自定义头部信息传递给服务器
-I/--head 只显示响应报文首部信息
--limit-rate 设置传输速度
-u/--user 设置服务器的用户和密码
-0/--http1.0 使用HTTP1.0
curl实例:
[[email protected] ~]# curl 172.20.0.132
page web1
[[email protected] ~]# tail /var/log/httpd/web1_access_log
172.20.0.128 - - [05/May/2019:07:50:54 -0400] "GET / HTTP/1.1" 200 10 "-" "curl/7.29.0"
[[email protected] ~]# curl -A 'IE11' 172.20.0.132
page web1
[[email protected] ~]# tail -1 /var/log/httpd/web1_access_log
172.20.0.128 - - [05/May/2019:07:52:44 -0400] "GET / HTTP/1.1" 200 10 "-" "IE11"
[[email protected] ~]# curl -e "http://www.google.com.hk/search" http://172.20.0.132/index.html
page web1
[[email protected] ~]# tail -1 /var/log/httpd/web1_access_log
172.20.0.128 - - [05/May/2019:08:01:53 -0400] "GET /index.html HTTP/1.1" 200 10 "http://www.google.com.hk/search" "curl/7.29.0"
[[email protected] ~]# curl -I http://172.20.0.132/index.html
HTTP/1.1 200 OK
Date: Sun, 05 May 2019 12:05:32 GMT
Server: Apache/2.4.6 (CentOS)
Last-Modified: Sat, 04 May 2019 15:38:22 GMT
ETag: "a-58811a846a6dd"
Accept-Ranges: bytes
Content-Length: 10
Content-Type: text/html; charset=UTF-8
elinks工具
[[email protected] ~]# elinks http://172.20.0.132/index.html
[[email protected] ~]# elinks -dump http://172.20.0.132/index.html
page web1
使用mod_deflate模块压缩页面优化传输速度
适用场景
1）节约带宽，额外消耗CPU，同时可能有些较老浏览器不支持
2）压缩适用压缩的资源，例如文本文件
mod_deflate压缩实例：
[[email protected] ~]# wc -l /var/log/messages
409 /var/log/messages
[[email protected] ~]# cp /var/log/messages /vhosts/web1/htdocs/test.html
[[email protected] ~]# chmod 644 /vhosts/web1/htdocs/test.html
[[email protected] ~]# vim /etc/httpd/conf/httpd.conf
[[email protected] ~]# tail -19 /etc/httpd/conf/httpd.conf
SetOutputFilter DEFLATE
# mod_deflate configuration
# Restrict compression to these MIME types
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/x-javascript
AddOutputFilterByType DEFLATE text/x-javascript
AddOutputFilterByType DEFLATE text/css
# Level of compression (Highest 9 - Lowest 1)
DeflateCompressionLevel 9
# Netscape 4.x has some problems
BrowserMatch ^Mozilla/4 gzip-only-text/html
# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip
# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
[[email protected] ~]# httpd -M | grep deflate
deflate_module (shared)
[[email protected] ~]# httpd -t
Syntax OK
[[email protected] ~]# systemctl reload httpd
[[email protected] ~]# curl -I http://172.20.0.132/test.html
HTTP/1.1 200 OK
Date: Sun, 05 May 2019 12:50:22 GMT
Server: Apache/2.4.6 (CentOS)
Last-Modified: Sun, 05 May 2019 12:23:28 GMT
ETag: "a907-588230d163a0f"
Accept-Ranges: bytes
Content-Length: 43271
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8
[[email protected] ~]# curl -I --compressed http://172.20.0.132/test.html
HTTP/1.1 200 OK
Date: Sun, 05 May 2019 12:50:56 GMT
Server: Apache/2.4.6 (CentOS)
Last-Modified: Sun, 05 May 2019 12:23:28 GMT
ETag: "a907-588230d163a0f-gzip"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 2853
Content-Type: text/html; charset=UTF-8
https
SSL会话的简化过程
1）客户端发送可供选择的加密方式，并向服务器请求证书
2）服务器端发送证书以及选定的加密方式给客户端
3）客户端取得证书，并进行证书验证
如果信任给其发证书的CA
（1）验证证书来源的合法性，用CA的公钥解密证书上数字签名
（2）验证证书的内容的合法性，完整性验证
（3）检查证书的有效期限
（4）检查证书是否被吊销（大多数浏览器不做此步骤）
（5）证书中拥有者的名字，与访问的目标主机要一致
4）客户端生成临时会话**（对称**），并使用服务器端的公钥加密此数据发送给服务器，完成**交换
5）服务此**加密用户请求的资源，响应给客户端
注意：SSL会话是基于IP地址创建，所以单IP的主机上，仅仅可以使用一个HTTPS虚拟主机
配置httpd支持https
1)为服务器申请数字证书
测试：通过私有CA发证书
（1）创建私有CA
（2）在服务器创建证书签署请求
（3）CA签证
2）配置httpd支持ssl，及使用的证书
# yum install -y mod_ssl
配置文件 /etc/httpd/conf.d/ssl.conf
DocumentRoot
ServerName
SSLCertificateFile
SSLCertificateKeyFile
3)测试基于https访问相应的主机
# openssl s_client
配置https实例：
[[email protected] ~]# cd /etc/pki/CA
[[email protected] CA]# (umask 077;openssl genrsa -out private/cakey.pem 2048)
Generating RSA private key, 2048 bit long modulus
................+++
.................+++
e is 65537 (0x10001)
[[email protected] CA]# ll private/
total 4
-rw-------. 1 root root 1679 May 5 09:40 cakey.pem
[[email protected] CA]# touch index.txt
[[email protected] CA]# echo 01 > serial
[[email protected] CA]# openssl req -new -x509 -key private/cakey.pem -out cacert.pem -days 7300
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:Beijing
Locality Name (eg, city) [Default City]:Beijing
Organization Name (eg, company) [Default Company Ltd]:IBM
Organizational Unit Name (eg, section) []:example
Common Name (eg, your name or your server's hostname) []:ca.example.com
Email Address []:[email protected]
[[email protected] ~]# cd /etc/httpd/ssl
[[email protected] ssl]# (umask 077;openssl genrsa -out httpd.key 1024)
Generating RSA private key, 1024 bit long modulus
............................++++++
........................................++++++
e is 65537 (0x10001)
[[email protected] ssl]# openssl req -new -key httpd.key -out httpd.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:Beijing
Locality Name (eg, city) [Default City]:Beijing
Organization Name (eg, company) [Default Company Ltd]:IBM
Organizational Unit Name (eg, section) []:example
Common Name (eg, your name or your server's hostname) []:web1.example.com
Email Address []:[email protected]
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
[[email protected] ssl]# scp httpd.csr [email protected]:/tmp/
The authenticity of host '172.20.0.132 (172.20.0.132)' can't be established.
ECDSA key fingerprint is SHA256:2kh6qcdfS88uBPN4RI9/yGdd83S9wY3a16+A2qf7ImE.
ECDSA key fingerprint is MD5:18:4d:e5:3e:76:44:e7:99:c0:e5:bd:48:1b:34:99:da.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.20.0.132' (ECDSA) to the list of known hosts.
[email protected]'s password:
httpd.csr 100% 708 531.2KB/s 00:00
[[email protected] CA]# openssl ca -in /tmp/httpd.csr -out certs/web1.example.com.crt -days 365
Using configuration from /etc/pki/tls/openssl.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 1 (0x1)
Validity
Not Before: May 5 14:01:01 2019 GMT
Not After : May 4 14:01:01 2020 GMT
Subject:
countryName = CN
stateOrProvinceName = Beijing
organizationName = IBM
organizationalUnitName = example
commonName = web1.example.com
emailAddress = [email protected]
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Comment:
OpenSSL Generated Certificate
X509v3 Subject Key Identifier:
B6:46:88:4F:38:8D:00:1C:2F:30:71:35:6B:65:C1:06:E9:D8:75:83
X509v3 Authority Key Identifier:
keyid:34:12:65:B0:9A:D9:02:EE:0C:EE:B1:CB:58:85:65:FB:DF:46:18:C0
Certificate is to be certified until May 4 14:01:01 2020 GMT (365 days)
Sign the certificate? [y/n]:y
1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
[[email protected] CA]# ll newcerts/
total 4
-rw-r--r-- 1 root root 3884 May 5 10:01 01.pem
[[email protected] CA]# ll certs/
total 4
-rw-r--r-- 1 root root 3884 May 5 10:01 web1.example.com.crt
[[email protected] CA]# /usr/bin/scp certs/web1.example.com.crt [email protected]:/etc/httpd/ssl/
web1.example.com.crt 100% 3884 691.1KB/s 00:00
[[email protected] ssl]# httpd -M | grep ssl
[[email protected] ssl]# yum install mod_ssl -y
[[email protected] ssl]# httpd -M | grep ssl
ssl_module (shared)
[[email protected] ssl]# rpm -ql mod_ssl
/etc/httpd/conf.d/ssl.conf
/etc/httpd/conf.modules.d/00-ssl.conf
/usr/lib64/httpd/modules/mod_ssl.so
/usr/libexec/httpd-ssl-pass-dialog
/var/cache/httpd/ssl
[[email protected] ssl]# cd /etc/httpd/conf.d
[[email protected] conf.d]# cp ssl.conf{,.bak}
[[email protected] conf.d]# vim ssl.conf
[[email protected] conf.d]# httpd -t
Syntax OK
[[email protected] conf.d]# systemctl restart httpd
[[email protected] conf.d]# grep -v ^# ssl.conf | grep -v ^$
Listen 443 https
SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
<VirtualHost *:443>
DocumentRoot /vhosts/web1/htdocs
ServerName web1.example.com
ErrorLog logs/ssl_error_log
TransferLog logs/ssl_access_log
LogLevel warn
SSLEngine on
SSLProtocol all -SSLv2 -SSLv3
SSLCipherSuite HIGH:3DES:!aNULL:!MD5:!SEED:!IDEA
SSLCertificateFile /etc/httpd/ssl/web1.example.com.crt
SSLCertificateKeyFile /etc/httpd/ssl/httpd.key
<Files ~ "\.(cgi|shtml|phtml|php3?)$">
SSLOptions +StdEnvVars
</Files>
<Directory "/var/www/cgi-bin">
SSLOptions +StdEnvVars
</Directory>
BrowserMatch "MSIE [2-5]" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0
CustomLog logs/ssl_request_log \
"%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
</VirtualHost>
httpd自带的工具程序
apachectl，httpd自带的服务控制脚本，支持start，stop
apxs，由httpd-devel包提供，扩展httpd使用第三方模块的工具
rotatelogs，日志滚动工具
suexec，访问某些具有特殊权限配置的资源时，临时切换至指定用户运行
ab，apache benchmark
http压力测试工具
ab 命令行工具，测试效果有限
webbench 命令行工具，测试效果有限
http_load 命令行工具，测试效果有限
jmeter 开源测试工具
loadrunner 商业测试工具，有专门的测试认证，使用很难
tcpcopy 网易研发的工具
ab [OPTIONS] URL
-n 总的请求数
-c 模拟并发数
-k 以持久连接模式测试
ulimit -n # 调整当前用户所同时打开的文件数
ab测试实例：
[[email protected] ~]# ab -c 100 -n 10000 http://172.20.0.132/test.html
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 172.20.0.132 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests
Server Software: Apache/2.4.6
Server Hostname: 172.20.0.132
Server Port: 80
Document Path: /test.html
Document Length: 207 bytes
Concurrency Level: 100
Time taken for tests: 1.746 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Non-2xx responses: 10000
Total transferred: 4060000 bytes
HTML transferred: 2070000 bytes
Requests per second: 5727.73 [#/sec] (mean)
Time per request: 17.459 [ms] (mean)
Time per request: 0.175 [ms] (mean, across all concurrent requests)
Transfer rate: 2270.95 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 0.7 1 10
Processing: 5 17 3.1 16 28
Waiting: 1 16 3.1 16 27
Total: 9 17 3.2 17 28
Percentage of the requests served within a certain time (ms)
50% 17
66% 18
75% 19
80% 19
90% 22
95% 23
98% 25
99% 27
100% 28 (longest request)