YARN资源配置
Yarn资源配置需要平衡服务器cpu，内存，磁盘资源。Hortonworks推荐两个Container使用1个cpu，1块磁盘能达到很好平衡。
Hortonworks提供了yarn内存参数设置脚本，执行yarn-utils.sh(下载地址https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.1/bk_installing_manually_book/content/rpm-chap1-9.html)脚本生成配置参数。
示例：python yarn-utils.py -c 40 -m 224 -d 12 -k False
参数：
|
Option |
Description |
|
-c CORES |
The number of cores on each host. |
|
-m MEMORY |
The amount of memory on each host in GB. |
|
-d DISKS |
The number of disks on each host. |
|
-k HBASE |
"True" if HBase is installed, "False" if not. |
内存配置：
系统及hbase预留内存参考：
|
Total Memory per Node |
Recommended Reserved System Memory |
Recommended Reserved HBase Memory |
|
4 GB |
1 GB |
1 GB |
|
8 GB |
2 GB |
1 GB |
|
16 GB |
2 GB |
2 GB |
|
24 GB |
4 GB |
4 GB |
|
48 GB |
6 GB |
8 GB |
|
64 GB |
8 GB |
8 GB |
|
72 GB |
8 GB |
8 GB |
|
96 GB |
12 GB |
16 GB |
|
128 GB |
24 GB |
24 GB |
|
256 GB |
32 GB |
32 GB |
|
512 GB |
64 GB |
64 GB |
计算每台服务器最多有多少个containers
containers = min (2*CORES, 1.8*DISKS, (Total available RAM) / MIN_CONTAINER_SIZE)
CORES:cpu数
DISKS：磁盘数
Total available RAM：剩余内存数（减去系统及其它已使用的内存）
MIN_CONTAINER_SIZE：container最小的容量大小
MIN_CONTAINER_SIZE参考值：
|
Total RAM per Node |
Recommended Minimum Container Size |
|
Less than 4 GB |
256 MB |
|
Between 4 GB and 8 GB |
512 MB |
|
Between 8 GB and 24 GB |
1024 MB |
|
Above 24 GB |
2048 MB |
The final calculation is to determine the amount of RAM per container:
RAM-per-container = max(MIN_CONTAINER_SIZE, (Total Available RAM) / containers))
|
Configuration File |
Configuration Setting |
Value Calculation |
|
yarn-site.xml |
yarn.nodemanager.resource.memory-mb |
= containers * RAM-per-container |
|
yarn-site.xml |
yarn.scheduler.minimum-allocation-mb |
= RAM-per-container |
|
yarn-site.xml |
yarn.scheduler.maximum-allocation-mb |
= containers * RAM-per-container |
|
mapred-site.xml |
mapreduce.map.memory.mb |
= RAM-per-container |
|
mapred-site.xml |
mapreduce.reduce.memory.mb |
= 2 * RAM-per-container |
|
mapred-site.xml |
mapreduce.map.java.opts |
= 0.8 * RAM-per-container |
|
mapred-site.xml |
mapreduce.reduce.java.opts |
= 0.8 * 2 * RAM-per-container |
|
yarn-site.xml (check) |
yarn.app.mapreduce.am.resource.mb |
= 2 * RAM-per-container |
|
yarn-site.xml (check) |
yarn.app.mapreduce.am.command-opts |
= 0.8 * 2 * RAM-per-container |
以上是Hortonworks的计算内存公式。具体可以参考官网：https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.1/bk_installing_manually_book/content/rpm-chap1-11.html
Cloudera CDH也提供了yarn资源配置的Excel表格，根据表格填写对应的数值计算并校验数值是否合理。下载地址：http://tiny.cloudera.com/yarn-tuning-guide
实际配置的时候参考Hortonworks配置值对Excel中的值进行调整及实际生产任务大小情况调整。这样下来基本可以获得比较理想的效果