您的位置: 首页  >  文章  >  Linux下大数据集群安装presto+kudu(3节点)

Linux下大数据集群安装presto+kudu(3节点)

分类: 文章 • 2023-03-27 17:01:07

1、准备工作

关闭selinux、防火墙、安装基础软件工具、安装JDK(jdk1.8.0_211)、所有主机配置好SSH免密登录、配置hosts文件（127.0.0.1的2行删除）

cat /etc/hosts

192.168.0.107 test3

192.168.0.108 test4

192.168.0.109 test5

2、安装kudu

cd /data&&rpm -ivh kudu-1.10.0-1.x86_64.rpm

mkdir -p /data/kudu/master/logs /data/kudu/master/wals /data/kudu/master/data

mkdir -p /data/kudu/tserver/logs /data/kudu/tserver/data /data/kudu/tserver/wals

vi /etc/kudu/conf/master.gflagfile

## Comma-separated list of the RPC addresses belonging to all Masters in this cluster.

## NOTE: if not specified, configures a non-replicated Master.

#--master_addresses=

--master_addresses=test107:7051,test108:7051,test109:7051

--log_dir=/data/kudu/master/logs

--fs_wal_dir=/data/kudu/master/wals

--fs_data_dirs=/data/kudu/master/data

vi /etc/kudu/conf/tserver.gflagfile

#Comma separated addresses of the masters which the tablet server should connect to.

--tserver_master_addrs=test107:7051,test108:7051,test109:7051

--log_dir=/data/kudu/tserver/logs

--fs_wal_dir=/data/kudu/tserver/wals

--fs_data_dirs=/data/kudu/tserver/data

修改kudu目录权限

chown -R kudu:kudu /data/kudu

在test3、test4、test5三台主机执行kudu master启动命令

systemctl start kudu-master

在所有节点执行kudu tserver启动命令

systemctl start kudu-tserver

查看状态：

3.安装presto

mkdir -p /data/presto-server-331/etc/catalog cd /data/presto-server-331/etc 主节点配置文件

vi config.properties

coordinator=true

node-scheduler.include-coordinator=false

http-server.http.port=18080

discovery-server.enabled=true

discovery.uri=http://192.168.0.107:18080

query.max-memory=8GB

query.max-memory-per-node=1GB

query.max-run-time=600s

从节点配置文件

coordinator=false

http-server.http.port=18080

discovery.uri=http://192.168.0.107:18080

query.max-memory=8GB

query.max-memory-per-node=1GB

以下所有节点相同

vi jvm.config

-server

-Xmx20G

-XX:+UseG1GC

-XX:G1HeapRegionSize=32M

-XX:+UseGCOverheadLimit

-XX:+ExplicitGCInvokesConcurrent

-XX:+HeapDumpOnOutOfMemoryError

-XX:OnOutOfMemoryError=kill -9 %p

-XX:+CMSClassUnloadingEnabled

-XX:+AggressiveOpts

-DHADOOP_USER_NAME=root

-Dpresto-temporarily-allow-java8=true           

#这里会报一个错，加上-Dpresto-temporarily-allow-java8=true一句就OK了

Linux下大数据集群安装presto+kudu(3节点)

vi log.properties

com.facebook.presto=INFO

vi node.properties

node.environment=mycluster

#这里需要注意，node_id的101跟着主机名修改，在centos102主节，即改为102，不能重复

node.id=node_test_107

node.data-dir=/data/presto/

cd /data/presto-server-331/etc/catalog   

vi hive.properties

#这里需要注意，连接名称影响调用的驱动版本，目前必须是hive-hadoop2

connector.name=hive-hadoop2

##hive元数据库

hive.metastore.uri=thrift://test3:9083

hive.config.resources=/data/hadoop-3.1.2/etc/hadoop/core-site.xml,/data/hadoop-3.1.2/etc/hadoop/hdfs-site.xml

vi kudu.properties

connector.name=kudu

## List of Kudu master addresses, at least one is needed (comma separated)

## Supported formats: example.com, example.com:7051, 192.0.2.1, 192.0.2.1:7051,

##                    [2001:db8::1], [2001:db8::1]:7051, 2001:db8::1

kudu.client.master-addresses=test3:7051,test4:7051,test5:7051

## Kudu does not support schemas, but the connector can emulate them optionally.

## By default, this feature is disabled, and all tables belong to the default schema.

## For more details see connector documentation.

kudu.schema-emulation.enabled=true

## Prefix to use for schema emulation (only relevant if `kudu.schema-emulation.enabled=true`)

## The standard prefix is `presto::`. Empty prefix is also supported.

## For more details see connector documentation.

#kudu.schema-emulation.prefix=

#######################

### Advanced Kudu Java client configuration

#######################

## Default timeout used for administrative operations (e.g. createTable, deleteTable, etc.)

#kudu.client.defaultAdminOperationTimeout = 30s

kudu.client.default-admin-operation-timeout = 60s

## Default timeout used for user operations

#kudu.client.defaultOperationTimeout = 30s

kudu.client.default-operation-timeout = 60s

## Default timeout to use when waiting on data from a socket

#kudu.client.defaultSocketReadTimeout = 30s

kudu.client.default-socket-read-timeout = 60s

## Disable Kudu client's collection of statistics.

#kudu.client.disableStatistics = false

启动presto服务

/data/presto-server-331/bin/launcher start

到这里安装完成，中间有报错的，找对应的日志排查一下，最后附上几个主要服务的web页面地址：

kudu http://192.168.0.107:8050/

presto http://192.168.0.107:18080

Linux下大数据集群安装presto+kudu(3节点)

Linux下大数据集群安装presto+kudu(3节点)

原文参考博客

https://blog.csdn.net/fly0512/article/details/100863889

相关推荐