Linux Zookeeper安装(服务器架设篇)

zookeeper简介

ZooKeeper是一个分布式的,开放源码的分布式应用程序协调服务,是Google的Chubby一个开源的实现,是Hadoop和Hbase的重要组件。它是一个为分布式应用提供一致性服务的软件,提供的功能包括:配置维护、名字服务、分布式同步、组服务等。
ZooKeeper的目标就是封装好复杂易出错的关键服务,将简单易用的接口和性能高效、功能稳定的系统提供给用户。
ZooKeeper包含一个简单的原语集,提供Java和C的接口。ZooKeeper源代码中,提供了分布式独享锁、选举、队列的接口,代码在zookeeper-x.x.x\src\recipes。其中分布锁和队列有Java和C两个版本,选举只有Java版本。

zookeeper的原理

ZooKeeper是以Fast Paxos算法为基础的,paxos算法存在活锁的问题,即当有多个proposer交错提交时,有可能互相排斥导致没有一个proposer能提交成功,而Fast Paxos作了一些优化,通过选举产生一个leader,只有leader才能提交propose。=。因此要想弄懂ZooKeeper首先得对Fast Paxos有所了解。

ZooKeeper的基本运转流程:

  1. 选举Leader。
  2. 同步数据。
  3. 选举Leader过程中算法有很多,但要达到的选举标准是一致的。
  4. Leader要具有最高的zxid。
  5. 集群中大多数的机器得到响应并follow选出的Leader

zookeeper的特点

在Zookeeper中,znode是一个跟Unix文件系统路径相似的节点,可以往这个节点存储或获取数据。如果在创建znode时Flag设置为EPHEMERAL,那么当创建这个znode的节点和Zookeeper失去连接后,这个znode将不再存在在Zookeeper里,Zookeeper使用Watcher察觉事件信息。当客户端接收到事件信息,比如连接超时、节点数据改变、子节点改变,可以调用相应的行为来处理数据。Zookeeper的Wiki页面展示了如何使用Zookeeper来处理事件通知,队列,优先队列,锁,共享锁,可撤销的共享锁,两阶段提交。那么Zookeeper能作什么事情呢,简单的例子:假设我们有20个搜索引擎的服务器(每个负责总索引中的一部分的搜索任务)和一个总服务器(负责向这20个搜索引擎的服务器发出搜索请求并合并结果集),一个备用的总服务器(负责当总服务器宕机时替换总服务器),一个web的cgi(向总服务器发出搜索请求)。搜索引擎的服务器中的15个服务器提供搜索服务,5个服务器正在生成索引。这20个搜索引擎的服务器经常要让正在提供搜索服务的服务器停止提供服务开始生成索引,或生成索引的服务器已经把索引生成完成可以搜索提供服务了。使用Zookeeper可以保证总服务器自动感知有多少提供搜索引擎的服务器并向这些服务器发出搜索请求,当总服务器宕机时自动启用备用的总服务器。

安装

准备工作
在安装zookeeper之前我们要确保自己主机上已经有了Java环境,建议安装Oracle JDK8.x

环境 版本
RHEL/CentOS 7.x
Java 8.x
Zookeeper 3.4.x
1
2
3
4
5
6
7
8
9
10
11
# 检查java环境
[thinktik@localhost ~]$ java -version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)
[thinktik@localhost ~]$ ls
apache-maven-3.6.0-bin.tar.gz httpd-2.4.37.tar.gz jenkins.war sonarqube-6.7.5.tar
httpd java8 maven zookeeper-3.4.13.tar.gz
httpd-2.4.37 jdk-8u191-linux-x64.tar.gz sonarqube-6.7.5
# 解压
[thinktik@localhost ~]$ tar -zxvf zookeeper-3.4.13.tar.gz

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[thinktik@localhost ~]$ ls
apache-maven-3.6.0-bin.tar.gz httpd-2.4.37.tar.gz jenkins.war sonarqube-6.7.5.tar
httpd java8 maven zookeeper-3.4.13
httpd-2.4.37 jdk-8u191-linux-x64.tar.gz sonarqube-6.7.5 zookeeper-3.4.13.tar.gz
[thinktik@localhost ~]$ cd zookeeper-3.4.13
[thinktik@localhost zookeeper-3.4.13]$ ls
bin dist-maven lib README_packaging.txt zookeeper-3.4.13.jar.asc
build.xml docs LICENSE.txt recipes zookeeper-3.4.13.jar.md5
conf ivysettings.xml NOTICE.txt src zookeeper-3.4.13.jar.sha1
contrib ivy.xml README.md zookeeper-3.4.13.jar
# 创建data文件夹保持数据
[thinktik@localhost zookeeper-3.4.13]$ mkdir data
# 创建logs文件夹保存日志
[thinktik@localhost zookeeper-3.4.13]$ mkdir logs
[thinktik@localhost zookeeper-3.4.13]$ pwd
/home/thinktik/zookeeper-3.4.13
[thinktik@localhost zookeeper-3.4.13]$ cd conf/
[thinktik@localhost conf]$ ls
configuration.xsl log4j.properties zoo_sample.cfg
# 将样例配置文件复制一份为zoo.cfg
[thinktik@localhost conf]$ cp zoo_sample.cfg zoo.cfg
# 修改配置文件
[thinktik@localhost conf]$ vim zoo.cfg

配置文件参考(这里我将dataDir和dataLogDir路径改为了我刚才创建文件夹)

单机版启动

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
[thinktik@localhost conf]$ cd ..
[thinktik@localhost zookeeper-3.4.13]$ ls
bin dist-maven LICENSE.txt recipes zookeeper-3.4.13.jar.sha1
build.xml docs logs src
conf ivysettings.xml NOTICE.txt zookeeper-3.4.13.jar
contrib ivy.xml README.md zookeeper-3.4.13.jar.asc
data lib README_packaging.txt zookeeper-3.4.13.jar.md5
[thinktik@localhost zookeeper-3.4.13]$ cd bin/
[thinktik@localhost bin]$ ls
README.txt zkCli.cmd zkEnv.cmd zkServer.cmd zkTxnLogToolkit.cmd
zkCleanup.sh zkCli.sh zkEnv.sh zkServer.sh zkTxnLogToolkit.sh
# 启动
[thinktik@localhost bin]$ ./zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /home/thinktik/zookeeper-3.4.13/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
# 状态查询为正在运行,单例模式
[thinktik@localhost bin]$ ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/thinktik/zookeeper-3.4.13/bin/../conf/zoo.cfg
Mode: standalone

# 检查端口监听ok
[thinktik@localhost bin]$ netstat -nlp | grep 2181
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::2181 :::* LISTEN 1958/java
# 检查运行线程ok
[thinktik@localhost bin]$ ps -ef |grep zoo
thinktik 1958 1 0 11:17 pts/0 00:00:00 java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,CONSOLE -cp /home/thinktik/zookeeper-3.4.13/bin/../build/classes:/home/thinktik/zookeeper-3.4.13/bin/../build/lib/*.jar:/home/thinktik/zookeeper-3.4.13/bin/../lib/slf4j-log4j12-1.7.25.jar:/home/thinktik/zookeeper-3.4.13/bin/../lib/slf4j-api-1.7.25.jar:/home/thinktik/zookeeper-3.4.13/bin/../lib/netty-3.10.6.Final.jar:/home/thinktik/zookeeper-3.4.13/bin/../lib/log4j-1.2.17.jar:/home/thinktik/zookeeper-3.4.13/bin/../lib/jline-0.9.94.jar:/home/thinktik/zookeeper-3.4.13/bin/../lib/audience-annotations-0.5.0.jar:/home/thinktik/zookeeper-3.4.13/bin/../zookeeper-3.4.13.jar:/home/thinktik/zookeeper-3.4.13/bin/../src/java/lib/*.jar:/home/thinktik/zookeeper-3.4.13/bin/../conf: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /home/thinktik/zookeeper-3.4.13/bin/../conf/zoo.cfg
thinktik 1978 1890 0 11:19 pts/0 00:00:00 grep --color=auto zoo
# 客户端连接ok
[thinktik@localhost bin]$ ./zkCli.sh
Connecting to localhost:2181
2018-11-11 11:48:15,274 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 04:05 GMT
2018-11-11 11:48:15,277 [myid:] - INFO [main:Environment@100] - Client environment:host.name=localhost
2018-11-11 11:48:15,277 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_191
2018-11-11 11:48:15,279 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2018-11-11 11:48:15,279 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/home/thinktik/java8/jre
2018-11-11 11:48:15,280 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/home/thinktik/zookeeper-3.4.13/bin/../build/classes:/home/thinktik/zookeeper-3.4.13/bin/../build/lib/*.jar:/home/thinktik/zookeeper-3.4.13/bin/../lib/slf4j-log4j12-1.7.25.jar:/home/thinktik/zookeeper-3.4.13/bin/../lib/slf4j-api-1.7.25.jar:/home/thinktik/zookeeper-3.4.13/bin/../lib/netty-3.10.6.Final.jar:/home/thinktik/zookeeper-3.4.13/bin/../lib/log4j-1.2.17.jar:/home/thinktik/zookeeper-3.4.13/bin/../lib/jline-0.9.94.jar:/home/thinktik/zookeeper-3.4.13/bin/../lib/audience-annotations-0.5.0.jar:/home/thinktik/zookeeper-3.4.13/bin/../zookeeper-3.4.13.jar:/home/thinktik/zookeeper-3.4.13/bin/../src/java/lib/*.jar:/home/thinktik/zookeeper-3.4.13/bin/../conf:
2018-11-11 11:48:15,280 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2018-11-11 11:48:15,280 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2018-11-11 11:48:15,280 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2018-11-11 11:48:15,280 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2018-11-11 11:48:15,280 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2018-11-11 11:48:15,280 [myid:] - INFO [main:Environment@100] - Client environment:os.version=3.10.0-862.el7.x86_64
2018-11-11 11:48:15,280 [myid:] - INFO [main:Environment@100] - Client environment:user.name=thinktik
2018-11-11 11:48:15,280 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/home/thinktik
2018-11-11 11:48:15,281 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/home/thinktik/zookeeper-3.4.13/bin
2018-11-11 11:48:15,282 [myid:] - INFO [main:ZooKeeper@442] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@25f38edc
2018-11-11 11:48:15,320 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1029] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
Welcome to ZooKeeper!
JLine support is enabled
2018-11-11 11:48:15,489 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@879] - Socket connection established to localhost/127.0.0.1:2181, initiating session
[zk: localhost:2181(CONNECTING) 0] 2018-11-11 11:48:15,829 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1303] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x100003dc9010000, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

[zk: localhost:2181(CONNECTED) 0]
[zk: localhost:2181(CONNECTED) 0]

这里Zookeeper单机版就安装完成了

Zookeeper集群

单机模式的zk进程虽然便于开发与测试,但并不适合在生产环境使用。在生产环境下,我们需要使用集群模式来对zk进行部署。

注意在集群模式下,建议至少部署3个zk进程,或者部署奇数个zk进程。如果只部署2个zk进程,当其中一个zk进程挂掉后,剩下的一个进程并不能构成一个quorum的大多数。因此,部署2个进程甚至比单机模式更不可靠,因为2个进程其中一个不可用的可能性比一个进程不可用的可能性还大。

我继续按单机版的安装方式在例外的两个虚拟机上安装2个zookeeper,并互相开放防火墙。

主机 IP
CentOS01 192.168.50.186
CentOS02 192.168.50.164
CentOS03 192.168.50.30

关键点是你将所有的zookeeper节点都写在配置文件里面去,并在这些主机上开发端口。我开的是2181,2888,3888三个端口,3个主机都开。

我的集群配置添加如下3个节点配置(记得Linux打开端口):

1
2
3
server.1=192.168.50.186:2888:3888
server.2=192.168.50.164:2888:3888
server.3=192.168.50.30:2888:3888

同时不要忘了在这3个主机安装的zookeeper的dataDir建立各自的myid文件并填入自己的id.

1
2
3
4
5
6
7
8
9
10
# 我以3号zookeeper为例
[thinktik@localhost zookeeper]$ cd data/
[thinktik@localhost data]$ pwd
/home/thinktik/zookeeper/data
# 这个zookeeper配置dataDir下面我建立了myid
[thinktik@localhost data]$ ls
myid version-2 zookeeper_server.pid
# myid里面写上了它的id,id好与server.x的x对应。例如三号机是server.3=192.168.50.30:2888:3888那么myid里面填3
[thinktik@localhost data]$ cat myid
3

启动集群,这里3个zookeeper随便启动就可以,没什么先后顺序,每个zookeeper启动后会找自己其他节点的。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# 1号zookeeper我先启动。 2,3号我接着起
[thinktik@localhost bin]$ ./zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /home/thinktik/zookeeper-3.4.13/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[thinktik@localhost bin]$ ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/thinktik/zookeeper-3.4.13/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.
[thinktik@localhost bin]$ vim ../conf/zoo.cfg
[thinktik@localhost bin]$ vim ../data/myid
[thinktik@localhost bin]$ ./zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /home/thinktik/zookeeper-3.4.13/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

# 1,2,3都起后查看状态,1号成了leader
[thinktik@localhost bin]$ ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/thinktik/zookeeper-3.4.13/bin/../conf/zoo.cfg
Mode: leader
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# 启动2,3号
[thinktik@localhost bin]$ ./zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /home/thinktik/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[thinktik@localhost bin]$ vim ../data/myid
bash: vim: command not found
[thinktik@localhost bin]$ vi ../data/myid
[thinktik@localhost bin]$ ./zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /home/thinktik/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

# 1,2,3都起后查看状态,2,3号成了leader
[thinktik@localhost bin]$ ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/thinktik/zookeeper/bin/../conf/zoo.cfg
Mode: follower

验证下集群的功能

leader:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
[zk: localhost:2181(CONNECTED) 0] ls/
ZooKeeper -server host:port cmd args
stat path [watch]
set path data [version]
ls path [watch]
delquota [-n|-b] path
ls2 path [watch]
setAcl path acl
setquota -n|-b val path
history
redo cmdno
printwatches on|off
delete path [version]
sync path
listquota path
rmr path
get path [watch]
create [-s] [-e] path data acl
addauth scheme auth
quit
getAcl path
close
connect host:port
# 创建节点
[zk: localhost:2181(CONNECTED) 1] create /testNode '测试'
Created /testNode
# 节点已创建
[zk: localhost:2181(CONNECTED) 2] ls /
[zookeeper, testNode]

follower

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# 查询节点,发现leader的节点已经同步,一致性没问题
[zk: localhost:2181(CONNECTED) 1] ls /
[zookeeper, testNode]
# 查看这个节点的内容,也没毛病
[zk: localhost:2181(CONNECTED) 3] get /testNode
测试
cZxid = 0x100000002
ctime = Sun Nov 11 13:28:20 CST 2018
mZxid = 0x100000002
mtime = Sun Nov 11 13:28:20 CST 2018
pZxid = 0x100000002
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 6
numChildren = 0

这里基本集群就ok了。

本文除了阅读了Zookeeper官方文档还参考的其他人写的文章: