所属分类: linux&Unix 整理: FengNet.Com 更新日期:2003/8/12 8:24:37 阅读次数:9949

Sino-trade.com集群技术报告


拓扑图:
点击打开新窗口

架构说明:
三层架构:
第一层:集群调度器(两台)
第二层:web服务器(三台)
第三层:sino-trade.com数据库(DELL服务器一台)
软件包:
内核以及补丁:
linux-2.4.18.tar.gz
hidden-2.4.5-1.diff
linux-2.4.18-ipvs-1.0.2.patch.gz
aa.config(我的内核配置文件)
集群软件:
ipvsadm:
ipvs-1.0.2.tar.gz
mon:
Mon-0.11.tar.gz
mon-0.99.1.tar.gz
Time-HiRes-01.20.tar.gz
Period-1.20.tar.gz
Convert-BER-1.31.tar.gz
HeartBeat:
heartbeat-0.4.9.1.tar.gz
NFS:
REDHAT系统自带(7.2)
安装方法:
第一步:配置内核 :
配置内核,是相对简单的步骤,需要以下软件包:
linux-2.4.18.tar.gz(www.kernel.org)
linux-2.4.18-ipvs-1.0.2.patch.gz(www.linuxvirtual.org)
hidden-2.4.5-1.diff(http://linuxvirtualserver.org/~julian/)
对于LVS集群器
1/cp linux-2.4.18.tar.gz /usr/src
2/cd /usr/src
3/tar xvzf linux-2.4.18.tar.gz
4/cd linux
5/cp /home/sun/linux-2.4.18-ipvs-1.0.2.patch.gz ./
6/cp /home/sun/hidden-2.4.5-1.diff ./
7/gunzip linux-2.4.18-ipvs-1.0.2.patch.gz
8/patch -p1 < linux-2.4.18-ipvs-1.0.2.patch
9/patch -p1 < hidden-2.4.5-1.diff
10/make menuconfig (导入aa.config)
然后将内核编译后,运行新内核。
对于real server
1/cp /home/sun/linux-2.4.18.tar.gz /usr/src
2/cd /usr/src
3/tar xvzf linux-2.4.18.tar.gz
4/cd linux
5/cp /home/sun/hidden-2.4.5-1.diff ./
6/patch -p1 < hidden-2.4.5-1.diff(因为实际服务器,所以,不打IPVS补丁)
7/make meunconfig
然后是编译内核,用新内核启动
编辑:/etc/rc.d/rc.local
加入:
/bin/echo 1 >/proc/sys/net/ipv4/ip_forward
/sbin/modprobe ipip
/sbin/ifconfig tunl0 0.0.0.0 up
/bin/echo 1 > /proc/sys/net/ipv4/conf/all/hidden
/bin/echo 1 > /proc/sys/net/ipv4/conf/tunl0/hidden
/sbin/ifconfig tunl0 218.106.209.45 netmask 255.255.255.255 broadcast 218.106.209.255 up
(至此www的服务器配置完毕,装入apache即刻,具体步骤省略)
第二步:mon的安装:
mon安装在lvs的集群调度器和LVS2上面:
Mon-0.11.tar.gz(ftp://ftp.kernel.org/pub/software/admin/mon/)
mon-0.99.1.tar.gz(同上)
Time-HiRes-01.20.tar.gz(www.cpan.org,自己搜索吧,一个perl模块)
Period-1.20.tar.gz(www.cpan.org,跟上面一样)
Convert-BER-1.31.tar.gz(www.cpan.org,一样)
步骤:
1.安装perl模块,就是1.3.4.5,三个perl模块和一个Mon编译进去:
用:
A>cd <模块>
B>perl MakeFile.pl
C>make
D>make install
成功的话,那就开始mon安装了,
2.直接tar xvzf mon-0.99.1.tar.gz,然后将其cp到/usr/local/mon,没什么原因,是为了以后管理方便。
至此,mon安装完毕
然后是麻烦一点的,配置mon:
我们这里需要两个文件:
1。lvs.alert (/usr/local/mon/alert.d)
#!/usr/bin/perl
#
# lvs.alert - Linux Virtual Server alert for mon
#
# It can be activated by mon to remove a real server when the
# service is down, or add the server when the service is up.
#
#
use Getopt::Std;
getopts ("s:g:h:t:l:P:V:R:W:F:u");

$ipvsadm = "/sbin/ipvsadm";
$protocol = $opt_P;
$virtual_service = $opt_V;
$remote = $opt_R;

if ($opt_u) {
$weight = $opt_W;
if ($opt_F eq "nat") {
$forwarding = "-m";
} elsif ($opt_F eq "tun") {
$forwarding = "-i";
} else {
$forwarding = "-g";
}

if ($protocol eq "tcp") {
system("$ipvsadm -a -t $virtual_service -r $remote -w $weight $forwarding");
} else {
system("$ipvsadm -a -u $virtual_service -r $remote -w $weight $forwarding");
}
} else {
if ($protocol eq "tcp") {
system("$ipvsadm -d -t $virtual_service -r $remote");
} else {
system("$ipvsadm -d -u $virtual_service -r $remote");
}
};
2.mon.cf(mon的启动配置文件,放到/usr/local/mon/etc):
#
# The mon.cf file
#
#
# global options
#
cfbasedir = /usr/local/mon/etc
alertdir = /usr/local/mon/alert.d
mondir = /usr/local/mon/mon.d
maxprocs = 20
histlength = 100
randstart = 3s

#
# group definitions (hostnames or IP addresses)
#
hostgroup www1 218.106.209.42

hostgroup www2 218.106.209.43

hostgroup www3 218.106.209.44

#
# Web server 1
#
watch www1
service http
interval 10s
monitor http.monitor
period wd {Sun-Sat}
alert mail.alert sun
upalert mail.alert sun
alert lvs.alert -P tcp -V 218.106.209.45:80 -R 218.106.209.42 -W 1 -F tun
upalert lvs.alert -P tcp -V 218.109.209.45:80 -R 218.106.209.42 -W 1 -F tun -u

#
# Web server 2
#
watch www2
service http
interval 10s
monitor http.monitor
period wd {Sun-Sat}
alert mail.alert sun
upalert mail.alert sun
alert lvs.alert -P tcp -V 218.106.209.45:80 -R 218.106.209.43 -W 1 -F tun
upalert lvs.alert -P tcp -V 218.106.209.45:80 -R 218.106.209.43 -W 1 -F tun -u

#
# Web server 3
#
watch www3
service http
interval 10s
monitor http.monitor
period wd {Sun-Sat}
alert mail.alert sun
upalert mail.alert sun
alert lvs.alert -P tcp -V 218.106.209.45:80 -R 218.106.209.44 -W 1 -F tun
upalert lvs.alert -P tcp -V 218.106.209.45:80 -R 218.106.209.44 -W 1 -F tun -u
第三步:HeartBeat的安装(heartbeat的安装需要redhat v7.2的全部安装):
HearBeat需要安装在lvs和lvs2的两台集群器上面:
heartbeat-0.4.9.1.tar.gz
步骤:
tar xvzf heartbeat-0.4.0.1.tar.gz
cd heartbeat-0.4.0.1
make
make install
(没有问题的话,heartbeat安装完毕)
测试串口线路:(假设安装在com1)
在一台机器上面执行:
cat < /dev/ttyS0
在另外的机器上面执行:
echo hello > /dev/ttyS0
如果,在执行cat的机器上面看到hello的字样,然后将两台机器执行的命令翻转过来测试,如果一切正常,那么说明,com1的串口线路没有故障
heartbeat的配置文件:(两台机器都要完成)
配置/etc/hosts(根据自己的情况调整)
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 lvs localhost.localdomain localhost
218.106.209.42 web1
218.106.209.43 web2
218.106.209.44 web3
218.106.209.41 lvs
218.106.209.46 lvs2
然后是:ha.cf(/etc/ha.d/ha.cf)
#
# There are lots of options in this file. All you have to have is a set
# of nodes listedJ {"node ...}
# and one of {serial, udp, or mcast}
#
#
# Note on logging:
# If any of debugfile, logfile and logfacility are defined then they
# will be used. If debugfile and/or logfile are not defined and
# logfacility is defined then the respective logging and debug
# messages will be loged to syslog. If logfacility is not defined
# then debugfile and logfile will be used to log messges. If
# logfacility is not defined and debugfile and/or logfile are not
# defined then defaults will be used for debugfile and logfile as
# required and messages will be sent there.
#
# File to wirte debug messages to
debugfile /var/log/ha-debug
#
#
# File to write other messages to
#
logfile /var/log/ha-log
#
#
# Facility to use for syslog()/logger
#
logfacility local0
#
#
# keepalive: how many seconds between heartbeats
#
keepalive 2
#
# deadtime: seconds-to-declare-host-dead
#
deadtime 10
#
#
# Very first dead time (initdead)
#
# On some machines/OSes, etc. the network takes a while to come up
# and start working right after youve been rebooted. As a result
# we have a separate dead time for when things first come up.
# It should be at least twice the normal dead time.
#
#initdead 120
#
# hopfudge maximum hop count minus number of nodes in config
#hopfudge 1
#
# serial serialportname ...
serial /dev/ttyS0
#
#
# Baud rate for serial ports...
#
baud 19200
#
# What UDP port to use for communication?
#
udpport 694
#
# What interfaces to heartbeat over?
#
udp eth0
#
# Set up a multicast heartbeat medium
# mcast [dev] [mcast group] [port] [ttl] [loop]
#
# [dev] device to send/rcv heartbeats on
# [mcast group] multicast group to join (class D multicast address
# 224.0.0.0 - 239.255.255.255)
# [port] udp port to sendto/rcvfrom (no real reason to differ
# from the port used for broadcast heartbeats)
# [ttl] the ttl value for outbound heartbeats. this effects
# how far the multicast packet will propagate. (0-255)
# [loop] toggles loopback for outbound multicast heartbeats.
# if enabled, an outbound packet will be looped back and
# received by the interface it was sent on. (0 or 1)
#
#
mcast eth0 225.0.0.1 694 1 1
#
# Watchdog is the watchdog timer. If our own heart doesnt beat for
# a minute, then our machine will reboot.
#
watchdog /dev/watchdog
#
# "Legacy" STONITH support
# Using this directive assumes that there is one stonith
# device in the cluster. Parameters to this device are
# read from a configuration file. The format of this line is:
#
# stonith <stonith_type> <configfile>
#
# NOTE: it is up to you to maintain this file on each node in the
# cluster!
#
#stonith baytech /etc/ha.d/conf/stonith.baytech
#
# STONITH support
# You can configure multiple stonith devices using this directive.
# The format of the line is:
# stonith_host <hostfrom> <stonith_type> <params...>
# <hostfrom> is the machine the stonith device is attached
# to or * to mean it is accessible from any host.
# <stonith_type> is the type of stonith device (a list of
# supported drives is in /usr/lib/stonith.)
# <params...> are driver specific parameters. To see the
# format for a particular device, run:
# stonith -l -t <stonith_type>
#
#
# Note that if you put your stonith device access information in
# here, and you make this file publically readable, youre asking
# for a denial of service attack ;-)
#
#
#stonith_host * baytech 10.0.0.3 mylogin mysecretpassword
#stonith_host ken3 rps10 /dev/ttyS1 kathy 0
#stonith_host kathy rps10 /dev/ttyS1 ken3 0
#
# Tell what machines are in the cluster
# node nodename ... -- must match uname -n
node lvs
node lvs2
再次是:haresources(/etc/ha.d/haresources)
lvs 218.106.209.45 lvs
然后是:authkeys(/etc/ha.d//etc/ha.d/)注意:authkeys的属性必须是:600(用chmod 600 authkeys调整)
#auth 1
#1 crc
#2 sha1 HI!
#3 md5 Hello!
auth 1
1 crc
还有:lvs(/etc/rc.d/init.d/)
#!/bin/sh

#
# You probably want to set the path to include
# nothing but local filesystems.
#
PATH=/bin:/usr/bin:/sbin:/usr/sbin
export PATH

IPVSADM=/sbin/ipvsadm

case "$1" in
start)
if [ -x $IPVSADM ]
then
echo 1 > /proc/sys/net/ipv4/ip_forward
$IPVSADM -A -t 218.106.209.45:80
$IPVSADM -a -t 218.106.209.45:80 -r 218.106.209.42 -i
$IPVSADM -a -t 218.106.209.45:80 -r 218.106.209.43 -i
$IPVSADM -a -t 218.106.209.45:80 -r 218.106.209.44 -i
fi
;;
stop)
if [ -x $IPVSADM ]
then
$IPVSADM -C
fi
;;
*)
echo "Usage: lvs {start|stop}"
exit 1
esac

exit 0
最后是:mon(/etc/rc.d/init.d)
#!/bin/sh

#
# the mon start/stop shell code
#
#

case "$1" in
start)
/usr/local/mon/mon -B /usr/local/mon/etc &
;;
stop)
killall mon
;;
*)
echo "Usage: mon {start|stop}"
exit 1
esac

exit 0
第四步:安装NFS
相对简单,但是对于安全配置如下:
NFS服务器配置:
NFS服务器安装在lvs2上面
打开:portmap/nfs选项(setup中)
编辑:/etc/hosts.deny
portmap:ALL
编辑:/etc/hosts.allow
portmap:218.106.209.42
portmap:218.106.209.43
portmap:218.106.209.44
编辑:/etc/hosts
同上描述。
NFS客户端配置:
NFS客户端安装在www服务器上面:
打开:portmap选项(setup中)
编辑:/etc/hosts.deny
portmap:ALL
编辑:/etc/hosts.allow
portmap:218.106.209.42
portmap:218.106.209.43
portmap:218.106.209.44
编辑:/etc/rc.d/rc.local
加入:/bin/mount -t nfs 218.106.209.42:/session /tmp
总结:
heartbeat/mon/lvs需要安装在:lvs和lvs2上面
www服务器只需要安装新内核和打开portmap服务即可


--------------------------------------------------------------------------------

相关文章
Elasticsearch之配置详解 2017/6/5 9:18:45
Elasticsearch之配置详解 2017/6/5 9:18:38
ElasticSearch的部署、同步与调优 2017/6/5 9:16:20
干货 | 常见的大数据术语表(中英对照) 2017/1/24 16:15:22
Linux内核参数调优 2015/10/9 10:58:10
APACHE+TOMCAT的简单HA配置 2015/10/9 10:50:48
TCP/UDP常见端口参考 2013/5/23 15:26:35
iscsi 配置笔记 2013/4/27 10:53:04
MySQL Cluster开发环境简明部署 2012/3/28 13:35:11
搭建类似Amazon EC2的私有云计算平台 2012/3/20 16:09:38
使用apache+resin实现负载均衡设置 2009/3/27 12:26:46
apache + resin的多机部署方案实现方法 2009/3/27 12:25:47
HP小型机的信息的命令集 2008/5/15 18:26:59
Cisco IOS Cookbook 中文精简版第九章 BGP 2007/3/28 9:01:56
实现四台Web服务器的负载均衡 2006/5/8 19:32:09
在可扩展的网络中实施BGP 2006/3/30 22:16:03
BGP路由反射(RFC2796) 2006/3/30 21:51:06
HP小型机的信息的命令集 2006/1/6 14:33:13
Solaris 8.0 +Oracle 9.2.0 +VRTSvcs 双机安装文档 2005/12/2 15:53:28
WINDOWS 2000 ADVANCED SERVER群集安装 2005/12/2 15:49:31
如何部署BackupExec Oracle Agent 2005/12/2 15:12:44
BackupExec 10 SQL Agent使用指南 2005/12/2 15:10:32
Cisco IOS命名规则 2005/11/26 20:11:25
安全保障体系 2005/8/18 8:59:54
网络安全方面的专业词汇大全 2005/6/20 10:33:11
Solaris 系统管理命令及相关技术中英文对照 2005/4/13 10:36:48
如何配置RAID磁盘阵列之二 2004/11/24 9:09:16
一个安全保障体系的整套解决方案 2004/11/9 14:21:27
计算机常见故障代码(硬件) 2004/10/21 8:41:20
群集服务指南 2004/9/15 13:22:48
实现四台Web服务器的负载均衡 2004/9/1 11:15:32
29&3548配置方法 2004/8/29 12:02:51
实现四台服务器的负载均衡 2004/8/13 13:01:59
一种 Web 应用程序级别的 Cluster 实现技术 2004/7/14 16:23:12
Top Ten Tips for Secure Testing 2004/5/13 7:59:22
Ghost V8.0 使用详解 2004/3/30 14:52:47
Win2000命令全集 2004/3/16 17:07:20
双机硬件及w2k做群集解决方案 2004/2/21 0:24:25
网络端口及其详解分页 2004/2/17 9:03:12
CCNP路由部分精华 2004/1/6 11:10:54
switch命令大全 2003/11/25 9:06:50
Symantec.Ghost.8.0企业版使用全攻略 2003/11/4 8:57:43
用windows2000实现双机热备 2003/10/22 8:10:28
Switching 命令大全 2003/10/13 17:59:51
基于Linux的集群系统 2003/9/1 18:32:33
基于Linux的集群系统(二) 2003/9/1 18:27:32
基于linux的集群系统(一) 2003/9/1 18:25:40
常见端口表汇总 2003/9/1 10:31:24
Win2000命令全集 2003/8/27 14:06:40
在RedHat下建立群集 2003/8/12 9:13:09
Redhat advance server 2.1集群的安装与管理 2003/8/12 8:21:31
思科产品常见问题一百问 2003/7/30 14:17:30
Solaris 问答集 2003/7/19 19:06:09
安装OpenSSH 在SUN Solaris 8 for SPARC[ZT] 2003/7/15 8:29:08
Switching 命令大全-转 2003/7/13 19:06:01
[转帖]实现四台服务器的负载均衡 2003/7/13 19:03:25
转载cisco3524的管理vlan配置 2003/7/11 22:16:09
路由的中文笔记 2003/7/8 12:22:30
Linux下实现集群负载均衡 2003/5/25 21:04:02
Linux 学习手册--linux命令大全 2003/5/14 10:04:59
Linux 指令大全 2003/5/5 13:29:23
linux命令大全 2003/3/31 10:49:22


感性空间
设计&运维
网络技术
休闲娱乐
NetFilter
linux&Unix
网络安全
程序空间
软件考试
RFC&ISO
规划&规范
虚拟&存储
Apple技巧
云计算&大数据



文章搜索



站内搜索