Vagrant环境搭建

发表于 | 分类于 | 阅读次数:

一般开发时，我们基本上都是在windows或mac上开发，测试与正式环境一般都使用centos系统。开发环境搭建起来非常费时，并且经常出现开发时功能是正常的，但测试环境和生产环境不正常的情况。并且有新同事来时，还得从头搭建相关的环境，费力又费时。我们可以通过vagrant很好地解决这个问题。

介绍

Vagrant是一个基于Ruby的工具，用于创建和部署虚拟化开发环境。它使用Oracle的开源VirtualBox虚拟化系统，使用 Chef创建自动化虚拟环境。

安装

这边以mac为例介绍一下vagrant的安装与配置过程，安装过程很简单，直接下载以下两个软件安装即可。

vagrant

1
wget https://releases.hashicorp.com/vagrant/1.9.1/vagrant_1.9.1.dmg

VirtualBox

1
http://download.virtualbox.org/virtualbox/5.1.10/VirtualBox-5.1.10-112026-OSX.dmg

配置

以centos 7为例介绍一下vagrant的配置。

环境准备

下载vagrant-centos-7.2.box

1
wget https://github.com/CommanderK5/packer-centos-template/releases/download/0.7.2/vagrant-centos-7.2.box

由于是国外的网站，下载很慢，其实我们可以自己创建box，具体制作方法请参考使用packer制作vagrant-box.md

初始化

1
2
3
4
5
6
sudo mkdir -p /Vagrant/boxes/centos-7.2
sudo cp -a vagrant-centos-7.2.box /Vagrant/boxes/centos-7.2
cd /Vagrant/boxes/centos-7.2
sudo vagrant box add centos-7.2 vagrant-centos-7.2.box
sudo vagrant box list
sudo vagrant init centos-7.2

配置Vagrantfile

假设我们需要两台虚拟机

主机IP 主机名称 内存
192.168.10.6 k8s-master 1024m
192.168.10.6 k8s-node1 512m 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
# -*- mode: ruby -*-
# vi: set ft=ruby :

# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|

  #config.vm.box = "centos-7.2"
  #config.vm.hostname = "mydocker"
  #config.vm.network "private_network", ip: "192.168.10.9"
#
  #config.vm.synced_folder "/data/docker/registry", "/docker/registry"
  #config.vm.synced_folder "/data/docker/works", "/docker/works"
#
  #config.vm.provider "virtualbox" do |vb|
  #  #vb.gui = true
  #  vb.memory = "2048"
  #end
  #
  #config.vm.provision "shell", run: "always", inline: <<-SHELL
  #  systemctl restart network
  #SHELL
  #config.vm.provision "shell", path: "script.sh"

  config.vm.define :k8s_master do |k8s_master|
    k8s_master.vm.box = "centos-7.2"
    k8s_master.vm.hostname = "k8s-master"
    k8s_master.vm.network "private_network", ip: "192.168.10.6"
    k8s_master.vm.synced_folder "/data/docker/registry", "/docker/registry"
    k8s_master.vm.synced_folder "/data/docker/works", "/docker/works"
    k8s_master.vm.synced_folder "/data/docker/k8s", "/docker/k8s"
    k8s_master.vm.provider "virtualbox" do |vb|
      #vb.gui = true
      vb.memory = "1024"
    end
    k8s_master.vm.provision "shell", run: "always", inline: <<-SHELL
      systemctl restart network
    SHELL
    k8s_master.vm.provision "shell" do |s|
      s.path = "script.sh"
      s.args = ["--bip=10.1.10.1/24"]
    end
    #k8s_master.vm.provision "shell", path: "script.sh"
  end


  config.vm.define :k8s_node1 do |k8s_node1|
    k8s_node1.vm.box = "centos-7.2"
    k8s_node1.vm.hostname = "k8s-node1"
    k8s_node1.vm.network "private_network", ip: "192.168.10.7"
    k8s_node1.vm.synced_folder "/data/docker/registry", "/docker/registry"
    k8s_node1.vm.synced_folder "/data/docker/works", "/docker/works"
    k8s_node1.vm.synced_folder "/data/docker/k8s", "/docker/k8s"
    k8s_node1.vm.provider "virtualbox" do |vb|
      #vb.gui = true
      vb.memory = "1024"
    end
    k8s_node1.vm.provision "shell", run: "always", inline: <<-SHELL
      systemctl restart network
    SHELL
    k8s_node1.vm.provision "shell" do |s|
      s.path = "script.sh"
      s.args = ["--bip=10.1.20.1/24"]
    end
  end


  #config.vm.define :k8s_node2 do |k8s_node2|
  #  k8s_node2.vm.box = "centos-7.2"
  #  k8s_node2.vm.hostname = "k8s-node2"
  #  k8s_node2.vm.network "private_network", ip: "192.168.10.8"
  #  k8s_node2.vm.synced_folder "/data/docker/registry", "/docker/registry"
  #  k8s_node2.vm.synced_folder "/data/docker/works", "/docker/works"
  #  k8s_node2.vm.synced_folder "/data/docker/k8s", "/docker/k8s"
  #  k8s_node2.vm.provider "virtualbox" do |vb|
  #    #vb.gui = true
  #    vb.memory = "512"
  #  end
  #  k8s_node2.vm.provision "shell", run: "always", inline: <<-SHELL
  #    systemctl restart network
  #  SHELL
  #  k8s_node2.vm.provision "shell" do |s|
  #    s.path = "script.sh"
  #    s.args = ["--bip=10.1.30.1/24"]
  #  end
  #end
  
end

定义shell脚本

编写script.sh脚本，安装一些基本的软件，具体可以参考：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
#!/bin/sh
#http://www.360doc.com/content/14/1125/19/7044580_428024359.shtml
#http://blog.csdn.net/54powerman/article/details/50684844
#http://c.biancheng.net/cpp/view/2739.html
echo "scripting......"

filepath=/vagrant
bip=$1
hostname=$2

if [[ "$hostname" != "" ]]; then
  hostnamectl --static set-hostname $hostname
  sysctl kernel.hostname=$hostname
fi

#关闭内核安全(如果是vagrant方式，第一次完成后需要重启vagrant才能生效。)
sed -i 's;SELINUX=.*;SELINUX=disabled;' /etc/selinux/config
setenforce 0
getenforce

cat /etc/NetworkManager/NetworkManager.conf|grep "dns=none" > /dev/null
if [[ $? != 0 ]]; then
  echo "dns=none" >> /etc/NetworkManager/NetworkManager.conf
  systemctl restart NetworkManager.service
fi

systemctl disable iptables
systemctl stop iptables
systemctl disable firewalld
systemctl stop firewalld

ln -sf /usr/share/zoneinfo/Asia/Chongqing /etc/localtime

#logined limit
cat /etc/security/limits.conf|grep 100000 > /dev/null
if [[ $? != 0 ]]; then
cat >> /etc/security/limits.conf  << EOF
*               soft    nofile             100000
*               hard    nofile             100000
*               soft    nproc              100000
*               hard    nproc              100000
EOF
fi

sed -i 's;4096;100000;g' /etc/security/limits.d/20-nproc.conf

#systemd service limit
cat /etc/systemd/system.conf|egrep '^DefaultLimitCORE' > /dev/null
if [[ $? != 0 ]]; then
cat >> /etc/systemd/system.conf << EOF
DefaultLimitCORE=infinity
DefaultLimitNOFILE=100000
DefaultLimitNPROC=100000
EOF
fi

cat /etc/sysctl.conf|grep "net.ipv4.ip_local_port_range" > /dev/null
if [[ $? != 0 ]]; then
cat >> /etc/sysctl.conf  << EOF
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.ip_forward = 1
EOF
sysctl -p
fi

su - root -c "ulimit -a"

#echo '192.168.10.6 k8s-master
#192.168.10.7   k8s-node1
#192.168.10.8   k8s-node2' >> /etc/hosts

##sed -i 's;en_GB;zh_CN;' /etc/sysconfig/i18n

#yum -y install gcc kernel-devel
mv -f /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
#wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.163.com/.help/CentOS7-Base-163.repo
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

tee /etc/yum.repos.d/docker.repo <<-'EOF'
[docker]
name=Docker Repository
baseurl=http://mirrors.aliyun.com/docker-engine/yum/repo/main/centos/7/
enabled=1
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/docker-engine/yum/gpg
EOF
#
#tee /etc/yum.repos.d/k8s.repo <<-'EOF'
#[k8s-repo]
#name=kubernetes Repository
##baseurl=https://rpm.mritd.me/centos/7/x86_64
##baseurl=file:///docker/works/yum
#baseurl=http://www.gcalls.cn/yum
#enabled=1
#gpgcheck=1
##gpgkey=https://cdn.mritd.me/keys/rpm.public.key
##gpgkey=file:///docker/works/yum/gpg
#gpgkey=http://www.gcalls.cn/yum/gpg
#EOF

yum -y install epel-release

yum clean all
yum makecache

#yum -y install createrepo rpm-sign rng-tools yum-utils 
yum -y install bind-utils bridge-utils ntpdate setuptool iptables system-config-securitylevel-tui system-config-network-tui \
 ntsysv net-tools lrzsz telnet lsof vim dos2unix unix2dos zip unzip

#install docker-compose-----------------------------------------------

rpm -e docker-1.10.3-59.el7.centos.x86_64 \
 docker-common-1.10.3-59.el7.centos.x86_64 \
 container-selinux-1.10.3-59.el7.centos.x86_64 > /dev/null 2>&1

yum install docker-engine -y

yum -y install python2-pip
pip install -U docker-compose
##yum install -y etcd kubernetes
##sed -i "s;^ExecStart=/usr/bin/dockerd$;ExecStart=/usr/bin/dockerd --registry-mirror=https://3gbbfq7n.mirror.aliyuncs.com;" /usr/lib/systemd/system/docker.service
#
#sed -i "s;^ExecStart=/usr/bin/dockerd$;ExecStart=/usr/bin/dockerd ${bip} --live-restore \
sed -i "s;^ExecStart=/usr/bin/dockerd$;ExecStart=/usr/bin/dockerd \
--registry-mirror=http://3fecfd09.m.daocloud.io \
--registry-mirror=https://3gbbfq7n.mirror.aliyuncs.com \
--registry-mirror=http://zhaoxunyong.m.alauda.cn;" \
/usr/lib/systemd/system/docker.service

#mkdir -p /etc/systemd/system/docker.service.d
#cat >> /etc/systemd/system/docker.service.d/http-proxy.conf  << EOF
#[Service]
#Environment="HTTP_PROXY=http://thenorth.f.ftq.me:52579"
#Environment="HTTPS_PROXY=http://thenorth.f.ftq.me:52579"
#Environment="NO_PROXY=127.0.0.1,localhost,10.0.0.0/8,172.0.0.0/8,192.168.0.0/16,*.zerofinance.net"
#EOF
#
#systemctl daemon-reload
#systemctl show --property=Environment docker

systemctl restart docker
systemctl enable docker

#cd /docker/works/images/Vagrant环境搭建/k8s/
#./importK8s.sh
#
#docker load -i /docker/works/images/Vagrant环境搭建/others/redis-master.tar 
#docker load -i /docker/works/images/Vagrant环境搭建/others/guestbook-redis-slave.tar 
#docker load -i /docker/works/images/Vagrant环境搭建/others/guestbook-php-frontend.tar
#
#docker load -i /docker/works/images/Vagrant环境搭建/k8s/tar/quagga.tar
#docker run -itd --name=router --privileged --net=host index.alauda.cn/georce/router
#docker start `docker ps -a |grep 'index.alauda.cn/georce/router'|awk '{print $1}'`

#install docker-engine end-----------------------------------------------

#mkdir /usr/local/java > /dev/null 2>&1 
#cd $filepath/files
#tar zxf jdk-8u111-linux-x64.tar.gz -C /usr/local/java/
#ln -sf /usr/local/java/jdk1.8.0_111 /usr/local/java/jdk
#
#cat /etc/profile|grep "JAVA_HOME" > /dev/null
#if [[ $? != 0 ]]; then
#cat >> /etc/profile  << EOF
# export JAVA_HOME=/usr/local/java/jdk
# export PATH=\$JAVA_HOME/bin:\$PATH
#EOF
# source /etc/profile
#fi#

centos6的脚本会有所不同，可以参考

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
#!/bin/sh
#http://www.360doc.com/content/14/1125/19/7044580_428024359.shtml
echo "scripting......"

filepath=/vagrant

sed -i 's;en_GB;zh_CN;' /etc/sysconfig/i18n

yum -y install yum-fastestmirror

if [ ! -f "/etc/yum.repos.d/CentOS-Base.repo.from.aliyun.backup" ]; then 
  mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.from.aliyun.backup
  #wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.163.com/.help/CentOS6-Base-163.repo
  wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo
  yum clean all
  yum makecache
fi 

#wget http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-8.noarch.rpm
rpm -ivh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

#yum install gcc gcc-c++ make bind-untils libevent libevent-devel sysstat autoconf \
# curl curl-devel -y
#yum install gcc gcc-c++ kernel-devel make autoconf libevent libevent-devel bind-untils

yum -y install ntpdate net-tools setuptool iptables system-config-securitylevel-tui system-config-network-tui \
 ntsysv net-tools lrzsz telnet lsof dos2unix unix2dos zip unzip vim curl curl-devel
 
#升级内核为3.10
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
#cd /etc/yum.repos.d/
rpm -ivh http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm
yum --enablerepo=elrepo-kernel install kernel-lt kernel-lt-devel -y
#修改grub.conf文件的default=0
sed -i 's;^default=.*;default=0;' /etc/grub.conf
sed -i 's;^default=.*;default=0;' /boot/grub/grub.conf 

/opt/VBoxGuestAdditions-4.3.30/init/vboxadd setup
#reboot uname -r

#chkconfig --level 35 memcached on
#service iptables stop
#chkconfig --level 35 iptables on

#关闭内核安全(如果是vagrant方式，第一次完成后需要重启vagrant才能生效。)
sed -i 's;SELINUX=.*;SELINUX=disabled;' /etc/selinux/config
setenforce 0
getenforce

ln -sf /usr/share/zoneinfo/Asia/Chongqing /etc/localtime

cat /etc/security/limits.conf|grep 65535 > /dev/null
if [[ $? != 0 ]]; then
cat >> /etc/security/limits.conf  << EOF
    *               soft    nofile             65535
    *               hard    nofile             65535
    *               soft    nproc              65535
    *               hard    nproc              65535
EOF
fi

cat /etc/sysctl.conf|grep "net.ipv4.ip_local_port_range" > /dev/null
if [[ $? != 0 ]]; then
cat >> /etc/sysctl.conf  << EOF
  net.ipv4.tcp_fin_timeout = 30
  net.ipv4.tcp_keepalive_time = 300
  net.ipv4.tcp_tw_reuse = 1
  net.ipv4.tcp_tw_recycle = 1
  net.ipv4.ip_local_port_range = 1024 65535
EOF
  sysctl -p
fi

#mkdir /usr/local/java > /dev/null 2>&1 
#cd $filepath/files
#tar zxf jdk-8u91-linux-x64.tar.gz -C /usr/local/java/
#ln -sf /usr/local/java/jdk1.8.0_91 /usr/local/java/jdk
#
#cat /etc/profile|grep "JAVA_HOME" > /dev/null
#if [[ $? != 0 ]]; then
#cat >> /etc/profile  << EOF
#export JAVA_HOME=/usr/local/java/jdk
#export PATH=\$JAVA_HOME/bin:\$PATH
#EOF
# source /etc/profile
#fi
#
#yum -y install libevent libevent-devel zlib zlib-devel pcre pcre-devel openssl openssl-devel

基本命令

启动：

1
sudo vagrant up

第一次时初始化环境，会调用script.sh脚本。如果需要每次都调用script.sh脚本，需要为provision指定run:”always”属性
启动时运行，在启动命令加 –provision 参数,适用于 vagrant up 和 vagrant reload
vm启动状态时，执行 vagrant provision 命令。

第一次启动后，运行sestatus命令显示的还是enable状态，需要重启一下vagrant才会显示disabled。

关闭：

1
sudo vagrant halt

重启：

1
sudo vagrant reload

销毁：

1
sudo vagrant destroy

注意：此操作会删除已创建虚拟机中的所有内容，操作请小心！！！

登录虚拟机：

1
sudo vagrant ssh

如果有多个虚拟机时，后面需要加上对应的名称

1
sudo vagrant ssh k8s_master

导出:
将配置好的环境打包给其他同事使用

1
vagrant package --output centos-export.box

卸载

1
2
3
sudo rm -rf /Applications/Vagrant
sudo rm -f /usr/local/bin/vagrant
sudo pkgutil --forget com.vagrant.vagrant

快照

安装

1
vagrant plugin install vagrant-vbox-snapshot

操作命令

参考http://blog.huatai.me/2015/12/03/use-vagrant-snapshot-create-backup/
支持的参数如下：

1
2
3
4
5
vagrant snapshot take [vm-name] <SNAPSHOT_NAME>   # take snapshot, labeled by NAME
vagrant snapshot list [vm-name]                   # list snapshots
vagrant snapshot back [vm-name]                   # restore last taken snapshot
vagrant snapshot delete [vm-name] <SNAPSHOT_NAME> # delete specified snapshot
vagrant snapshot go [vm-name] <SNAPSHOT_NAME>     # restore specified snapshot

创建快照:

1
vagrant snapshot take k8s_master k8s_master_snapshot

虚构机的名称可以通过以下命令查看：

1
2
3
4
5
6
7
8
9
10
vagrant status
Current machine states:

k8s_master                running (virtualbox)
k8s_node1                 running (virtualbox)
k8s_node2                 running (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.

查看快照列表:

1
vagrant snapshot list

含有多个虚拟机时，需要指定对应的名称：

1
vagrant snapshot list k8s_master

从指定快照中恢复:

1
vagrant snapshot go "k8s_master_snapshot"

删除一个快照:

1
vagrant snapshot delete "k8s_master_snapshot"

异常解决

/sbin/mount.vboxsf: mounting failed with the error: No such device的解决办法：

1
2
3
yum -y install gcc kernel-devel make
/opt/VBoxGuestAdditions-5.1.10/init/vboxadd setup
sudo vagrant reload

Hyper-V

如果有在windows 10安装docker的话，只能使用Hyper-V，VirtualBox启动不了，总结一下使用Hyper-V的使用过程：
(Hyper-V需要在管理员模式下运行才行)

下载hyperv对应的box文件

1
2
3
4
#centos 7.3
wget https://vagrantcloud.com/centos/boxes/7/versions/1707.01/providers/hyperv.box
#centos 7.4
wget https://vagrantcloud.com/centos/boxes/7/versions/1708.01/providers/hyperv.box

编写Vagrantfile

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
config.vm.define :k8s_master do |k8s_master|
  k8s_master.vm.box = "centos-7.3"
  k8s_master.vm.hostname = "k8s-master"
  k8s_master.vm.network "private_network", ip: "192.168.10.6"
  #k8s_master.vm.synced_folder "d:/data/docker", "/docker"
  k8s_master.vm.provider "hyperv" do |vb|
    #vb.gui = true
    vb.memory = "1024"
    vb.cpus = 1
  end
  k8s_master.vm.provision "shell", run: "always", inline: <<-SHELL
    systemctl restart network
  SHELL
  k8s_master.vm.provision "shell" do |s|
    s.path = "script.sh"
    #s.args = ["k8s-master", "--bip=10.1.10.1/24"]
    s.args = ["k8s-master"]
  end
  #k8s_master.vm.provision "shell", path: "script.sh"
end

配置Hyper-V网络

参考https://quotidian-ennui.github.io/blog/2016/08/17/vagrant-windows10-hyperv/
如果不配置网络的话，Hyper-V连接不到网络：

也可以手动添加：
hyperv.png

好像Hyper-V不能用private_network，只能用public_network…

注意：
对应的box没有wget，可能需要在script.sh脚本中加入：

1
yum -y install wget

生成Hyper-V box

还在继续研究中…

参考

http://www.vincentguo.cn/default/26.html
http://topmanopensource.iteye.com/blog/2002302
http://blog.csdn.net/54powerman/article/details/50684844
https://my.oschina.net/u/148605/blog/385049