I've decided to play a little bit with Proxmox, this web-interface cluster distro, that allows easly to migrate VM among nodes. Proxmox is basically based on Redhat Cluster. The Proxmox suite miss a tool for handling the cluster resources (at least I didn't find it); still, the container part is based on OpenVZ, outdated by LXC. Despite this, the worst thing now is the lack of the documentation for v3.0.
Scenario
I want to build a small dev environment where a new vm (I will use containers actually - CT) will be spawned by developers everytime they need one. Since the software set for those vm will be large (database, backend, general tools ..) I will create a CT template. The other machines thus, will be cloned from this one
Index
In this Proxmox experiment I will configure in HA:
1) a service IP, on which reaching the cluster
2) a CT template, from where originate the CT clones
4) a script for the CT creation/spawing automation
The document assumes that a 2nodes Proxmox "cluster", will be already setup and running. Briefly the setup here is
* Cluster name: vicinet
* Node01: 'proxmox' - 192.168.1.13
* Node02: 'proxmox2' - 192.168.1.14
* Storage: NFS /mnt/pve/shared on a separate node
Configure FENCING first
In order to proceed with any HA resource, we need to configure the fencing. Fencing is basically the procedure by which you exclude a node from the cluster in case of malfunction, sending to this latter a shutdown/reboot command. Such commands could be sent by Apc ups as well as an IPMI device and so on. The software which will trigger the device (ups, ipmi..) to send commands is called Agent
In the Redhat Cluster, the package 'fence-agent' will provides several fence agents that will probably cover your device too (apc, cisco, ibmblade .. but also
ipmi, ilo and many more). On Proxmox the package is called 'fence-agents-pve'
NOTE: Proxmox works with VMs (kvm) and Containers (OpenVZ). Since my experiment here was just to test the creation/automation of Linux Containers I could quietly setup Proxmox in VirtualBox
For my fencing setup, I found this fence_vbox; you can also find agents for vmware, virsh, in the case
Enable fencing on Debian on each node
Uncomment the last line of the file /etc/default/redhat-cluster-pve
# this file is sourced by the following init scripts: # /etc/init.d/cpglockd # /etc/init.d/cman # /etc/init.d/rgmanager FENCE_JOIN="yes"
reload these services in this order and *outside the /etc/pve directory*
/etc/init.d/rgmanager stop /etc/init.d/cman reload /etc/init.d/rgmanager start /etc/init.d/pve-cluster restart
(I did an alias in .bashrc, like 'alias restart-cluster='...')
Test if the fencing was joined
proxmox:~# fence_tool ls
fence domain
member count 2
victim count 0
victim now 0
master nodeid 2
wait state none
members 1 2
Edit the cluster.conf file with the proper procedure (cluster.conf.new ..) adding the fencing setup; I will just report mine, just to have an example:
<clusternodes>
<clusternode name="proxmox" nodeid="1" votes="1">
<fence>
<method name="1">
<device action="reboot" ipaddr="Proxmox" name="VBox"/>
</method>
</fence>
</clusternode>
<clusternode name="proxmox2" nodeid="2" votes="1">
<fence>
<method name="1">
<device action="reboot" ipaddr="Proxmox2" name="VBox"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice agent="fence_vbox" login="zmo" name="VBox" host="192.168.1.65"/>
</fencedevices>
* 'proxmox' 'proxmox2', are the hostnames of the proxmox nodes
* 'Proxmox' 'Proxmox2', are the GUEST names in VirtualBox
* 'VBox', names the fence configuration to use
* '192.168.1.65', is the IP of the VirtualBox HOST
As said, this is just an example of fencing configuration in Proxmox; test and use your own agent and values
Add a cluster IP
This will be the cluster IP, that will migrate between node in case of fails. Configure this migration pattern, writing a first <failoverdomain> section
<rm>
<failoverdomains>
<failoverdomain name="vicinet" ordered="1" restricted="0">
<failoverdomainnode name="proxmox" priority="1"/>
<failoverdomainnode name="proxmox2" priority="2"/>
</failoverdomain>
</failoverdomains>
</rm>
A resource assigned to this failoverdomain, will have 'proxmox' as preferred node and in case proxmox get failed will be migrate on 'proxmox2'. So, a brief explanation
* ordered: assign a preference order for the resource by priority (1=max, 100=min)
* restricted: the resource could not migrate (or even started manually) if the nodes defined are not available. We turn 'restricted' off (0)
Add the IP configuration:
<rm>
<failoverdomains>
<failoverdomain name="vicinet" ordered="1" restricted="0">
<failoverdomainnode name="proxmox" priority="1"/>
<failoverdomainnode name="proxmox2" priority="2"/>
</failoverdomain>
</failoverdomains>
<resources>
<ip address="192.168.1.100" monitor_link="5"/>
</resources>
<service autostart="1" domain="vicinet" name="ip_cluster" recovery="relocate">
<ip ref="192.168.1.100"/>
</service>
</rm>
* monitor_link is just a monitor for the link state, just for that ip
* relocate means that the service will move on automatically on the preferred node ('proxmox' in this case) when, for example, this will be back online from a down
* autostart, the service is started by the cluster itself
Reload the cluster and see that you have a new resource on your cluster (use clustat
Cluster Status for vicinet @ Sun Dec 29 10:54:32 2013 Member Status: Quorate Member Name ID Status ------ ---- ---- ----- proxmox 1 Online, Local, rgmanager proxmox2 2 Online, rgmanager Service Name Owner (Last) State ------- ---- ----- ------ ----- service:ip_cluster proxmox started
Also see the new interface on the node proxmox (use 'ip tools' and not 'ifconfig')
3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether 01:E0:27:29:3e:fc brd ff:ff:ff:ff:ff:ff
inet 192.168.1.13/24 brd 192.168.1.255 scope global vmbr0
inet 192.168.1.100/24 scope global secondary
