Building a Sun Cluster using Solaris 10, on VMware Server
One of the things I’ve done on my week off is get Sun Cluster working on VMware Server. There’s a few small tricks to it, but generally it hasn’t been as horrible as the first time I did it many months ago on VMware (a 4 node cluster, with panics galore - no fun).
NOTE: To run Sun Cluster on Solaris 10, you will need to be running your VMs as 64-bit guests (Sun Cluster, on Solaris 10, on x86/x64 DOES NOT run on Solaris 10 32-bit - found this out the hard way a long time ago!).
To do 64-bit VMs, only some CPUs are supported. In this case I am using a relatively new AMD 64 x2, and thats perfect for this purpose. In this build I’m using Solaris u3 is because u4 seems to have a few issues on VMware (lots of kernel panics on boot I’ve found), and Sun Cluster 3.1 8/05 (u4), even though Solaris Cluster 3.2 is out, Sun Cluster 3.1u4 is still what most things are certified against. I will build a 3.2 cluster at some point later on.
So, lets start with configuring VMware:
-Configure at least 2 additional host based networks, on Linux you will need to run
vmware-config.pl
I have configured several more (seeing as it is easy to do it all at once).
When it asks you about networking, you want to configure additional host based networks. The scheme I have used is:
172.16.0.0/255.255.255.0
172.16.1.0/255.255.255.0
.
.
172.16.11.0/255.255.255.0
NOTE: What subnets you specify here isn’t that relevant as your host will never talk on these networks, the host to host cluster interconnects will use the interfaces, and will likely use different subnets. Each subnet just needs to be different for each vmnet adapter. You need not use /24’s either, you could go down to a very small subnet size (/30 for example). If this makes no sense to you, do not worry too much, keep going anyway just following what I’ve done above.
Now, create 2 VMs - Typical, Solaris 10 64-bit (will not work if 64-bit is not selected), and in my case I preallocated 32GB of disk.
For each VM:
-Add at least 2 additional ethernet interfaces (e1000gXs) - these will be used for interconnects. Put each one on a different vmnet adapter. I used vmnet1 and vmnet2 in this case. Make sure when you do this, you do the interfaces identically on both VMs as they will need to talk to each other across these interfaces, but nothing else.
-Disable snapshots (for performance)
-Add a single disk in a different location for quorom, to one host only. I added quorom in /virtuals/cluster-disks/mail-store-quorom.vmdk. Quorom should be as small as possible, I believe the smallest disk you can build is ~100meg, so do that (0.1GB)
Boot each node for a tick, then shut it down straight away. This is to build the initial vmx config additional information that is not populated till you bring up a VM (ethernet address for e1000g0, is what we really want to be populated here).
-With each VM shut down, edit the .vmx file for each VM and add in the lines:
Configure shared quorom device:
scsi0:1.present = "TRUE" scsi0:1.fileName = "/virtuals/cluster-disks/mail-store-quorom.vmdk" scsi0.sharedBus = "virtual" disk.locking = "false"
Obviously there’s no need to re-add the filename line for the host you configured the disk on initially.
Next for each VM configure CPU (core) binding - if you don’t do this and you’re using a 64 bit AMD chip, you’ll get some interesting behaviour because the timestamping on each core of these CPUs is different, it messes with Solaris which expects to be on one CPU. Cluster will panic more often if you don’t do this 
processor0.use = "TRUE" processor1.use = "FALSE"
And I do the reverse of the above for the other host’s vmx file:
processor0.use = "FALSE" processor1.use = "TRUE"
-Now kick off a Solaris build using the minimal profile from my last post, or a SUNWCXall (all packages) will do if you don’t mid the extra build time wait and you have disk space up your sleeve. You could of course just do a straight install off the CD, just make sure you use a custom partitioning scheme ensuring there is a512MB /globaldevices slice of your disk.
-Install Sun Cluster from anything Java Enterprise System 2005Q4 or above (JES 5 at time of writing). Obtain it for free fromhttp://www.sun.com/software/javaenterprisesystem/getit.jsp. All the JES’ from (and including) 2005Q4 have Sun Cluster 3.1u4 (that is 8/05). It’s worth noting Sun Cluster 3.2 is out and has been for a while, but I’m not sure how much stuff is certified against it. I will try it out later, for the moment I’ll go with 3.1u4.
Note: there is nearly no initial config when you install Sun Cluster from the JES installer. I noticed in the most recent release of JES (and possibly some previous releases, I’ve missed a few) it asks if you want to allow Sun Cluster to be configured remotely. For simplicity, answer yes. It makes the cluster config very easy from there.
-Add /usr/cluster/bin to your path for convenience on both hosts
-run scinstall on ONE host (/usr/cluster/bin/scinstall if you did not follow the above step).
You’ll get a menu… with the first item where we want to be.
* 1) Install a cluster or cluster node
So select 1, then..
1) Install all nodes of a new cluster
1 again, then yes to continue
Please select from one of the following options:
1) Typical
2) Custom
Select 1
Then select a cluster name, in this case I’ve gone with mail-store-clus as this is to become a cluster of Sun Messaging Server 6.3 Mail Stores
Next you are asked for other nodes in the cluster, in this case the only other node for me is mail-store1, so I type that in
Node name (Control-D to finish): mail-store1 Node name (Control-D to finish):
This is the complete list of nodes:
mail-store0 mail-store1
Is it correct (yes/no) [yes]?
and then ctrl-D, then yes it is correct
Attempting to contact "mail-store1" ... done
Searching for a remote install method ... done
The Sun Cluster framework software is already installed on each of the new nodes of this cluster. And, it is able to complete the configuration process without remote shell access.
Looking good so far! Enter to continue.
Select the first cluster transport adapter for "mail-store0":
1) e1000g1
2) e1000g2
3) Other
Go with 1, then the next transport adapter, 2. NOTE: If you have plumb’d these devices, they will not work. These cards need to be unplumb’d in that case.
Searching for any unexpected network traffic on "e1000g1" ... done
Verification completed. No traffic was detected over a 10 second
sample period.
Next up, quorom. This is why we setup the shared disk earlier:
Do you want to disable automatic quorum device selection (yes/no) [no]?
(go with the default, no)
Is it okay to begin the installation (yes/no) [yes]?
yes, it sure is!
During the installation process, sccheck(1M) is run on each of the
new cluster nodes. If sccheck(1M) detects problems, you can either
interrupt the installation process or check the log files after
installation has completed.
Interrupt the installation for sccheck errors (yes/no) [no]?
default is fine, no
and off we go:
Installation and Configuration
Log file - /var/cluster/logs/install/scinstall.log.630
Testing for "/globaldevices" on "mail-store0" ... done
Testing for "/globaldevices" on "mail-store1" ... done
Starting discovery of the cluster transport configuration.
The following connections were discovered:
mail-store0:e1000g1 switch1 mail-store1:e1000g1
mail-store0:e1000g2 switch2 mail-store1:e1000g2
Completed discovery of the cluster transport configuration.
Started sccheck on "mail-store0".
Started sccheck on "mail-store1".
sccheck completed with no errors or warnings for "mail-store0".
sccheck completed with no errors or warnings for "mail-store1".
Configuring "mail-store1" ... done
Rebooting "mail-store1" ...
And the second node reboots, then the first
Rebooting "mail-store1" ... done
Configuring "mail-store0" ... done
Rebooting "mail-store0" ...
Log file - /var/cluster/logs/install/scinstall.log.630
Rebooting ...
updating /platform/i86pc/boot_archive...this may take a minute
Connection to mail-store0 closed by remote host.
Connection to mail-store0 closed.
Let the first node boot, and you’ll see a bunch of stuff on the console. Don’t stress, it’s (probably) normal. It is normal to see a few errors at first boot.
Let the cluster sort it’s stuff out (give it a couple of minutes) then run scstat to check the status of the cluster. It should look something like:
-bash-3.00$ scstat
------------------------------------------------------------------
-- Cluster Nodes --
Node name Status
--------- ------
Cluster node: mail-store1 Online
Cluster node: mail-store0 Online
------------------------------------------------------------------
-- Cluster Transport Paths --
Endpoint Endpoint Status
-------- -------- ------
Transport path: mail-store1:e1000g2 mail-store0:e1000g2 Path online
Transport path: mail-store1:e1000g1 mail-store0:e1000g1 Path online
------------------------------------------------------------------
-- Quorum Summary --
Quorum votes possible: 3
Quorum votes needed: 2
Quorum votes present: 3
-- Quorum Votes by Node --
Node Name Present Possible Status
--------- ------- -------- ------
Node votes: mail-store1 1 1 Online
Node votes: mail-store0 1 1 Online
-- Quorum Votes by Device --
Device Name Present Possible Status
----------- ------- -------- ------
Device votes: /dev/did/rdsk/d2s2 1 1 Online
------------------------------------------------------------------
-- Device Group Servers --
Device Group Primary Secondary
------------ ------- ---------
-- Device Group Status --
Device Group Status
------------ ------
-- Multi-owner Device Groups --
Device Group Online Status
------------ -------------
------------------------------------------------------------------
------------------------------------------------------------------
-- IPMP Groups --
Node Name Group Status Adapter Status
--------- ----- ------ ------- ------
IPMP Group: mail-store1 sc_ipmp0 Online e1000g0 Online
IPMP Group: mail-store0 sc_ipmp0 Online e1000g0 Online
------------------------------------------------------------------
And we have a basic, working cluster!
Discovered Problems
Interconnect (”Cluster Transport”) is marked faulted
For example, if you do an scstat, or an scstat -W you see:
Transport path: mail-store1:e1000g2 mail-store0:e1000g2 faulted
Transport path: mail-store1:e1000g1 mail-store0:e1000g1 Path online
(at boot it might be “waiting” for quite some time)
In some cases you can disconnect and reconnect the adapter in VMware. However, in others you may have to be more drastic.
Check you can ping the other node via this path - if you can, then you should be all good to run the following commands:
scconf -c -m endpoint=mail-store0:e1000g2,state=disabled
where mail-store0 is your current node, and e1000g2 is the failed adapter. After you’ve done this, you can re-enable it:
scconf -c -m endpoint=mail-store0:e1000g2,state=enabled
And you should now have an online path shortly afterwards:
bash-3.00# scstat -W
-- Cluster Transport Paths --
Endpoint Endpoint Status
-------- -------- ------
Transport path: mail-store1:e1000g2 mail-store0:e1000g2 Path online
Transport path: mail-store1:e1000g1 mail-store0:e1000g1 Path online
All good!
Cluster Panics with pm_tick delay [number] exceeds [another number]
Try the following:
- Stop VMs being paged to disk in VMWare (only use physical memory for your VMs). This is a VMWare server, host setting from memory
- Ensure Memory Trimming is disabled for your VMware Server Sun Cluster Guests
- On each Cluster node, in order, configure the heartbeats to be father apart, and have a longer timeout:
scconf -c -w heartbeat_timeout=60000 scconf -c -w heartbeat_quantum=10000
Hopefully this will leave you with a much more stable cluster on VMware.
