UNIXSysAdmin.net

One of the things I’ve done on my week off is get Sun Cluster working on VMware Server. There’s a few small tricks to it, but generally it hasn’t been as horrible as the first time I did it many months ago on VMware (a 4 node cluster, with panics galore – no fun).

NOTE: To run Sun Cluster on Solaris 10, you will need to be running your VMs as 64-bit guests (Sun Cluster, on Solaris 10, on x86/x64 DOES NOT run on Solaris 10 32-bit – found this out the hard way a long time ago!).

To do 64-bit VMs, only some CPUs are supported. In this case I am using a relatively new AMD 64 x2, and thats perfect for this purpose. In this build I’m using Solaris u3 is because u4 seems to have a few issues on VMware (lots of kernel panics on boot I’ve found), and Sun Cluster 3.1 8/05 (u4), even though Solaris Cluster 3.2 is out, Sun Cluster 3.1u4 is still what most things are certified against. I will build a 3.2 cluster at some point later on.

So, lets start with configuring VMware:

-Configure at least 2 additional host based networks, on Linux you will need to run

vmware-config.pl

I have configured several more (seeing as it is easy to do it all at once).

When it asks you about networking, you want to configure additional host based networks. The scheme I have used is:

172.16.0.0/255.255.255.0

172.16.1.0/255.255.255.0

.

.

172.16.11.0/255.255.255.0

NOTE: What subnets you specify here isn’t that relevant as your host will never talk on these networks, the host to host cluster interconnects will use the interfaces, and will likely use different subnets. Each subnet just needs to be different for each vmnet adapter. You need not use /24’s either, you could go down to a very small subnet size (/30 for example). If this makes no sense to you, do not worry too much, keep going anyway just following what I’ve done above.

Now, create 2 VMs – Typical, Solaris 10 64-bit (will not work if 64-bit is not selected), and in my case I preallocated 32GB of disk.

For each VM:

-Add at least 2 additional ethernet interfaces (e1000gXs) – these will be used for interconnects. Put each one on a different vmnet adapter. I used vmnet1 and vmnet2 in this case. Make sure when you do this, you do the interfaces identically on both VMs as they will need to talk to each other across these interfaces, but nothing else.

-Disable snapshots (for performance)

-Add a single disk in a different location for quorom, to one host only. I added quorom in /virtuals/cluster-disks/mail-store-quorom.vmdk. Quorom should be as small as possible, I believe the smallest disk you can build is ~100meg, so do that (0.1GB)

Boot each node for a tick, then shut it down straight away. This is to build the initial vmx config additional information that is not populated till you bring up a VM (ethernet address for e1000g0, is what we really want to be populated here).

-With each VM shut down, edit the .vmx file for each VM and add in the lines:

Configure shared quorom device:

scsi0:1.present = "TRUE"
scsi0:1.fileName = "/virtuals/cluster-disks/mail-store-quorom.vmdk"
scsi0.sharedBus = "virtual"
disk.locking = "false"

Obviously there’s no need to re-add the filename line for the host you configured the disk on initially.

Next for each VM configure CPU (core) binding – if you don’t do this and you’re using a 64 bit AMD chip, you’ll get some interesting behaviour because the timestamping on each core of these CPUs is different, it messes with Solaris which expects to be on one CPU. Cluster will panic more often if you don’t do this 🙂

processor0.use = "TRUE"
processor1.use = "FALSE"

And I do the reverse of the above for the other host’s vmx file:

processor0.use = "FALSE"
processor1.use = "TRUE"

-Now kick off a Solaris build using the minimal profile from my last post, or a SUNWCXall (all packages) will do if you don’t mid the extra build time wait and you have disk space up your sleeve. You could of course just do a straight install off the CD, just make sure you use a custom partitioning scheme ensuring there is a 512MB /globaldevices slice of your disk.

-Install Sun Cluster from anything Java Enterprise System 2005Q4 or above (JES 5 at time of writing). Obtain it for free from http://www.sun.com/software/javaenterprisesystem/getit.jsp. All the JES’ from (and including) 2005Q4 have Sun Cluster 3.1u4 (that is 8/05). It’s worth noting Sun Cluster 3.2 is out and has been for a while, but I’m not sure how much stuff is certified against it. I will try it out later, for the moment I’ll go with 3.1u4.

Note: there is nearly no initial config when you install Sun Cluster from the JES installer. I noticed in the most recent release of JES (and possibly some previous releases, I’ve missed a few) it asks if you want to allow Sun Cluster to be configured remotely. For simplicity, answer yes. It makes the cluster config very easy from there.

-Add /usr/cluster/bin to your path for convenience on both hosts

-run scinstall on ONE host (/usr/cluster/bin/scinstall if you did not follow the above step).

You’ll get a menu… with the first item where we want to be.

* 1) Install a cluster or cluster node

So select 1, then..

1) Install all nodes of a new cluster

1 again, then yes to continue

Please select from one of the following options:

1) Typical

2) Custom

Select 1

Then select a cluster name, in this case I’ve gone with mail-store-clus as this is to become a cluster of Sun Messaging Server 6.3 Mail Stores

Next you are asked for other nodes in the cluster, in this case the only other node for me is mail-store1, so I type that in

Node name (Control-D to finish):  mail-store1
 Node name (Control-D to finish):

This is the complete list of nodes:

mail-store0
mail-store1

Is it correct (yes/no) [yes]?

and then ctrl-D, then yes it is correct

Attempting to contact "mail-store1" ... done

Searching for a remote install method ... done

The Sun Cluster framework software is already installed on each of
the new nodes of this cluster. And, it is able to complete the
configuration process without remote shell access.

Looking good so far! Enter to continue.

Select the first cluster transport adapter for "mail-store0":

1) e1000g1

2) e1000g2

3) Other

Go with 1, then the next transport adapter, 2. NOTE: If you have plumb’d these devices, they will not work. These cards need to be unplumb’d in that case.

    Searching for any unexpected network traffic on "e1000g1" ... done

    Verification completed. No traffic was detected over a 10 second

    sample period.

Next up, quorom. This is why we setup the shared disk earlier:

Do you want to disable automatic quorum device selection (yes/no) [no]?

(go with the default, no)

    Is it okay to begin the installation (yes/no) [yes]?

yes, it sure is!

    During the installation process, sccheck(1M) is run on each of the
    new cluster nodes. If sccheck(1M) detects problems, you can either
    interrupt the installation process or check the log files after
    installation has completed.

    Interrupt the installation for sccheck errors (yes/no) [no]?

default is fine, no

and off we go:

  Installation and Configuration

    Log file - /var/cluster/logs/install/scinstall.log.630

    Testing for "/globaldevices" on "mail-store0" ... done
    Testing for "/globaldevices" on "mail-store1" ... done

    Starting discovery of the cluster transport configuration.

    The following connections were discovered:

        mail-store0:e1000g1  switch1  mail-store1:e1000g1
        mail-store0:e1000g2  switch2  mail-store1:e1000g2

    Completed discovery of the cluster transport configuration.

    Started sccheck on "mail-store0".
    Started sccheck on "mail-store1".

    sccheck completed with no errors or warnings for "mail-store0".
    sccheck completed with no errors or warnings for "mail-store1".

    Configuring "mail-store1" ... done
    Rebooting "mail-store1" ...

And the second node reboots, then the first

    Rebooting "mail-store1" ... done

    Configuring "mail-store0" ... done
    Rebooting "mail-store0" ... 

Log file - /var/cluster/logs/install/scinstall.log.630

Rebooting ... 

updating /platform/i86pc/boot_archive...this may take a minute
Connection to mail-store0 closed by remote host.
Connection to mail-store0 closed.

Let the first node boot, and you’ll see a bunch of stuff on the console. Don’t stress, it’s (probably) normal. It is normal to see a few errors at first boot.

Let the cluster sort it’s stuff out (give it a couple of minutes) then run scstat to check the status of the cluster. It should look something like:

-bash-3.00$ scstat
------------------------------------------------------------------

-- Cluster Nodes --

                    Node name           Status
                    ---------           ------
  Cluster node:     mail-store1         Online
  Cluster node:     mail-store0         Online

------------------------------------------------------------------

-- Cluster Transport Paths --

                    Endpoint               Endpoint               Status
                    --------               --------               ------
  Transport path:   mail-store1:e1000g2    mail-store0:e1000g2    Path online
  Transport path:   mail-store1:e1000g1    mail-store0:e1000g1    Path online

------------------------------------------------------------------

-- Quorum Summary --

  Quorum votes possible:      3
  Quorum votes needed:        2
  Quorum votes present:       3

-- Quorum Votes by Node --

                    Node Name           Present Possible Status
                    ---------           ------- -------- ------
  Node votes:       mail-store1         1        1       Online
  Node votes:       mail-store0         1        1       Online

-- Quorum Votes by Device --
                    Device Name         Present Possible Status
                    -----------         ------- -------- ------
  Device votes:     /dev/did/rdsk/d2s2  1        1       Online

------------------------------------------------------------------

-- Device Group Servers --

                         Device Group        Primary             Secondary
                         ------------        -------             ---------

-- Device Group Status --

                              Device Group        Status
                              ------------        ------              

-- Multi-owner Device Groups --

                              Device Group        Online Status
                              ------------        -------------

------------------------------------------------------------------
------------------------------------------------------------------

-- IPMP Groups --

              Node Name           Group   Status         Adapter   Status
              ---------           -----   ------         -------   ------
  IPMP Group: mail-store1         sc_ipmp0 Online         e1000g0   Online

  IPMP Group: mail-store0         sc_ipmp0 Online         e1000g0   Online

------------------------------------------------------------------

And we have a basic, working cluster!

Discovered Problems

Interconnect (“Cluster Transport”) is marked faulted

For example, if you do an scstat, or an scstat -W you see:

  Transport path:   mail-store1:e1000g2    mail-store0:e1000g2    faulted

  Transport path:   mail-store1:e1000g1    mail-store0:e1000g1    Path online

(at boot it might be “waiting” for quite some time)

In some cases you can disconnect and reconnect the adapter in VMware. However, in others you may have to be more drastic.

Check you can ping the other node via this path – if you can, then you should be all good to run the following commands:

 scconf -c -m endpoint=mail-store0:e1000g2,state=disabled

where mail-store0 is your current node, and e1000g2 is the failed adapter. After you’ve done this, you can re-enable it:

scconf -c -m endpoint=mail-store0:e1000g2,state=enabled

And you should now have an online path shortly afterwards:

bash-3.00# scstat -W

-- Cluster Transport Paths --

                    Endpoint               Endpoint               Status

                    --------               --------               ------

  Transport path:   mail-store1:e1000g2    mail-store0:e1000g2    Path online

  Transport path:   mail-store1:e1000g1    mail-store0:e1000g1    Path online

All good!

Cluster Panics with pm_tick delay [number] exceeds [another number]

Try the following:

1. Stop VMs being paged to disk in VMWare (only use physical memory for your VMs). This is a VMWare server, host setting from memory
2. Ensure Memory Trimming is disabled for your VMware Server Sun Cluster Guests
3. On each Cluster node, in order, configure the heartbeats to be father apart, and have a longer timeout:

scconf -c -w heartbeat_timeout=60000 

scconf -c -w heartbeat_quantum=10000

Hopefully this will leave you with a much more stable cluster on VMware.

So I’ve spent this week off from work, and I have been spending my time working on building a jumpstart environment to deploy a stack of hosts into VMWare, so as I can build various kinds of Solaris hosts, including LDAP replicas (Directory Server 6.2), Mail stores and gateways using Sun Cluster and Messaging Server 6.3, and Oracle RAC on Sun Cluster.

I thought I’d give you some code I’ve written (its spectacularly awful code, but it works and works quickly) to build a minimal Solaris profile. The whole idea is to use SUNWCreq as your base for your profile, and only add packages you want from there. It makes my build around 730meg with all the stuff I need to run Sun Cluster and a bunch of utilities I find handy, without all the additional stuff that I won’t be using at this point in time.

To use this utility you will need to know the name of the package, and be running this on a Solaris host that actually has the package installed (in this case I’m running it on my jumpstart server, which is built using SUNWCXall – that is, everything). On a host like this you can locate the name of the package you are after by finding a known file from the package in /var/sadm/install/contents. eg.

bash-3.00# grep /usr/bin/bash /var/sadm/install/contents
 /usr/bin/bash f none 0555 root bin 611132 31995 1178118881 SUNWbash

Then you can use:

java SPDB SUNWbash

to get a profile-friendly output of all SUNWbash’s dependecies, excluding those already in SUNWCreq.

To build a reasonably minimised host ready for Sun Cluster 3.1u4, using Solaris 10 u3, with utilities I like to have installed (including the SUNWinst package so I can take a flar at the end and gcc if I’d like to do some compiling):

java SPDB SUNWgcc SUNWgmake SUNWhea SUNWluzone SUNWman SUNWpl5u SUNWpool SUNWsshcu \

SUNWsshdr SUNWsshu SUNWsshdu SUNWvolu SUNWvolr SUNWwgetr SUNWwgetu SUNWzoneu \

SUNWzoner SUNWzfsu SUNWzfsr SUNWbind SUNWdhcsu SUNWdhcsr SUNWinst SUNWbash \

SUNWtcatu SUNWcacaort > /var/tmp/pkgList

Then use that output in a profile you might have something like:

install_type   initial_install
 system_type standalone
 # start with the minimal required number of packages
 cluster        SUNWCreq

#additional packages

<INSERT package list here, in vi use:

:r /var/tmp/pkgList

to read it in>

#patches

patch 118855-36 nfs 10.0.0.1:/jumpstart/patches

# format the entire disk for Solaris
 fdisk   all   solaris all

# define how the disk is partitioned
 boot_device any update
 partitioning      explicit
 filesys rootdisk.s0 4096 /
 filesys rootdisk.s1 2048 swap
 filesys rootdisk.s3 512  /globaldevices
 filesys rootdisk.s4 free /var

for example.

You can get my first cut of the code here: http://unixsysadmin.net/files/SPDB.class

Its so dodgy that I haven’t even packaged it up. I will make it much more useable in future (differing repositories of information, find dependencies based on the binary/file you’re after, so you need not know package names and so on).

Some might say there is a tool that does what I required and perhaps a much better job (http://blogs.sun.com/gbrunett/entry/solaris_package_companion for example). I haven’t used this tool, and to be honest I wanted to learn some more about how the package dependency structure worked in Solaris, so this was for both my own learning, and immediate need of having a minimal install of solaris, with just what I need – to basically reduce jumpstart host deployment time.

NOTES:

-SPDB stands for Solaris Profile Dependency Builder. I think I’ll change that when I come up with a good name. Enjoy!

-Java 1.5 is required for this package to work correctly

I have released a bug fixed release of findUsedSpace. This fixes a bug on Solaris 10+ Systems, that contain zones. There was an endless loop when you reached the proc filesystem of a zone, when running from the global.

This RC is likely to have one more minor code clean up before it is released as findUsedSpace 1.0.

There is a feature request for the next release to skip autofs,ctfs,objfs,devfs,dev and any other filesystems on a UNIX (Solaris specifically) platform which isn’t really relevant when you’re trying to figure out what has chewed all the space on a disk.

For those who are unaware what the tool does, it basically allows you to do something very similar to du, by finding how much space a parent directory and all its children are taking up (or a single file), however, it is based on thresholds, and allows you to be very clear on what you are looking for (say, files and directories over 1 gigabyte). It runs at virtually the same speed as native du (tiny little bit slower in my tests, but the difference is negligible).

The other benefit over du is it is pure java, and platform independent. It is tested on Windows and Solaris, but should run on anything. Though written on a 1.6 JVM, it is compiled for a 1.2 JVM, so as even really old systems can run the software.

You can download a Solaris packaged release, or a zip file release from here: http://unixsysadmin.net/files

You’ll notice its packaged as AdeptTools (even though it’s just one tool at this point). The idea was/is that I will gradually write more and more code as new problems present themselves to me, and package it as a toolkit. Not sure if it will work out that way in the end, we’ll see. Due to somethings ending up platform dependent, it may end up that each tool is released separately, and as part of a package if you want the lot.