You can install SMC using the SGI Admin Install DVD with SGI Management Center software bundle. In this method, SMC is automatically installed during infrastructure setup on the admin node (see “configure-cluster Command Cluster Configuration Tool”).

---

Note: SMC requires an LK license. Save the license to /etc/lk/keys.dat .

---

This section describes how to install the SGI Management Center (SMC) software on a system already running SMC for Altix ICE 1.1 software.

---

Note: SMC requires an LK license. Save the license to /etc/lk/keys.dat .

---

To install SGI Management Center (SMC) on a system running SMC for Altix ICE 1.1, perform the following steps:

When using the configure-cluster command and adding the SLES media, only the first DVD is needed. The second SLES DVD, that contains the source RPMs, is not required for cluster operation and cannot be added to the repositories.

To install SLES 11 software images on the system admin controller, perform the following steps:

Use the configure-cluster command and RHEL media to install software on the admin node

To install RHEL 6 software images on the system admin controller, perform the following steps:

This section describes how to configure Red Hat Enterprise Linux 6 on the system admin controller (admin node).

To perform the initial configuration of a RHEL6 Admin node, perform the following steps:

---

Note: SGI suggests that you run the configure-cluster command either from the VGA screen or from an ssh session to the admin node. Avoid running the configure-cluster command from a serial console.

---

Figure 2-9. Cluster Configuration Tool: Main Menu Screen

It allows you to perform the following:

Information on using this tool is described in the procedure in the following section, see “Installing SMC for Altix ICE Admin Node Software ”.

This section describes how to use configure-cluster command to configure the system administrator controller (admin node) for your Altix ICE system.

To use the configure-cluster command to configure system admin controller (admin node), perform the following steps:

After finishing with the configure-cluster command, you probably will want to modify the ntp configuration to point at a time server on your network. Please do not remove entries that serve the cluster networks.

By default, the configuration in /etc/ntp.conf directs requests to public servers of the pool.ntp.org project.

server 0.rhel.pool.ntp.org
server 1.rhel.pool.ntp.org
server 2.rhel.pool.ntp.org

To direct requests to the your company ntp server, for example ntp.mycompany.com, comment or delete the rhel.pool.ntp.org entries and insert the local entry, as follows:

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.rhel.pool.ntp.org
#server 1.rhel.pool.ntp.org
#server 2.rhel.pool.ntp.org
server ntp.mycompany.com

Restart the ntp server, as follows:

# /etc/init.d/ntpd restart

You may want to reach network services outside of your SGI Altix ICE system. For this type of access, SGI recommends using Network Address Translation (NAT), also known as IP Masquerading or Network Masquerading. Depending on the amount of network traffic and your site needs, you may want to have multiple service nodes providing NAT services.

To enable NAT on your service node, perform the following steps:

system-1:/ # netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
128.162.244.0   0.0.0.0         255.255.255.0   U         0 0          0 eth0
172.16.0.0      0.0.0.0         255.255.0.0     U         0 0          0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 eth1
127.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 lo
0.0.0.0         128.162.244.1   0.0.0.0         UG        0 0          0 eth0

tcpdump -i eth1 ip proto ICMP # Dump ping packets on the public side of service node.
tcpdump -i ib1 ip proto ICMP # Dump ping packets on the IB fabric side of service node.
tcpdump -i eth1 port nfs # Dump NFS traffic on the eth1 side of service node.
tcpdump -i ib1 port nfs # Dump NFS traffic on the eth1 side of service node.

You may want to configure service node(s) to act as NAT gateways for your cluster (see “Service Node Configuration for NAT”) and to have the host names for the compute nodes in the cluster resolve through external DNS servers.

You need to reserve a large block of IP addresses on your house network. If you configure to resolve via external DNS, you need to do it for both the ib0 and ib1 networks, for all node types. In other words, ALL -ib* addresses need to be provided by external DNS. This includes compute nodes, leader nodes, and service nodes. Careful planning is required to use this feature. Allocation of IP addresses will often require assistance from a network administrator of your site.

Once the IP addresses have been allocated on the house network, you need to tell the SMC for Altix ICE software the IP addresses of the DNS servers on the house network that the SMC for Altix ICE software can query for hostname resolution.

To do this, use the configure-cluster tool (see “configure-cluster Command Cluster Configuration Tool”). The menu item that handles this operation is Configure External DNS Masters (optional).

Some important considerations are, as follows:

For information on setting up DNS, see Figure 2-29.

Assuming the installation has either NAT or Gateway operations configured on one or more service nodes, the compute nodes can directly mount the house NFS server's exports (see the exports(5) man page).

To allow the compute nodes to directly mount the house NFS server's exports, perform the following steps:

This section describes two different ways to configure NIS for service nodes and compute blades when you want to use the house network NIS server, as follows:

The first approach would be used in the case where a service node is configured with network address translation (NAT) or gateway operations so that the compute nodes can access the house network directly.

The second approach may be used if the compute nodes do not have direct access to the house network.

To setup NIS with the compute nodes directly accessing the house NIS infrastructure, perform the following steps:

To setup NIS with a service node as a NIS slave server to the house NIS master, perform the following:

The following example uses /dev/disk/by-id/scsi-3600508e0000000008dced2cfc3c1930a ID as the empty disk on which you will put your data. It is very important that you know this for sure. In “Setting Up an NFS Home Server on a Service Node for Your Altix ICE System”, an example is provided that allows you to determine where your root disk is located so you can avoid accidently destroying it. Remember, in some cases, /dev/sdb will be the root drive and /dev/sda or /dev/sdc may be the data drive. Please confirm that you have selected the right device, and use the persistent device name to help prevent accidental overwriting of the root disk.

---

Note: Steps 1 through 7 of this procedure are performed on the service node. Steps 8 and 9 are performed from the system admin controller (admin node).

---

For information on centrally managed user accounts, see “Setting Up a NIS Server for Your Altix ICE System”. It describes NIS master set up. In this design, the master server residing on the service node provides the filesystem and the NIS slaves reside on the rack leader controllers. If you have more than one home server, you need to export all home filesystems on all home servers to the server acting as the NIS master. You also need to export the filesystems to the NIS master using the no_root_squash exports flag.

If you want to use NAS server for scratch storage or make home filesystems available on NAS, you can follow the instructions in “Setting Up an NFS Home Server on a Service Node for Your Altix ICE System”. In this example, you need to replace service0-ib1 with the ib1 InfiniBand host name for the NAS server and you need to know where on the NAS server the home filesystem is mounted to craft the sgi-fstab script properly.

In the procedures that follow in this section, here are some of the tasks you need to perform and system features you need to consider:

This section describes how to set up a service node as a NIS master. This section only applies to service nodes running SLES.

To set up a SLES service node as a NIS master, from the service node, perform the following steps:

---

Note: These instructions use the text-based version of YaST. The graphical version of YaST may be slightly different.

---

This section describes how to use YaST to set up your other service nodes to be broadcast binding NIS clients. This section only applies to service nodes running SLES11.

---

Note: You do not do this on the NIS Master service node that you already configured as a client in “Setting Up a SLES Service Node as a NIS Master”.

---

To set up a service node as a NIS client, perform the following steps:

This section provides two sets of instructions for setting up rack leader controllers (leader nodes) as NIS slave servers. It is possible to make all these adjustments to the leader image in /var/lib/systemimager/images . Currently, SGI does not recommend using this approach.

---

Note: Be sure the InfiniBand interfaces are up and running before proceeding because the rack leader controller gets its updates from the NIS Master over the InfiniBand network. If you get a "can't enumerate maps from service0" error, check to be sure the InfiniBand network is operational.

---

Use the following set of commands from the system admin controller (admin node) to set up a rack leader controller (leader node) as a NIS slave server and client.

---

Note: Replace ice with your NIS domain name and service0 with the service node you set up as the master server.

---

admin:~ # cexec --head --all chkconfig ypserv on
admin:~ # cexec --head --all chkconfig ypbind on
admin:~ # cexec --head --all chkconfig portmap on
admin:~ # cexec --head --all chkconfig nscd on
admin:~ # cexec --head --all rcportmap start
admin:~ # cexec --head --all "echo ice > /etc/defaultdomain"
admin:~ # cexec --head --all "ypdomainname ice"
admin:~ # cexec --head --all "echo ypserver service0 > /etc/yp.conf"
admin:~ # cexec --head --all /usr/lib/yp/ypinit -s service0
admin:~ # cexec --head --all rcportmap start
admin:~ # cexec --head --all rcypserv start
admin:~ # cexec --head --all rcypbind start
admin:~ # cexec --head --all rcnscd start

This section describes how to set up the compute nodes to be NIS clients. You an configure NIS on the clients to use a server list that only contains the their rack leader controller (leader node). All operations are performed from the system admin controller (admin node).

To set up the compute nodes to be NIS clients, perform the following steps:

The NAS cube needs to get configured with each InfiniBand fabric interface in a separate subnet. These fabrics will be separated from each other logically, but attached to the same physical network. For simplicity, this guide assumes that the -ib1 fabric for the compute nodes has addresses assigned in the 10.149.0.0/16 network. This guide also assumes the lowest address the cluster management software has used is 10.149.0.1 and the highest is 10.149.1.3 (already assigned to the NAS cube).

For the NAS cube, you need to configure the large physical network into four, smaller subnets, each of which would be capable of containing all the nodes and service nodes. It will have subnets 10.149.0.0/18 , 10.149.64.0/18, 10.149.128.0/18 , and 10.149.192.0/18.

After the discovery of the storage node has happened, SGI personnel will need to log onto the NAS box and change the network settings to use the smaller subnets, and then define the other three adapters with the same offset within the subnet; for example: Initial configuration of the storage node had set ib0 fabric's IP to 10.149.1.3 netmask 255.255.0.0. After the addresses are changed, ib0=10.149.1.3:255.255.192.0, ib1=10.149.65.3:255.255.192.0 , ib2=10.149.129.3:255.255.192.0, ib3=10.149.193.3:255.255.192.0 . The NAS cube should now have all four adapter connections connected to the fabric with IP addresses which can be pinged from the service node.

---

Note: The service nodes and the rack leads will remain in the 10.149.0.0/16 subnet.

---

For the compute blades, log into the admin node and modify /opt/sgi/share/per-host-customization/global/sgi-setup-ib-configs file. Following the line iruslot=$1, insert:

# Compute NAS interface to use
IRU_NODE=`basename ${iruslot}`
RACK=`cminfo --rack`
RACK=$(( ${RACK} - 1 ))
IRU=`echo ${IRU_NODE} | sed -e s/i// -e s/n.*//`
NODE=`echo ${IRU_NODE} | sed -e s/.*n//`
POSITION=$(( ${IRU} * 16 + ${NODE} ))
POSITION=$(( ${RACK} * 64 + ${POSITION} ))
NAS_IF=$(( ${POSITION} % 4 ))
NAS_IPS[0]="10.149.1.3"
NAS_IPS[1]="10.149.65.3"
NAS_IPS[2]="10.149.129.3"
NAS_IPS[3]="10.149.193.3"

Then following the line $iruslot/etc/opt/sgi/cminfo add:

IB_1_OCT12=`echo ${IB_1_IP} | awk -F "." '{ print $1 "." $2 }'`
IB_1_OCT3=`echo ${IB_1_IP} | awk -F "." '{ print $3 }'`
IB_1_OCT4=`echo ${IB_1_IP} | awk -F "." '{ print $4 }'`
IB_1_OCT3=$(( ${IB_1_OCT3} + ${NAS_IF} * 64 ))
IB_1_NAS_IP="${IB_1_OCT12}.${IB_1_OCT3}.${IB_1_OCT4}"

Then change the IPADDR='${IB_1_IP}' and NETMASK='${IB_1_NETMASK}' lines to the following:

IPADDR='${IB_1_NAS_IP}'
NETMASK='255.255.192.0'

Then add the following to the end of the file:

# ib-1-vlan config
cat << EOF >$iruslot/etc/sysconfig/network/ifcfg-vlan1
# ifcfg config file for vlan ib1
BOOTPROTO='static'
BROADCAST=''
ETHTOOL_OPTIONS=''
IPADDR='${IB_1_IP}'
MTU=''
NETMASK='255.255.192.0'
NETWORK=''
REMOTE_IPADDR=''
STARTMODE='auto'
USERCONTROL='no'
ETHERDEVICE='ib1'
EOF
if [ $NAS_IF -eq 0 ]; then
    rm $iruslot/etc/sysconfig/network/ifcfg-vlan1
fi

To update the fstab for the compute blades, edit /opt/sgi/share/per-host-customization/global/sgi-fstab file. Perform the equivalent steps as above to add the # Compute NAS interface to use section into this file. Then to specify mount points, add lines similar to the following example:

# SGI NAS Server Mounts
${NAS_IPS[${NAS_IF}]}:/mnt/data/scratch     /scratch nfs    defaults 0 0

The example used in this section assumes that the home directory is mounted on the NIS Master service and that the NIS master is able to create directories and files on it as root. The following example use command line commands. You could also create accounts using YaST.

To create user accounts on the NIS server, perform the following steps:

If you add or remove a rack leader controller (leader node), for example, if you use discover command to discover a new rack of equipment, you will need to configure the new rack leader controller to be an NIS slave server as described in “Setting Up a SLES Service Node as a NIS Client”.

In addition, you need to add or remove the leader from the /var/yp/ypservers file on NIS Master service node. Remember to use the -ib1 name for the leader, as service nodes cannot resolve r2lead style names. For example, use r2lead-ib1.

# cd /var/yp && make

SGI supplies updates to SMC for Altix ICE software via the SGI update server at https://update.sgi.com/ . Access to this server requires a Supportfolio login and password. Access to SUSE Linux Enterprise Server updates requires a Novell login account and registration.

The initial installation process for the SGI Altix ICE system set up a number of package repositories in the /tftpboot directory on the admin node. The SMC for Altix ICE related packages are in directories located under the /tftpboot/sgi directory. For SUSE Linux Enterprise Linux 11 (SLES11), they are in /tftpboot/distro/sles11 .

When SGI releases updates, you may run sync-repo-updates (described later) to download the updated packages that are part of a patch. The sync-repo-updates command automatically positions the files properly under /tftpboot.

Once the local repositories contain the updated packages, it is possible to update the various SGI Altix ICE admin, leader, and managed service node images using the cinstallman command. The cinstallman command is used for all package updates including those within images, running nodes, including the admin node itself.

There is a small amount of preparation required, in order to setup an SGI Altix ICE system, so that updated packages can be downloaded from the SGI update server and the Linux distro server and then installed with the cinstallman command.

This following sections describe these steps, as follows:

This section explains how to update the local product package repositories needed to share updates on all of the various nodes on an SGI Altix ICE system.

Update the SGI Package Repositories on the Admin Node

SGI provides a sync-repo-updates script to help keep your local package repositories on the admin node synchronized with available updates for the SMC for Altix ICE, SGI Foundation, SGI Performance Suite, and SLES products. The script is located in /opt/sgi/sbin/sync-repo-updates on the admin node.

The sync-repo-updates script requires your Supportfolio user name and password. You can supply this on the command line or it will prompt you for it. With this login information, the script contacts the SGI update server and downloads the updated packages into the appropriate local package repositories.

For SLES, if you installed and configured the SMT tool as described in “SLES Admin Nodes: Update the SLES Package Repository”, the sync-repo-updates script will also download any updates to SLES from the Novell update server. When all package downloads are complete, the script updates the repository metadata.

Once the script completes, the local package repositories on the admin node should contain the latest available package updates and be ready to use with the cinstallman command.

The sync-repo-updates script operates on all repositories, not just the selected reposistory.

---

Note: You can use the crepo command to set up custom repositories. If you add packages to these custom repositories later, you need to use the yume --prepare --repo command on the custom repository so that the metadata is up to date. Run the cinstallman --yum-node --node admin clean all command and then the yum/yume/cinstallman command.

---

In 1.8 (or later), SLES updates are mirrored to the admin node using the SUSE Linux Enterprise Subscription Management Tool. The Subscription Management Tool (SMT) is used to mirror and distribute updates from Novell. SMC for Altix ICE software only uses the mirror abilities of this tool. Mechanisms within SMC for Altix ICE are used to deploy updates to installed nodes and images. SMT is described in detail in the SUSELinux Enterprise Subscription Management Tool Guide. A copy of this manual is in the SMT_en.pdf file located in the /usr/share/doc/manual/sle-smt_en directory on the admin node of your system. Use the scp(1) command to copy the manual to a location where you can view it, as follows:

admin :~ # scp /usr/share/doc/manual/sle-smt_en/SMT_en.pdf user@domain_name.mycompany.com:

Configuring the SMT Using YaST

At this point, your admin node should be registered with Novell. You should also have a Novell login available that is associated with the admin node. This Novell login will be used when configuring the SMT described in this section. If the Novell login does not have proper authorization, you will not be able to register the appropriate update channels. Contact Novell with any questions on how to obtain or properly authorize your Novell login for use with the SMT.

Procedure 2-21. Configuring SMT Using YaST

---

Note: In step 8, a window pops up asking you for the Database root password. View the file /etc/odapw. Enter the contents of that file as the password in the blank box.

---

To configure SMT using YaST, perform the following steps:

1. Start up the YaST tool, as follows:

   admin:~ # yast

   
   

2. Under Network Services, find SMT Configuration

3. For Enable Subscription Management Tool Service (SMT), check the box.

4. For NU User, enter your Novell user name.

5. For NU Password, enter your Novell password.

   ---

   Note: It is the mirror credentials you want. You can have a login that gets updates but cannot mirror the repository.

   ---

   
   

6. For NU E-Mail, use the email with which you registered.

7. For your SMT Server URL, just leave the default.

   It is a good idea to use the test feature. This will at least confirm basic functionality with your login. However, it does not guarantee that your login has access to all the desired update channels.

   Note that Help is available within this tool regarding the various fields.

8. When you click Next, a window pops up asking for the Database root password. View the file /etc/odapw. Enter the contents of that file as the password in the blank box.

   A window will likely pop up telling you that you do not have a certificate. You will then be given a chance to create the default certificate. Note that when that tool comes up, you will need to set the password for the certificate by clicking on the certificate settings.

Downloading the Updates from Novell and SGI

At this time, you should have your update channels registered. From here on, the sync-repo-updates script will do the rest of the work. That script will use SMT to download all the updates and position those updates in to the existing repositories so that the various nodes and images can be upgraded.

Run /opt/sgi/sbin/sync-repo-updates script.

After this completes, you need to update your nodes and images (see “Installing Updates on Running Admin, Leader, and Service Nodes ”).

---

Note: Be advised that the first sync with the Novell server will take a very long time.

---

This section describes how to keep your packages up to date on RHEL based admin nodes. The general idea is that we download all updates in to the RHEL repository, and then use SGI Management Center for ICE tools to deploy the updates to nodes and images.

Perform the following:

In situations where you have distributions (distros) present that do not match the distro on installed on the admin node, you will have to arrange to download the updates on your own.

# yume --prepare --repo /tftpboot/distro/sles11sp1

RHEL

You can register with RHN on a RHEL server on your network. Then, you can look at the /opt/sgi/sbin/sync-repo-updates script to see how it stages the packages (search for RHN in that file). Following that example, you can set up a server on your house network to stage the files. Then you can then copy the staged packages to the admin node in to the matching distro repository. You need to update the repository metadata after copying packages using yume in a way similar to this:

# yume --prepare --repo /tftpboot/distro/rhel6.0

---

Note: You can always make a managed service node provide the function of staging the updates.

---

This section explains how to update existing nodes and images to the latest packages in the repositories.

To install updates on the admin node, perform the following command from the admin node:

admin:~ # cinstallman --update-node --node admin

To install updates on all online leader nodes, perform the following command from the admin node:

admin:~ # cinstallman --update-node --node r\*lead

To install updates on all managed and online service nodes, perform the following from the admin node:

admin:~ # cinstallman --update-node --node service\*

To install updates on the admin, all online leader nodes, and all online and managed service nodes with one command, perform the following command from the admin node:

admin:~ # cinstallman --update-node --node \*

Please note the following:

You can also use the cinstallman command to update systemimager images with the latest software packages.

---

Note: Changes to the kernel package inside the compute image require some additional steps before the new kernel can be used on compute nodes (see “Additional Steps for Compute Image Kernel Updates” for more details). This note does not apply to leader or managed service nodes.

---

The following examples show how to upgrade the packages inside the three node images supplied by SGI:

admin:~ # cinstallman --update-image --image lead-sles11 
admin:~ # cinstallman --update-image --image service-sles11 
admin:~ # cinstallman --update-image --image compute-sles11

---

Note: Changes to the compute image on the admin node are not seen by the compute nodes until the updates have been pushed to the leader nodes with the cimage command. Updating leader and managed service node images ensure that the next time you add or re-discover or re-image a leader or service node, it will already contain the updated packages.

---

Before pushing the compute image to the leaders using the cimage command, it is good idea to clean the yum cache.

---

Note: The yum cache can grow and is in the writable portion of the compute blade image. This means it is replicated 64 times per compute blade image per rack and the space that may be used by compute blades is limited by design to minimize network and load issues on rack leader nodes.

---

To clean the yum cache, from the system admin controller (admin node), perform the following:

admin:~ # cinstallman --yum-image --image compute-sles11 clean all

Any time a compute image is updated with a new kernel, you will need to run some additional steps in order to make the new kernel available. The following example assumes that the compute node image name is compute-sles11 and that you have already updated the compute node image in the image directory per the instructions in “Creating Compute and Service Node Images Using the cinstallman Command” in Chapter 3. If you have named your compute node image something other than compute-sles11, replace this in the example that follows:

Only the leader node has XFS root filesystems. Partition layout for more than one slot is shown in Table 2-1.

Table 2-1 shows a partition table with two available slots. SMC for Altix ICE supports up to five available slots. After five slots, partitions are not available to support the slot.

Partition layout for a single root is shown in Table 2-2. Partition layout for single slot is the same layout that leader and service nodes have used previously. Legacy leader/service node layout is used for single slot, in order to generate the correct pxelinux chainload setup. Previously, the MBR bootloader was used. For multiroot, a chainload to a root slot boot partition is used.

Prior to 1.6 release, admin nodes had a different partition layout than either shown in Table 2-1 or Table 2-2. It had two partitions: swap and a single root. No separate /boot . Any newly installed admin node will have one of the two partition layouts described in the tables above. However, since admin nodes can be upgraded as opposed to re-installed, you may have one of three different partition layouts for admin nodes.

When you boot the admin node installation DVD, you are brought to a syslinux boot banner by default with a boot prompt, as in previous releases.

The multiroot feature support adds a few new parameters, as follows:

Leader and service nodes are installed, as previously. However, they mimic the admin node in terms of partition layout and which slot is used for what purpose.

Therefore, when a discover operation is performed, the slot used for installation is the same slot on which the admin node is currently booted. So you cannot choose what goes where, currently, it all matches the admin node.

If the leader or service node is found to have a slot count that does not match the admin node, the node is re-partitioned. It is assumed if the admin node changes its layout, all partitions on leaders and service nodes are re-initialized as well.

After the admin node is installed with 1.8 (or later), it will boot one of two ways. If only one root slot is configured, the MBR of the admin node will be used to boot the root as usual.

However, if more than one root slot is selected, then the grub loader in the MBR will direct you to a special grub menu that allows you to choose a root slot.

For the multi-root admin node, the sgidata partition is used to store some grub files and grub configuration information. Included is a chainload for each slot. Therefore, the first grub to come up on the admin node chooses between a list of slots. When a slot is selected, a chainload is performed and the grub representing that slot comes up.

This section describes how to handle resets, power cycles, and BMC dhcp leases when changing slots, as follows:

---

Note: In some cases, the IP address setup in one slot may be different than another. This problem can potentially affect leader and service node BMCs. After the admin node is booted up in to a new slot, it is possible the BMCs on the leaders and service nodes may have hung on to their old IP addresses. They will eventually time-out and grab new leases. This problem may manifest itself in cpower not being able to communication with the BMCs properly. If you have trouble connecting to leader and service node BMCs after switching slots on the admin node, give the nodes up to 15 minutes to grab a new leases that match the new slot.

---

The way leader and service nodes boot is dependent on whether the cascading dual-boot feature is in use or not, as explained in this section.

Leader and Service Node Booting on a System Configured with Multiple Roots Slots

When a system is configured with two or more root slots, it is using the cascading dual-boot feature.

In this case, the admin node creates leader and service PXE configuration files that direct the managed service and leader nodes to do one of the following:

• Boot from the currently configured slot

• Reinstall the currently configured slot

Which slot is current, is determined by the slot on which the admin node is booted. Therefore, the admin node and all managed service and leader nodes are always booted on the same slot number.

In order to configure a managed service or leader node to boot from a given slot, the admin node creates a PXE configuration file that is configured to load a chainloader. This chainloader is used to boot the appropriate boot partition of the managed service or leader node.

This means that, in a cascading dual-boot situation, the service and leader nodes do not have anything in their master boot record (MBR). However, each /boot has grub configured to match the associated root slot. A syslinux chainload is performed by PXE to start grub on the appropriate boot partition.

If, for some reason, a PXE boot fails to work properly, there will be no output at all from that node. This means that cascading dual-boot is heavily dependent on PXE boots working properly for its operation.

---

Note: Unlike the managed service and leader nodes, the admin node always has an MBR entry. See “Choosing a Slot to Boot the Admin Node”.

---

A script named /opt/sgi/sbin/clone-slot is available. This script allows you to clone a source slot to a destination slot. It then handles synchronizing the data and fixing up grub and fstabs to make the cloned slot a viable booting choice.

The script sanitizes the input values, then calls a worker script in parallel on all managed nodes and the admin node that does the actual work. The clone-slot script waits for all children to complete before exiting.

Important: If the slot you are using as a source is the mounted/active slot, the script will shut down mysql on the admin node prior to starting the backup operation and start it when the backup is complete. This ensures there is no data loss.

Use the cadmin command to control which slot on the admin node boots by default.

To show the slot that is currently the default, perform the following:

admin:~ # cadmin --show-default-root

To change it so slot 2 boots by default, peform the following:

admin:~ # cadmin --set-default-root --slot 2

You can use the cadmin command to control the grub labels the various slots have. When a slot is installed, the label is updated to be in this form:

slot 1: SMC for Altix ICE 2.0 / sles11: (none)

You can adjust the last part (none in the above example). The following are some example commands.

Show the currently configured grub root labels, as follows:

admin:~ # cadmin --show-root-labels

Set the customer-adjustable portion of the root label for slot 1 to say "life is good", as follows:

admin:~ # cadmin --show-root-labels
slot 1: SMC for Altix ICE 2.0 / sles11: first sles
slot 2: SMC for Altix ICE 2.0 / sles11: my party
slot 3: SMC for Altix ICE 2.0 / sles11: I can cry if I want to.
slot 4
admin:~ # cadmin --set-root-label --slot 1 --label "life is good"
admin:~ # cadmin --show-root-labels
slot 1: SMC for Altix ICE 2.0 / sles11: life is good
slot 2: SMC for Altix ICE 2.0 / sles11: my party
slot 3: SMC for Altix ICE 2.0 / sles11: I can cry if I want to.
slot 4

You can use the cadmin command to show the root slot you are currently booted in to on the admin node, as follows:

admin:~ # cadmin --show-current-root
admin node currently booted on slot: 2

The system admin controller (admin node) high availability (HA) solution makes use of qemu-KVM virtual machines managed by libvirt virtualization API and the high availability (HAE) components supplied by the SLES HAE to provide a high availability environment.

The two physical machines form an HA framework to host a single virtual machine. This virtual machine acts as the system admin controller (admin node) for the Altix ICE system.

If one physical machine fails, the other physical machine starts up the virtual machine and the admin node is brought back up.

It is important to read the release notes for the latest information, known problems, and workarounds needed to configure the system properly. In a scenario, such as HA, where one node can reset the other, it is all the more important to read the release notes prior to attempting the installation.

The major hardware components that comprise this solution include the following:

For a functional diagram of system admin controller (admin node) high availability (HA) solution, see Figure 2-41.

Figure 2-41. Admin Node High Availability Solution Functional Diagram

The network Ethernet connections are shown in Figure 2-42.

Figure 2-42. HAE Network Ethernet Connections

---

Note: In some configurations, Ethernet switches may be necessary between the Altix ICE CMCs and the admin nodes due to port availability.

---

The major software components that comprise this solution include the following:

An overview of how to install the system admin controller (admin node) high availability solution is, as follows:

For a detailed description of this procedure, see “HAE, Virtualization and Updates installation”.

This configuration makes use of STONITH technique, that is, “Shoot The Other Node In The Head”. This means that the physical nodes need to access each other's BMCs. You need to configure the BMCs to be responsive on your house network including having a host name and knowing the login/password for the admin account in the BMC. You should do this early in the process, it will be required in a later step.

You need to configure your external storage. The external storage must have a SAS connection to both physical admin nodes (see Figure 2-41). You must make available at least one LUN and that this LUN needs to appear as a disk to the physical admin nodes.

This section provides some general instructions about SLES 11 SP1 when installing the admin nodes, as follows:

Configure the interface connected to your house network as you normally would. Do not use DHCP but rather a static configuration.

A later step will reconfigure networking and renumber the Ethernet devices to match the necessary layout. It will be told to use the interface you configure now as the reference when it reconfigures networking on the physical admin nodes.

For information on the SUSE Linux Enterprise High Availability Extension (HAE) software package, see the SUSE High Availability Guide . This is required reading and the information contained in this guide is not replicated in this section.

On the physical admin nodes, the document can be found at /usr/share/doc/manual/sleha-guide_en-pdf.

It is available in the sleha-guide_en-pdf package from Novell, Inc.

This section describes the High Availability Extension (HAE), virtualization, and updates installation, as follows:

At this point, both physical admin nodes have the exact same software stack installed including SLES11 SP1, HAE, and all updates applied.

Reboot both admin nodes so that your are running with new kernels, updates, and so on.

Before proceeding to the next section, confirm that both physical admin nodes are now up and running and both with their house network connections configured.

From the SGI Management Center media, copy the RPM named sac-ha from the media on to both physical admin nodes.

Install the package on both physical admin nodes. This provides the scripts to configure both systems.

On the physical admin node that you want to use as primary, edit the file /etc/opt/sgi/sac-hae-initial-setup.conf.

On the second physical admin node, no configuration is necessary. Configuration is all done from the first admin node.

Make sure you read all of the comments in this file. The comments provide the instructions for filling out the file, which include:

Once the configuration file is filled out entirely, you can run /opt/sgi/sbin/sac-hae-initial-setup command.

This script will reconfigure networking on both physical nodes and then will do the HAE setup, and so on. You will be prompted a few times in the early part of the process. These prompts are related to getting the ssh keys and known hosts configuration set up and will involve entering the root password and/or the word yes as prompted. After ssh is set up, the rest of the scripts should run automated without input.

There are places where this process can fail. It is safe to rerun the scripts if they fail in the network set up phase. However, once HAE components are configured, reconfiguring networking can result in machines resetting themselves or each other for protection.

If you do restart the scripts to configure networking, pay special attention to keeping the phys1_current_house and phys2_current_house variables up to date in the /etc/opt/sgi/sac-hae-initial-setup.conf file. They should always point to the currently configured house interface as it preserves the network settings and applies them to the proper interfaces.

For information about shared disk device selection, see the comments in /etc/opt/sgi/sac-hae-initial-setup.conf regarding choosing the shared storage device.

The /etc/opt/sgi/sac-hae-initial-setup.conf file describes how to choose MAC addresses. It is important that you work out the MAC address ordering as described as the network will be reconfigured on both physical admin nodes to match. See the configuration file's notes about using ethtool to help you figure out which physical port matches a MAC address.

By default, the SGI SAC HA scripts set the password for the hacluster account to sgisgi. You will need this password if you run the crm_gui tool and wish to log in as the default hacluster user in that tool.

After the sac-hae-initial-setup script completes, the virtual machine for the admin node should be running on one of the two physical machines. You can identify which one by using the crm_gui command and looking at the virt-clone resource (or similarly use the crm command). If it is not started for some reason, go ahead and start the service.

Next, run the virt-manager command on the physical machine that is currently hosting the guest.

It should be at the Admin Node Installation DVD prompt. From this point forward, installation of SGI Management Center ICE is the same as it is on real hardware.

After the admin node virtual machine has been installed and the execution of the configure-cluster command is complete, you need to allocate the IP addresses used by the physical admin nodes to avoid an IP address conflict. This should be done prior to discovering any nodes.

In the example below, phys1 is the HEAD network IP for physical machine 1 and phys2 is for physical machine 2. The IP addresses provided match the IP address used with the SAC HA configuration scripts you ran earlier:

# discover --service 98,other
# cadmin --set-ip --node service98 --net head service98=172.23.254.253
# cadmin --set-hostname --node service98 phys1

# discover --service 99,other
# cadmin --set-ip --node service99 --net head service99=172.23.254.254
# cadmin --set-hostname --node service99 phys2

The sac-ha-initial-setup script sets up a basic default set of high availability (HA) rules for the cluster. They are meant to be a starting point for your site.

SGI will likely come up with additional suggested HA rules and configuration details over time. SGI will update the sac-hae-initial-setup.conf file for new releases but since that file is only used once ever, you could miss out on configuration updates.

SGI intends to also release HA configuration suggestions as service bulletins available on SGI SupportFolio.

Many of the components in SGI Management Center require a license. The license is associated with the virtual machine, and not the physical machine. For this reason, you will need to request the license based on the virtual machine environment once SGI Management Center ICE is installed.

The default HAE rules supplied by SGI will ensure a virtual machine named "sac" is always running. If the machine shuts down, it will be restarted on one of the physical machines.

The machine can be accessed with the virt-manager command from the physical machine.

In to avoid unexpected results, it is best to use crm_gui or crm to stop the virt-clone service when you wish to shut down the admin node virtual machine. Likewise, use crm or crm_gui to turn the service back on when desired. If you use virt-manager to shut the machine down, HAE will turn it back on for you automatically.

The default csync2 configuration set up by SGI will keep your admin node virtual machine configuration up to date on both nodes. However, csync2 is not automatic.

Therefore, if you make adjustments to the virtual machine configuration, such as CPU or memory use, you should run csync2 -xv to synchronize to the other physical admin node.

Some sites may prefer a more targeted approach to the csync2 configuration as it relates to the libvirt configuration files.

One thing to keep in mind is that you need the virtual machine configuration files to be the same on both physical admin nodes because the guest needs to run on both. This also helps facilitate live migration of the running virtual machine (VM) from one physical host to the other.

For more information on csync2, see SUSE Linux Enterprise High Availability Extension (HAE) High Availability Guide.

The SGI configuration helper scripts set up the virtual machine guest for you in the correct manner. There is one thing the script puts in the XML configuration file that is hard to specify in virt-manager . This is the <emulator> tag.

If you create your own virtual machine with virt-manager , it will not use the /opt/sgi/sbin/sgi-emulator-wrapper wrapper script but instead have a path to qemu-kvm .

It is important that the machine use the SGI sgi-emulator-wrapper wrapper script because this script provides proper serial number information to the virtual machine. Currently, this information is not used for license checks but is used by ESP to identify the serial number for your overall system.

System admin controllers (admin nodes) have a special system-wide Altix ICE serial number embedded in them that ESP uses. So the simple wrapper script for the emulator passes on this serial number information to the guest admin node.

Using the crm_gui or crm command line, it is possible to migrate the virtual machine live from one physical machine to the other. This can be useful, for example, if the system administrator needs to perform maintenance on one physical machine.

Note that this migration method makes use of the ssh transport. If your configuration does not allow the root user to ssh between physical machines without a password, you will need to modify the virt-clone rules and associated primitives to use a different transport. This will likely involve the set up of certificates for authentication for libvirt between the two physical machines. That is not covered in this section.

You can use the command line or GUI to migrate a virtualized admin node. This section describes the GUI method.

To use the crm_gui GUI to migrate to a virtualized admin node, perform the following:

This implementation of HA-RLC for SGI Altix ICE requires two lead nodes per rack and an extra network interface card (NIC) per HA-RLC. The extra NIC provides a direct network connection between the HA-RLC for the HAE software.

HA-RLC is currently implemented only on SUSE Linux Enterprise Server 11 Service Pack 1 (SLES 11 SP1) using the SLES 11 SP1 High Availability Extension (HAE). This restriction applies to only the admin and the rack leader nodes.

All current updates to package in the SLES 11 SP1 distribution (distro) and the SLES11 SP1 HAE product must be available to install a HA-RLC Altix ICE cluster.

The major software components that comprise this solution include the following:

It is required that updates to the SLES11 SP1 distro and to the SLES11 SP1 HAE be used with HA-RLC, in particular, when creating the HA-RLC leader images.

For information on the SUSE Linux Enterprise High Availability Extension (HAE) software package, see theSUSE Linux Enterprise High Availability Extension High Availability Guide. This is required reading and the information contained in the SLES HAE guide is not replicated in this manual.

The HAE document is available in the sleha-guide_en-pdf package from Novell, Inc.

After the repositories are created, updated, and the HA Repository selected, this package can be installed on the admin node, as follows:

admin:~ # cinstallman --yum-node --node admin install sleha-guide_en-pdf

You can find the HAE guide at the following location:

/usr/share/doc/manual/sleha-guide_en-pdf/SLEHA-guide_en.pdf

It is required that all compute nodes in HA-RLC protected racks use tmpfs roots, which implements a memory resident operating system. With the tmpfs root, no compute node state information is stored on a RLC, as is the case when the compute nodes use NFS roots.

---

Warning: It is critical that the HA-RLC compute nodes be booted with a tmpfs root. Compute nodes booted with NFS root will crash if a HA-RLC fails.

---

Since compute nodes using tmpfs roots store no state information on a RLC, the HA-RLC can be rebooted, and protected services migrated, without affecting the compute node functionality.

The connection of the rack leader controller (RLC) to the racks must be via the LOCAL port of the individual rack unit (IRU) chassis management controller (CMC).

In the case of an RLC with redundant management network (RMN) and a separate BMC port, it means that there are three Ethernet connections to the rack, two of which go to LOCAL ports on the CMC and one (the BMC connection) that goes to an LL port. In order to have four LOCAL ports in a rack, there must be four IRUs in that rack.

In addition, there is a direct Ethernet connection between the two HA-RLCs of each rack. This provides a communication path between HA-RLC that does not pass through the stacked CMCs and avoids cluster management traffic (see Figure 2-43).

Figure 2-43. CMC Network Cabling Diagram for HA-RLC

The SUSE Linux Enterprise High Availability Extension (HAE) software manages resources. Examples of managed resources include blademond, dhcpd, and conserver . The HAE software stops, starts and monitors these resources. Only a single instance of these resources runs per rack. These three resources also happen to be Linux Standards Base (LSB) services.

With HA-RLC, all resources are co-located on one or the other HA-RLCs. The HA-RLC on which the resources are running can be said to "host" the resources and can be referred to as the "active" HA-RLC.

The resources can be migrated between the two HA-RLCs for a rack, either in response to administrative commands or in response to a failure detected with one of the managed resources.

For detailed information on HAE software, see the SUSE Linux Enterprise High Availability Extension High Availability Guide (see “ SLES11 SP1 High Availability Extension (HAE) Documentation”).

The HA-RLC hostnames and interfaces for a rack are distinguished from each other with the addition of a "1" or a "2" to the standard RLC interface names.

For example, here are the names of the HA-RLC interfaces for racks 1 and 2. The names of the interfaces for racks 3 to n follow this same naming scheme.

The Rack 1 HA-RLC hostnames and interfaces are named, as follows:

The Rack 2 HA-RLC interfaces are named, as follows:

Changes to default SGI Altix ICE networking were needed to implement HA-RLC. In particular, HA-RLC addresses on the VLAN_HEAD, VLAN_BMC, and VLAN_GBE networks have been implemented as IP aliases that are managed as HAE resources.

Each HA-RLC also has a physical address via which the node is accessible from the admin node, regardless of where the HAE resources are hosted.

The IP alias address associated with r1lead is a logical rack alias and requests to r1lead will be directed to whichever HA-RLC is currently hosting the resources.

In contrast, the IP physical addresses associated with r1lead1 and r1lead2 always connect to that node. The BMC console connect is via the physical address.

admin:~ # console r1lead1

The console connections to the physical address are not disrupted by migration of the resources. The ssh connections to the physical addresses are also not disrupted by migration of resources. The ssh connections to the logical addresses are disrupted by migration.

Figure 2-43 shows the CMC network cabling diagram with a redundant system admin controller and redundant rack leader controller.

Each pair of HA-RLC comprise a separate instance of a HAE cluster. There is no communication between the separate HAE instances, that is, HAE instances running on different racks are independent.

The HAE software controls resources. LSB init scripts such as blademond, dhcpd, and conserver map directly to HAE resources.

HAE supplied scripts can be used for multiple HAE resources. For example, the script ocf::heartbeat:IPaddr2 is used to implement the three IP aliases used by HA-RLC.

All HA-RLC HAE resources are co-located, which means that all resources run simultaneously on one or the other of each HA-RLC pair.

A stonith-ipmi resource runs on each of the HA-RLCs. These stonith resources do not migrate with failure. The stonith-ipmi resources monitor peer BMC, but stonith action is not enabled.

If there is a failure of one of the HAE managed resources, the resources are all migrated to the other HA-RLC. Resources can also be administratively migrated between the HA-RLC of a rack.

As the HAE instance running on a rack is independent of HAE instances running on other racks, resources cannot be migrated to a HA-RLC of a different rack.

The HAE configuration is loaded when the HA-RLC boot for the first time. It is not intended that configuration be changed except via SGI update.

---

Warning: Making changes to the SGI supplied HAE resources is not supported. Changes to HAE resources are expected to be done only with SGI release upgrades.

---

Install the ICE Admin Autoinstall SLES 11 SP1 DVD as described in “Installing SLES11 on the Admin Node”.

Do NOT proceed to the steps described in the “configure-cluster Command Cluster Configuration Tool” section until you have finished creating repositories, as described in “Create SLES11 SP1 and HAE Repositories”.

Before running the configure-cluster command for a cluster that will include HA-RLC, you must create updated repositories for the SLES 11 SP1 distro and for SLES 11 SP1 HAE.

---

Warning: Updated repositories are required to build working HA-RLC images

---

Since the default leader image is created by configure-cluster , it is necessary that the updated repositories exist before running configure-cluster.

Create updated repositories for the distro and HAE using the method of your choice. The method of obtaining updates with which to create update repositories will vary from site to site. An example for creating these repositories from local update sources follows.

Use the crepo --select command to select the repositories so that they are used when building the default images.

The Cluster Configuration Tool ( configure-cluster) is used to perform initial cluster configuration (see “configure-cluster Command Cluster Configuration Tool”).

It is required when installing a cluster that will include HA-RLC racks that the SLES11 SP1 distro and SLES11 SP1 HAE repositories, and update repositories, be created before running the configure-cluster command.

Although it is possible to recover from the failure to create the updates repositories before running configure-cluster, this is not the supported method of installing HA-RLC clusters.

The configure-cluster instructions in this section are a supplement to the installation instructions provided in “configure-cluster Command Cluster Configuration Tool”.

To configure a HA-RLC cluster, perform the following steps:

After completing the Initial Cluster Setup Tasks with configure-cluster described in “Initial Cluster Configuration with HA-RLC”, the HA-RLC racks can be installed.

admin:~ # tail -f /var/log/messages

admin:~ # discover --rack 1,ha=all

admin:~ # discover --rack 1,3,ha=all

admin:~ # console r1lead1
admin:~ # console r1lead2

admin:~ # tail -f /var/log/messages

admin:~ # ssh r1lead crm_mon -1
============
Last updated: Mon Apr 18 14:14:22 2011
Stack: openais
Current DC: r1lead1 - partition with quorum
Version: 1.1.5-5ce2879aa0d5f43d01629bc20edc6868a9352002
2 Nodes configured, 2 expected votes
7 Resources configured.

============

Online: [ r1lead1 r1lead2 ]

blademond      (lsb:blademond):        Started r1lead1
conserver      (lsb:conserver):        Started r1lead1
dhcpd  (lsb:dhcpd):    Started r1lead1
Resource Group: alias-group
     head-alias (ocf::heartbeat:IPaddr2):       Started r1lead1
     vlan1-alias        (ocf::heartbeat:IPaddr2):       Started r1lead1
     vlan2-alias        (ocf::heartbeat:IPaddr2):       Started r1lead1
stonith-ipmi-1 (stonith:external/ipmi):        Started r1lead1
stonith-ipmi-2 (stonith:external/ipmi):        Started r1lead2

r1lead1:~ # tail -f /var/log/blademond
admin:~ # tail -f /var/log/discover-rack

setup-new-blades: booting blades finished

Jun 17 21:39:25 r1lead1 blademond: calling discover-rack on admin node
Jun 17 21:45:41 r1lead1 blademond: moving /var/opt/sgi/lib/blademond/slot_map.new to 
                                   /var/opt/sgi/lib/blademond/slot_map
Jun 17 21:45:42 r1lead1 blademond: INFO: pushing /var/opt/sgi/lib/blademond/slot_map to 
                                   r1lead2:/var/opt/sgi/lib/blademond
Jun 17 21:45:43 r1lead1 blademond: INFO: pushing /etc/dhcpd.conf.d/ice.conf to
                                   r1lead2:/etc/dhcpd.conf.d/ice.conf
Jun 17 21:45:43 r1lead1 blademond: sleeping ...

admin:~ # cexec rack_1: uptime

admin:~ # cpower --status r1i*n*

admin:~ # pdsh -w r1lead1,r1lead2 ls -l /tftpboot/pxelinux.cfg/ | \
awk '{print $1,$9,$10,$11}' | dshbak -c
----------------
r1lead[1-2]
----------------

C0A89F0A -> /tftpboot/compute-sles11sp1/pxelinux.config-2.6.32.29-0.3-default-tmpfs
C0A89F19 -> /tftpboot/compute-sles11sp1/pxelinux.config-2.6.32.29-0.3-default-tmpfs
C0A89F1A -> /tftpboot/compute-sles11sp1/pxelinux.config-2.6.32.29-0.3-default-tmpfs
C0A89F1B -> /tftpboot/compute-sles11sp1/pxelinux.config-2.6.32.29-0.3-default-tmpfs
C0A89F1C -> /tftpboot/compute-sles11sp1/pxelinux.config-2.6.32.29-0.3-default-tmpfs
C0A89F1D -> /tftpboot/compute-sles11sp1/pxelinux.config-2.6.32.29-0.3-default-tmpfs

The discovery of a single HA-RLC makes it possible to add a replacement HA-RLC to the cluster. The feature also makes it possible to first discover one in a pair of HA_RLCs and later discover the other HA-RLC pair.

SGI recommends that the initial HA-RLC discovery be of both of the rack HA-RLCs, that is, using ha=all.

For rack 1, discover first one of the HA-RLC and then discover the second HA-RLC. For example, first discover r1lead1 and after r1lead1 is fully configured, HAE is running, and the compute nodes have booted, then discover the second HA-RLC, r1lead2.

To discover HA-RLC r1lead1, perform the following:

admin:~ # discover --rack 1,ha=1

After the configuration of r1lead1 completes, discover HA-RLC r1lead2, as follows:

admin:~ # discover --rack 1,ha=2

---

Warning: Before starting the discovery of the second HA-RLC, all configuration steps must have completed for the first HA-RLC. This means that the rack compute nodes will have booted. Overlapping individual discovery of HA-RLCs is NOT supported.

---

For additional information about installing a single HA-RLC, see the section below, "Discover replacement HA-RLC".

After the HA-RLC installation completes, there are multiple methods of monitoring the HAE status of a rack as described in this section.

admin:~ # ssh r1lead crm_mon -1
============
Last updated: Tue May 17 15:27:23 2011
Stack: openais
Current DC: r1lead1 - partition with quorum
Version: 1.1.5-5ce2879aa0d5f43d01629bc20edc6868a9352002
2 Nodes configured, 2 expected votes
7 Resources configured.
============

Online: [ r1lead1 r1lead2 ]

 blademond      (lsb:blademond):        Started r1lead1
 conserver      (lsb:conserver):        Started r1lead1
 dhcpd  (lsb:dhcpd):    Started r1lead1
 Resource Group: alias-group
     head-alias (ocf::heartbeat:IPaddr2):       Started r1lead1
     vlan1-alias        (ocf::heartbeat:IPaddr2):       Started r1lead1
     vlan2-alias        (ocf::heartbeat:IPaddr2):       Started r1lead1
 stonith-ipmi-1 (stonith:external/ipmi):        Started r1lead1
 stonith-ipmi-2 (stonith:external/ipmi):        Started r1lead2

The list and order of resources may differ with release version.

One stonith resource runs on each HA-RLC and only functions to monitor
the BMC of the peer HA-RLC.  If only a single HA-RLC is installed,
there will be no stonith-impi resources configured.

sys-name:~ # pdsh -g leader crm_mon -s
r2lead2: Ok: 2 nodes online, 6 resources configured
r1lead2: Ok: 2 nodes online, 6 resources configured
r1lead1: Ok: 2 nodes online, 6 resources configured
r2lead1: Ok: 2 nodes online, 6 resources configured

crm_gui Utility

The crm_gui runs from a login to a HA-RLC. This tool provides significant functionality for working with HAE.

---

Warning: Making changes to the HAE configuration is not supported.

---

From the r1lead1 leader node, launch crm_gui, as follows:

r1lead1# crm_gui &

The CRM GUI appears, as shown in Figure 2-44.

Figure 2-44. CRM Cluster Status GUI

Hawk Browser Interface

The Hawk browser interface provides status and some basic commands for an individual HAE instance.

Start the Firefox browser on the admin node and point to the Hawk server (port 7630) at a HA-RLC physical interface. If pointed at a HA-RLC alias, the connection is dropped when the resources are migrated.

For example, to connect to the rack 1 HAE instance via the r1lead1 leader node, enter the following in your Firefox browser: https://r1lead1:7630/ .

Login as hacluster, using the default password sgisgi. Confirm the security exception. The default user hacluster is configured for HA-RLC in /etc/opt/sgi/conf.d/80-corosync .

By left-clicking on the wrench icon of a resource, the commands: Start, Stop, Move, Drop Relocation Rule and Clean Up run on a resource, as shown in Figure 2-45.

By left-clicking on the wrench icon of a HA-RLC, Online , Offline, and Fence Node functions are available, as shown in Figure 2-50. Fencing is not implemented.

For more information, see "Starting the HA Web Console and Logging In" in the SUSE Linux Enterprise High Availability Extension High Availability Guide.

Figure 2-45. Hawk Interface with Resource Control Pop-up Menu

Migrating resources is very useful for testing and is also useful for moving resources from a HA-RLC if maintenance is needed on that HA-RLC.

Three tools that can be used to migrate resources are , as follows:

---

Note: In response to an administrative migration using crm or using crm_gui, HAE creates a location constraint that forces the resources to migrate. This location constraint must be deleted for it to be possible for the resources to migrate back.

---

admin:~ # ssh r1lead crm resource migrate alias-group r1lead2
admin:~ # ssh r1lead crm_mon -1
admin:~ # ssh r1lead crm configure show | grep prefer
admin:~ # ssh r1lead crm configure delete cli-prefer-alias-group
admin:~ # ssh r1lead crm configure show | grep prefer
admin:~ # ssh r1lead crm_mon -1

crm_gui - Resource Migrate

For running the crm_gui from a HA-RLC, it is best to login via a physical interface. With the migration, a login via the rack HA alias is dropped.

For example, to login to r1lead1 and start crm_gui, perform the following:

admin:~ # ssh r1lead1 r1lead1# crm_gui &

For migrating all of the resources (all that migrate), migrate the alias-group, as shown in Figure 2-46 and Figure 2-47.

Figure 2-46. crm_gui Migrate alias-group Resource Example

Figure 2-47. crm_gui Migrate alias-group Resource Example

As with crm resource migrate, manual migration via the crm_gui creates a location constraint. This preferred location constraint must be deleted after the migration, either via the Constraints screen or via the button at the top of the Management screen, as shown in Figure 2-48.

Figure 2-48. CRM GUI Configuration Constraints Selection Example

Hawk Browser Interface -- Resource Migration

The Hawk browser interface provides status and some basic commands for an individual HAE instance.

Start Firefox on the admin node and point to the Hawk server (port 7630) at a HA-RLC physical interface. If pointed at a HA-RLC alias, the connection is dropped when the resources are migrated.

For example, to connect to the rack 1 HAE instance via r1lead1, enter the following: https://r1lead1:7630/ .

Login as hacluster and with the default password sgisgi. Confirm the security exception. The default hacluster user is configured for HA-RLC in /etc/opt/sgi/conf.d/80-corosync .

By left-clicking on the wrench icon of a resource, Start , Stop, Move, Drop Relocation Rule, and Clean Up run on a resource, as shown in Figure 2-49.

By clicking on the wrench icon of a HA-RLC, Online, Offline, and Fence Node functions are available, as shown in Figure 2-50. Fencing is not implemented.

For more information, see "Starting the HA Web Console and Logging In" in the SUSE Linux Enterprise High Availability Extension High Availability Guide.

This section describes two methods of resource migration using the Hawk browser interface.

Migrate Using Hawk - Method 1

On the Hawk interface, left-click on the alias-group wrench icon and select move, and then select the leader node to which to migrate the resources, as shown in Figure 2-49.

Figure 2-49. Using Hawk Browser Interface to Migrate Resources (Method 1 of 2)

Monitor that the resources do migrate. If they do not migrate, it is probably necessary to clear the migration constraint.

Clear the migration constraint with Drop Relocation Rule , which is accessible with a left-click on the alias-group wrench.

Migrate Using Hawk - Method 2

The Hawk interface commands Standby and Online can be used to migrate resources between HA-RLC, as shown in Figure 2-50.

Figure 2-50. Using Hawk Browser Interface to Migrate Resources (Method 2 of 2)

With both HA-RLCs online, and all of the resources started, select the wrench icon associated with the HA-RLC which is currently hosting the resources. Select Standby from the pop-up menu. Placing this HA-RLC in standby mode causes the resources to be migrated to the online HA-RLC.

This action does not create a migration constraint.

To migrate the resources back to the first HA-RLC, select online from the pop-up menu associated with the first HA-RLC (which is in standby mode) to bring it back online.

Then force the HA-RLC hosting the resources into standby mode.

After the resources have migrated, bring the HA-RLC in standby mode back online.

It is possible to take a HA-RLC offline, remove it from the cluster, and to then discover a replacement HA-RLC.

In this context "take offline" can include failure of the HA-RLC.

admin:~ # discover --rack 1,ha=1