Welcome to the Middleware section
Installation and configuration of a gLite 3.1 Compute Element on SL
Available Information
For an example, we'll show you how to install a Compute Element with Torque support on the GILDA VO and the Site BDII on the same host.
For a general description of what a Compute Element is, please see the
Compute Element Section on the
Middleware page.
This section explains how to install a CE on Scientific Linux. If you haven't already done so, please have a look at the
General Installation Procedure.
Pre-install Instructions
Please install Scientific Linux as described in the
General Installation Instructions.
Type of CE to Install
There are three different types of CE that can be installed.
- CE only
- CE with Torque Support
- CE with LSF Support
If you've decided what kind of a CE you're going to install, you also need to decide if you're going to install the Site BDII on the same host or not.
Compute Element Only
This type of installation is performed without setting up a batch system(LSF/Torque/PBS). Only a scheduler is installed and configured. Torque will also be installed, but not configured. This is due to dependencies with the Maui scheduler.
Compute Element with Torque
This installation will install and configure a Compute Element with support for Torque too. When the YAIM configuration is run, all the needed queues will be created and the group permissions to the queues will be set.
Compute Element with LSF
This type of installation will install a Compute Element with support for LSF. LSF itself will not be installed by this installation, but LSF will be configured.
LSF is licensed software and therefore the CE installation won't install LSF for you.
Host Certificates
You need host certificates that are signed by a trusted CA for your CE to function correctly. If you've received your certificates please install it into the following path with the following permissions:

Install the host-certificates:
mkdir /etc/grid-security
cp your_host_certificate /etc/grid-security/hostcert.pem
cp your_host_private_key /etc/grid-security/hostkey.pem
chmod 644 /etc/grid-security/hostcert.pem
chmod 400 /etc/grid-security/hostkey.pem
Download the Repositories
You can download the repositories as mentioned in the
Repository Settings Section.

Download the Repositories for CE with torque:
REPOS="ca dag ig jpackage gilda glite-lcg_ce_torque"
for name in $REPOS; do wget http://grid018.ct.infn.it/mrepo/repos/$name.repo -O /etc/yum.repos.d/$name.repo; done
Install Java and JDK

Please ensure that you have the correct Java engine installed before configuring services with YAIM.

Install Java
yum -y install xml-commons-resolver12 java-1.6.0-sun-compat jdk
Install the CA Certificates
Your CE will have to be able to verify certificates against the CA certificates. If you've downloaded the repos, you'll be able to install these certificates by using yum.

Install the CA certificates:
Install the CE
You can install the CE packages through yum. As mentioned earlier, you can install one of three types of CE. Please see the
Type of CE section? for a description of each.

Install CE with Torque:
yum -y install ig_CE_torque

Install the Site BDII on the same host (In this case)
Install the GILDA Utils
If you're going to connect to the
GILDA VO, you'll need to install the GILDA Utils. This package contains a list of users and groups that needs to be set.

Install GILDA Utils
yum -y install gilda_utils
Configure your Site Info File
The site-info.def file, is a file with a list of variables that is needed to configure your services. An example template is provided by the gilda-utils package and this file can be modified (by using a text editor of your choice) to be used in your installation.

Copy the ig-site-info.def template file provided by ig_yaim into the gilda dir and customise it:
cp /opt/glite/yaim/examples/siteinfo/site-info.def /opt/glite/yaim/etc/gilda/site-info.def

It could be useful to make a symbolic link to /etc/site-info.def for easy access
ln -s /opt/glite/yaim/etc/gilda/site-info.def /etc
Now you need to edit this file and set the following values according to your site. If a variable doesn't exist in this file, you can just add a new one and set its value.

For a full description of all these settings visit the
gLite website.

Add these Global values in your site-info.def file:
RB_HOST=srvslngrd010.uct.ac.za
WMS_HOST=srvslngrd010.uct.ac.za
PX_HOST=myproxy.cnaf.infn.it
LFC_HOST=devslngrd002.uct.ac.za
BDII_HOST=devslngrd001.uct.ac.za

Configure at least these settings in your site-info.def file
(change the values to reflect your site):
CE_HOST=YOUR_CE_HOST_NAME
TORQUE_SERVER=$CE_HOST
WN_LIST=/opt/glite/yaim/etc/gilda/wn-list.conf
GROUPS_CONF=/opt/glite/yaim/etc/gilda/ig-groups.conf
USERS_CONF=/opt/glite/yaim/etc/gilda/ig-users.conf
JAVA_LOCATION="/usr/java/latest"
SITE_EMAIL="grid-prod@$MY_DOMAIN" #Info may be e-mailed to this address
SITE_NAME=MY_SITENAME #Your site's name
SITE_LOC="Bloemfontein, SOUTH AFRICA" #Your location. City, COUNTRY
SITE_LAT=-29.063 #Your institude's latitude (Max 3 des)
SITE_LONG=26.116 #Your institude's longitude (Max 3 des)
SITE_WEB=http://www.sagrid.ac.za #Your institude's web site
#DON'T SET THESE TWO VALUES ANYMORE.
#Comment them out if they exist.
#SITE_TIER="GILDA Testbed" #Site's Tier - Don't set anymore
#SITE_SUPPORT_SITE="grid-prod@$MY_DOMAIN" #Site support e-mail address. Don't set anymore
SITE_DESC="South Africa - University of ......" #Put a meaningful description of your site here
JOB_MANAGER=lcgpbs #Set the type of batch server that you are using ( lcgsge if using SGE )
CE_BATCH_SYS=pbs #Set your batch system type
TORQUE_SERVER=$CE_HOST #Specify your Torque server's name here.
BATCH_SERVER=$CE_HOST #The name of your batch server
BATCH_BIN_DIR=/usr/bin #The full path to your batch server's executables
BATCH_VERSION=torque-2.1.9-4 #The version of your batch server
#The following settings are focused on your WORKER NODES even if the parameter starts with CE_.
CE_CPU_MODEL=Opteron #The type of CPUs used on your WNs
CE_CPU_VENDOR=AMD #The Brand of CPUs of your WNs
CE_CPU_SPEED=3000 #The CPU Speed (In Hrtz) of your WNs
CE_OS=ScientificSL #The Operating System used on the WNs
CE_OS_RELEASE=4.6 #The OS version
CE_OS_VERSION=SL #An abbreviation used for your OS
CE_MINPHYSMEM=2048 #Minimum Physical Memory of WNs
CE_MINVIRTMEM=4096 #Minimum Virtual Memory of WNs
CE_SMPSIZE=2 #Number of SMPs per WN
CE_SI00=1000 #CPU Integer Base: see http://www.spec.org/cpu2000/results/cint2000.html
CE_SF00=1200 #CPU Float Base: see http://www.spec.org/cpu2000/results/cfp2000.html
CE_OUTBOUNDIP=TRUE #TRUE if outbound connectivity is enabled at your site, FALSE otherwise
CE_INBOUNDIP=TRUE #TRUE if inbound connectivity is enabled at your site, FALSE otherwise
DPM_HOST=se.$MY_DOMAIN #Your Closest SE to use
SE_LIST=$DPM_HOST #The SEs to be used at your site
BDII_REGIONS="CE SE BDII" #The type of services provided by your site
BDII_CE_URL="ldap://$CE_HOST:2170/mds-vo-name=resource,o=grid" #The CE URL that processes will use
BDII_SE_URL="ldap://$DPM_HOST:2170/mds-vo-name=resource,o=grid" #The SE URL that processes will use
BDII_HTTP_URL="http://hep.phy.uct.ac.za/grid/bdii-update.conf" #The BDII sites on SAGrid that processes will use
VOS="gilda" #The VOS that your site belongs to
ALL_VOMS="gilda" #The VOMS that your site belongs to
QUEUES="short long infinite gilda" #List of available queues to the grid
SHORT_GROUP_ENABLE=$VOS #Enable Access for all the VOS to this Queue
LONG_GROUP_ENABLE=$VOS #Enable Access for all the VOS to this Queue
INFINITE_GROUP_ENABLE=$VOS #Enable Access for all the VOS to this Queue
SHORT_GROUP_ENABLE=$VOS #Enable Access for all the VOS to this Queue
LONG_GROUP_ENABLE=$VOS #Enable Access for all the VOS to this Queue
INFINITE_GROUP_ENABLE=$VOS #Enable Access for all the VOS to this Queue
GILDA_GROUP_ENABLE=gilda #Enable Access for the gilda group to this Queue
Configure the Worker Node List
The file specified as WN_LIST in your site-info.def file, has to contain a list of all your WNs hostnames. On a clean configuration, this file won't exist and needs to be created. List the full hostnames of the nodes you want to use. One host entry per line.

Edit the /opt/glite/yaim/etc/gilda/wn-list.conf file (or whatever you defined as WN_LIST):
cat > /opt/glite/yaim/etc/gilda/wn-list.conf <<EOF
node01.your_domain.ac.za
..
node20.your_domain.ac.za
EOF

The above statement is used for scripts, you may prefer to edit this file with a trusted text editor.

WARNING: Its important to configure this file before you run the YAIM configure command later in this document.
Configure the Users and Groups config files
Two files exists that contains the users' and groups' information respectively. These files are used to create new users on the CE. The default installation contains example files that you can use to create the needed users. If you have installed the gilda-utils too, another two example files available. Follow these instructions to add the needed users and groups.

Copy the general users and groups example files to /opt/glite/yaim/etc/gilda/
cp /opt/glite/yaim/examples/ig-groups.conf /opt/glite/yaim/etc/gilda/
cp /opt/glite/yaim/examples/ig-users.conf /opt/glite/yaim/etc/gilda/

Append gilda users and groups definitions to /opt/glite/yaim/etc/gilda/ig-users.conf if your site is a member of the GILDA VO
cat /opt/glite/yaim/etc/gilda/gilda_ig-users.conf >> /opt/glite/yaim/etc/gilda/ig-users.conf
cat /opt/glite/yaim/etc/gilda/gilda_ig-groups.conf >> /opt/glite/yaim/etc/gilda/ig-groups.conf
Run the YAIM configuration script
After you've installed and configured all the needed services by running the above statements, you're ready to configure the gLite services.

Before you run the update command, you may first test to see if your configuration file is correct by sourcing it. You shouldn't see any errors.
source /opt/glite/yaim/etc/gilda/site-info.def

If your configuration file is in the correct format, you may also want to test if you have defined all the needed variables before you do the physical configuration:
/opt/glite/yaim/bin/ig_yaim -v -s /opt/glite/yaim/etc/gilda/site-info.def -n ig_CE_torque -n BDII_site

To finally configure the services execute:
/opt/glite/yaim/bin/ig_yaim -c -s /opt/glite/yaim/etc/gilda/site-info.def -n ig_CE_torque -n BDII_site

NOTE:
# The -c parameter specifies to configure
# The -s /opt/... specifies where to find the site-info.def file
# The -n ig_CE_torque specifies to configure as a CE node with torque
# The -n BDII_site specifies to configure as a Site BDII Collector node
# More than one -n node-types can be specified as seen above
Testing
Testing BDII

Check if the local GRIS and the site BDII are running on CE and are publishing the right information (CPU, site name and so on)
ldapsearch -x -h YOUR_CE_HOSTNAME -p 2170 -b mds-vo-name=resource,o=grid

The result should look something like this:
# extended LDIF
#
# LDAPv3
# base <mds-vo-name=resource,o=grid> with scope sub
# filter: (objectclass=*)
# requesting: ALL
#
# resource, grid dn: Mds-Vo-name=resource,o=grid objectClass: GlueTop
...
...
...
...
# search result search: 2 result: 0 Success

Now search for your specific Site Name
ldapsearch -x -h YOUR_CE_HOSTNAME -p 2170 -b mds-vo-name=YOUR_SITE_NAME,o=grid

The results should look something like:
# extended LDIF
#
# LDAPv3
# base <mds-vo-name=YOUR_SITE_NAME,o=grid> with scope sub
# filter: (objectclass=*)
# requesting: ALL
#
...
...
...
...
# search result search: 2 result: 0 Success
# numResponses: 39
# numEntries: 38
Testing Torque
Log in as a normal non-root user.

Create a simple test job
cat > test.sh <<EOF
#!/bin/bash
sleep 20 #(its useful to see the job status)
hostname -f
EOF

Submit the test job to the local queue
Fast Installation Script
The following shell script is provided to show the steps that will be taken to install the CE gLite service.
Use this script at own risk.
#######Set Variables
HOST_TYPE=CE
SITE_INFO_FILE=/etc/site-info.def #Where the site-info.def file should reside
INSTALL_HOST=vm.sagrid.ac.za #The host where to copy the site-info.def file and certificates from
INSTALL_BASE=/installs/Grid/Certs #The path where to copy the certificates from
CMD_HOSTNAME=/bin/hostname #The command to execute to determine the hostname
CMD_NSLOOKUP=/usr/bin/nslookup #The command to execute to resolve the ip-address from the DNS
REPOS="ca dag ig jpackage gilda" #Repos that should be downloaded
CONFIG_CMD="/opt/glite/yaim/bin/ig_yaim" #The Yaim config command
CONFIG_TEST="-v" #The command option to parse, to verify the config file
CONFIG_OPTIONS="-c" #The command option to parse, to configure the services
CONFIG_FILE="-s \$SITE_INFO_FILE" #How to load the site-info.def config file
RECONFIG_CMD="/root/bin/reconfig.grid" #A file to create to re-execute the config at a later stage
YUM_PACKAGES="ntp xml-commons-resolver11 java-1.5.0-sun-compat jdk lcg-CA" #Packages to install
REQ_CERTS=1 #Is a host certificate requested
EXIT_ON_ERROR=0 #Should script exit if an error is encountered
#If this script is executed directly in a bash shell, don't exit the shell on errors.
[ $0 == "/bin/bash" ] || EXIT_ON_ERROR=1
#Set specific variables for different installation types
REPOS="$REPOS glite-lcg_ce_torque"
REQ_VARS="JAVA_LOCATION RB_HOST WMS_HOST LB_HOST PX_HOST HLR_RESOURCE LFC_HOST BDII_HOST CE_HOST TORQUE_SERVER WN_LIST GROUPS_CONF \
USERS_CONF SITE_EMAIL SITE_NAME SITE_LOC SITE_LAT SITE_LONG SITE_WEB SITE_DESC JOB_MANAGER CE_BATCH_SYS TORQUE_SERVER BATCH_SERVER \
BATCH_BIN_DIR BATCH_VERSION CE_CPU_MODEL CE_CPU_VENDOR CE_CPU_SPEED CE_OS CE_OS_RELEASE CE_OS_VERSION CE_MINPHYSMEM CE_MINVIRTMEM \
CE_SMPSIZE CE_SI00 CE_SF00 CE_OUTBOUNDIP CE_INBOUNDIP DPM_HOST SE_LIST BDII_REGIONS BDII_CE_URL BDII_SE_URL BDII_HTTP_URL VOS \
ALL_VOMS QUEUES"
YAIM_OPTIONS="ig_CE_torque ig_BDII_site"
YUM_PACKAGES="$YUM_PACKAGES ig_CE_torque ig_BDII gilda_utils"
FW_UDP_PORTS=""
FW_TCP_PORTS="2119 2170:2173 9002 20000:25000"
REF_SITE="https://twiki.cern.ch/twiki/bin/view/LCG/Site-info_configuration_variables#BDII \
http://twiki.cern.ch/twiki/bin/view/LCG/Site-info_configuration_variables#cream_CE"
POST_CONFIG=""
#Test to see if the FQDN is set
if [ `$CMD_HOSTNAME -s` == `$CMD_HOSTNAME -f` ]; then
echo "The hostname and the domain needs to be set first"
echo "Set the FQDN in /etc/hosts, /etc/sysconfig/network and execute:"
echo "hostname full-hostname.full-domain"
[ $EXIT_ON_ERROR -eq 0 ] || exit 1
fi
#Get the ip-address from the DNS server
IP_FROM_DNS=$($CMD_NSLOOKUP `$CMD_HOSTNAME -f`|grep Name -A 1|grep -v Name|sed "s/Address://g"|sed 's/[ \t]*//' )
if [ "$IP_FROM_DNS" != `$CMD_HOSTNAME -i` ]; then
echo "According to your DNS:"
echo "The IP Address for `$CMD_HOSTNAME -f` should be $IP_FROM_DNS"
echo "The IP Address is currently `$CMD_HOSTNAME -i`"
[ $EXIT_ON_ERROR -eq 0 ] || exit 1
else
echo "Running on `hostname -f`"
fi
######Set the Certificates' permissions
if [ $REQ_CERTS -ne 0 ]; then
mkdir /etc/grid-security
if [ -e /etc/grid-security/hostcert.pem ]; then
chmod 644 /etc/grid-security/hostcert.pem
chmod 400 /etc/grid-security/hostkey.pem
else
echo "Certificates could not be found under /etc/grid-security"
echo "Please coppy your hostkey.pem and hostcert.pem to /etc/grid-security"
[ $EXIT_ON_ERROR -eq 0 ] || exit 1
fi
else
echo "No Certificates needed for this host"
fi
###################################################################
echo
echo "Downloading the Repositories"
echo
for name in $REPOS
do
wget http://grid-it.cnaf.infn.it/mrepo/repos/$name.repo -O /etc/yum.repos.d/$name.repo
done
#Download certain Repos from different host
GILDA_REPOS="ca gilda"
for name in $GILDA_REPOS
do
wget http://grid018.ct.infn.it/mrepo/repos/$name.repo -O /etc/yum.repos.d/$name.repo
done
###################################################################
echo
echo "Installing Packages:"
echo $YUM_PACKAGES
echo
INSTALL_DONE=""
INSTALL_FAILED=""
for package in $YUM_PACKAGES
do
echo
echo "Trying to install $package:"
echo
yum -y install $package
if [ $? -ne 0 ]; then
echo
echo "Failed to install package $package"
echo
INSTALL_FAILED="$INSTALL_FAILED $package"
else
INSTALL_DONE="$INSTALL_DONE $package"
fi
done
echo
echo "Installed Packages: $INSTALL_DONE"
if [ "x$INSTALL_FAILED" != "x" ]; then
echo "Failed to Install: $INSTALL_FAILED"
[ $EXIT_ON_ERROR -eq 0 ] || exit 1
fi
###################################################################
echo
echo "Configuring the IPTables Firewall"
echo
/sbin/iptables -D RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
if [ "x$FW_TCP_PORTS" != "x" ]; then
for port in $FW_TCP_PORTS
do
/sbin/iptables -D RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport $port -j ACCEPT
/sbin/iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport $port -j ACCEPT
/sbin/iptables -D RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --sport $port -j ACCEPT
/sbin/iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --sport $port -j ACCEPT
done
fi
if [ "x$FW_UDP_PORTS" != "x" ]; then
for port in $FW_UDP_PORTS
do
/sbin/iptables -D RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport $port -j ACCEPT
/sbin/iptables -A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport $port -j ACCEPT
/sbin/iptables -D RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --sport $port -j ACCEPT
/sbin/iptables -A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --sport $port -j ACCEPT
done
fi
iptables -A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
/sbin/iptables-save > /etc/sysconfig/iptables
/etc/init.d/iptables restart
###################################################################
echo
echo "Retrieving site-info.def file"
echo
ln -s $SITE_INFO_FILE /etc
chown root: $SITE_INFO_FILE
chmod 640 $SITE_INFO_FILE
if ! [ -r $SITE_INFO_FILE ]; then
echo "The file $SITE_INFO_FILE was not found or is not readable"
[ $EXIT_ON_ERROR -eq 0 ] || exit 2
fi
###################################################################
echo
echo "Testing the site configuration file $SITE_INFO_FILE:"
echo
source $SITE_INFO_FILE >> /dev/null
if [ $? -ne 0 ]; then
echo "Invalid syntax in file: '$SITE_INFO_FILE'"
[ $EXIT_ON_ERROR -eq 0 ] || exit 2
fi
#Verify that required variables are set
error_var=""
for var in $REQ_VARS
do
temp_var=$(cat $SITE_INFO_FILE|grep "^[\W]*$var=.*$")
if [ "x$temp_var" == "x" ]; then
error_var="$error_var $var"
fi
done
if [ "x$error_var" != "x" ]; then
echo "The following variables should be set in $SITE_INFO_FILE"
echo "$error_var"|sed "s/\W/\n/g"
[ $EXIT_ON_ERROR -eq 0 ] || exit 3
fi
###################################################################
echo
echo "Creating $RECONFIG_CMD to reconfigure node when needed"
echo
#Set the command line options for the yaim configure script
for option in $YAIM_OPTIONS
do
OPTIONS="$OPTIONS -n $option"
done
mkdir -p `dirname $RECONFIG_CMD`
cat > $RECONFIG_CMD << EOF
#!/bin/bash
##For more information see:
# $REF_SITE
SITE_INFO_FILE=$SITE_INFO_FILE
source \$SITE_INFO_FILE
RESULT=\$?
if [ \$RESULT -eq 0 ]; then
$CONFIG_CMD $CONFIG_TEST $CONFIG_FILE $OPTIONS
RESULT=\$?
fi
if [ \$RESULT -eq 0 ]; then
$CONFIG_CMD $CONFIG_OPTIONS $CONFIG_FILE $OPTIONS
else
echo "The file-syntax in '\$SITE_INFO_FILE' is invalid"
echo "Script can not continue"
exit 1
fi
exit 0
EOF
chmod 750 $RECONFIG_CMD
###################################################################
echo
echo "Copying MPI-services to $(dirname $SITE_INFO_FILE)"
echo
if [ -e /opt/glite/yaim/examples/siteinfo/services ]; then
/bin/cp -pvr /opt/glite/yaim/examples/siteinfo/services $(dirname $SITE_INFO_FILE)
fi
###################################################################
echo
echo "Configuring User and Group files"
echo
/bin/cp -f /opt/glite/yaim/examples/ig-groups.conf /opt/glite/yaim/etc/gilda/
/bin/cp -f /opt/glite/yaim/examples/ig-users.conf /opt/glite/yaim/etc/gilda/
cat /opt/glite/yaim/etc/gilda/gilda_ig-users.conf >> /opt/glite/yaim/etc/gilda/ig-users.conf
cat /opt/glite/yaim/etc/gilda/gilda_ig-groups.conf >> /opt/glite/yaim/etc/gilda/ig-groups.conf
###################################################################
echo ##############################################################
echo
echo "Done configuring the host as a $HOST_TYPE node"
echo
echo "The host is now ready to be configured."
echo "To configure the host execute:"
echo ". $RECONFIG_CMD"
[ $EXIT_ON_ERROR -eq 0 ] || exit 0
--
VanEckA - 19 Aug 2008