Welcome to the Middleware section

Installation and configuration of a gLite 3.1 Compute Element on SL

Available Information

HELP For an example, we'll show you how to install a Compute Element with Torque support on the GILDA VO and the Site BDII on the same host.

For a general description of what a Compute Element is, please see the Compute Element Section on the Middleware page.

This section explains how to install a CE on Scientific Linux. If you haven't already done so, please have a look at the General Installation Procedure.

Pre-install Instructions

Please install Scientific Linux as described in the General Installation Instructions.

Type of CE to Install

There are three different types of CE that can be installed.

  1. CE only
  2. CE with Torque Support
  3. CE with LSF Support

If you've decided what kind of a CE you're going to install, you also need to decide if you're going to install the Site BDII on the same host or not.

Compute Element Only

This type of installation is performed without setting up a batch system(LSF/Torque/PBS). Only a scheduler is installed and configured. Torque will also be installed, but not configured. This is due to dependencies with the Maui scheduler.

Compute Element with Torque

This installation will install and configure a Compute Element with support for Torque too. When the YAIM configuration is run, all the needed queues will be created and the group permissions to the queues will be set.

Compute Element with LSF

This type of installation will install a Compute Element with support for LSF. LSF itself will not be installed by this installation, but LSF will be configured. LSF is licensed software and therefore the CE installation won't install LSF for you.

Host Certificates

You need host certificates that are signed by a trusted CA for your CE to function correctly. If you've received your certificates please install it into the following path with the following permissions:

HELP Install the host-certificates:

mkdir /etc/grid-security
cp your_host_certificate /etc/grid-security/hostcert.pem
cp your_host_private_key /etc/grid-security/hostkey.pem
chmod 644 /etc/grid-security/hostcert.pem
chmod 400 /etc/grid-security/hostkey.pem

Download the Repositories

You can download the repositories as mentioned in the Repository Settings Section.

HELP Download the Repositories for CE with torque:

REPOS="ca dag ig jpackage gilda glite-lcg_ce_torque"
for name in $REPOS; do wget http://grid018.ct.infn.it/mrepo/repos/$name.repo -O /etc/yum.repos.d/$name.repo; done

Install Java and JDK

ALERT! Please ensure that you have the correct Java engine installed before configuring services with YAIM.

HELP Install Java

yum -y install xml-commons-resolver12 java-1.6.0-sun-compat jdk

Install the CA Certificates

Your CE will have to be able to verify certificates against the CA certificates. If you've downloaded the repos, you'll be able to install these certificates by using yum.

HELP Install the CA certificates:

yum -y install lcg-CA

Install the CE

You can install the CE packages through yum. As mentioned earlier, you can install one of three types of CE. Please see the Type of CE section? for a description of each.

HELP Install CE with Torque:

yum -y install ig_CE_torque

HELP Install the Site BDII on the same host (In this case)

yum -y install ig_BDII

Install the GILDA Utils

If you're going to connect to the GILDA VO, you'll need to install the GILDA Utils. This package contains a list of users and groups that needs to be set.

HELP Install GILDA Utils

yum -y install gilda_utils

Configure your Site Info File

The site-info.def file, is a file with a list of variables that is needed to configure your services. An example template is provided by the gilda-utils package and this file can be modified (by using a text editor of your choice) to be used in your installation.

HELP Copy the ig-site-info.def template file provided by ig_yaim into the gilda dir and customise it:

cp /opt/glite/yaim/examples/siteinfo/site-info.def /opt/glite/yaim/etc/gilda/site-info.def
TIP It could be useful to make a symbolic link to /etc/site-info.def for easy access
ln -s /opt/glite/yaim/etc/gilda/site-info.def /etc

Now you need to edit this file and set the following values according to your site. If a variable doesn't exist in this file, you can just add a new one and set its value.

IDEA! For a full description of all these settings visit the gLite website.

HELP Add these Global values in your site-info.def file:

RB_HOST=srvslngrd010.uct.ac.za
WMS_HOST=srvslngrd010.uct.ac.za
PX_HOST=myproxy.cnaf.infn.it
LFC_HOST=devslngrd002.uct.ac.za
BDII_HOST=devslngrd001.uct.ac.za

HELP Configure at least these settings in your site-info.def file (change the values to reflect your site):

CE_HOST=YOUR_CE_HOST_NAME
TORQUE_SERVER=$CE_HOST
WN_LIST=/opt/glite/yaim/etc/gilda/wn-list.conf
GROUPS_CONF=/opt/glite/yaim/etc/gilda/ig-groups.conf
USERS_CONF=/opt/glite/yaim/etc/gilda/ig-users.conf
JAVA_LOCATION="/usr/java/latest"
SITE_EMAIL="grid-prod@$MY_DOMAIN"                       #Info may be e-mailed to this address
SITE_NAME=MY_SITENAME                                   #Your site's name
SITE_LOC="Bloemfontein, SOUTH AFRICA"                   #Your location. City, COUNTRY
SITE_LAT=-29.063                                        #Your institude's latitude (Max 3 des)
SITE_LONG=26.116                                        #Your institude's longitude (Max 3 des)
SITE_WEB=http://www.sagrid.ac.za                        #Your institude's web site

#DON'T SET THESE TWO VALUES ANYMORE.
#Comment them out if they exist.

#SITE_TIER="GILDA Testbed"                              #Site's Tier - Don't set anymore
#SITE_SUPPORT_SITE="grid-prod@$MY_DOMAIN"               #Site support e-mail address. Don't set anymore

SITE_DESC="South Africa - University of ......"         #Put a meaningful description of your site here
JOB_MANAGER=lcgpbs                                      #Set the type of batch server that you are using ( lcgsge if using SGE )
CE_BATCH_SYS=pbs                                        #Set your batch system type
TORQUE_SERVER=$CE_HOST                                  #Specify your Torque server's name here.
BATCH_SERVER=$CE_HOST                                   #The name of your batch server
BATCH_BIN_DIR=/usr/bin                                  #The full path to your batch server's executables
BATCH_VERSION=torque-2.1.9-4                            #The version of your batch server

#The following settings are focused on your WORKER NODES even if the parameter starts with CE_.
CE_CPU_MODEL=Opteron                                    #The type of CPUs used on your WNs
CE_CPU_VENDOR=AMD                                       #The Brand of CPUs of your WNs
CE_CPU_SPEED=3000                                       #The CPU Speed (In Hrtz) of your WNs
CE_OS=ScientificSL                                      #The Operating System used on the WNs
CE_OS_RELEASE=4.6                                       #The OS version
CE_OS_VERSION=SL                                        #An abbreviation used for your OS
CE_MINPHYSMEM=2048                                      #Minimum Physical Memory of WNs
CE_MINVIRTMEM=4096                                      #Minimum Virtual Memory of WNs
CE_SMPSIZE=2                                            #Number of SMPs per WN
CE_SI00=1000                                            #CPU Integer Base: see http://www.spec.org/cpu2000/results/cint2000.html
CE_SF00=1200                                            #CPU Float Base: see http://www.spec.org/cpu2000/results/cfp2000.html
CE_OUTBOUNDIP=TRUE                                      #TRUE if outbound connectivity is enabled at your site, FALSE otherwise
CE_INBOUNDIP=TRUE                                       #TRUE if inbound connectivity is enabled at your site, FALSE otherwise
DPM_HOST=se.$MY_DOMAIN                                  #Your Closest SE to use
SE_LIST=$DPM_HOST                                       #The SEs to be used at your site
BDII_REGIONS="CE SE BDII"                               #The type of services provided by your site
BDII_CE_URL="ldap://$CE_HOST:2170/mds-vo-name=resource,o=grid"    #The CE URL that processes will use
BDII_SE_URL="ldap://$DPM_HOST:2170/mds-vo-name=resource,o=grid"   #The SE URL that processes will use
BDII_HTTP_URL="http://hep.phy.uct.ac.za/grid/bdii-update.conf"    #The BDII sites on SAGrid that processes will use
VOS="gilda"                                             #The VOS that your site belongs to
ALL_VOMS="gilda"                                        #The VOMS that your site belongs to
QUEUES="short long infinite gilda"                      #List of available queues to the grid
SHORT_GROUP_ENABLE=$VOS                                 #Enable Access for all the VOS to this Queue
LONG_GROUP_ENABLE=$VOS                                  #Enable Access for all the VOS to this Queue
INFINITE_GROUP_ENABLE=$VOS                              #Enable Access for all the VOS to this Queue
SHORT_GROUP_ENABLE=$VOS                                 #Enable Access for all the VOS to this Queue
LONG_GROUP_ENABLE=$VOS                                  #Enable Access for all the VOS to this Queue
INFINITE_GROUP_ENABLE=$VOS                              #Enable Access for all the VOS to this Queue
GILDA_GROUP_ENABLE=gilda                                #Enable Access for the gilda group to this Queue

Configure the Worker Node List

The file specified as WN_LIST in your site-info.def file, has to contain a list of all your WNs hostnames. On a clean configuration, this file won't exist and needs to be created. List the full hostnames of the nodes you want to use. One host entry per line.

HELP Edit the /opt/glite/yaim/etc/gilda/wn-list.conf file (or whatever you defined as WN_LIST):

cat > /opt/glite/yaim/etc/gilda/wn-list.conf  <<EOF
node01.your_domain.ac.za
..
node20.your_domain.ac.za
EOF

TIP The above statement is used for scripts, you may prefer to edit this file with a trusted text editor.

ALERT! WARNING: Its important to configure this file before you run the YAIM configure command later in this document.

Configure the Users and Groups config files

Two files exists that contains the users' and groups' information respectively. These files are used to create new users on the CE. The default installation contains example files that you can use to create the needed users. If you have installed the gilda-utils too, another two example files available. Follow these instructions to add the needed users and groups.

HELP Copy the general users and groups example files to /opt/glite/yaim/etc/gilda/

cp /opt/glite/yaim/examples/ig-groups.conf     /opt/glite/yaim/etc/gilda/
cp /opt/glite/yaim/examples/ig-users.conf      /opt/glite/yaim/etc/gilda/

HELP Append gilda users and groups definitions to /opt/glite/yaim/etc/gilda/ig-users.conf if your site is a member of the GILDA VO

cat /opt/glite/yaim/etc/gilda/gilda_ig-users.conf    >> /opt/glite/yaim/etc/gilda/ig-users.conf
cat /opt/glite/yaim/etc/gilda/gilda_ig-groups.conf   >> /opt/glite/yaim/etc/gilda/ig-groups.conf

Run the YAIM configuration script

After you've installed and configured all the needed services by running the above statements, you're ready to configure the gLite services.

IDEA! Before you run the update command, you may first test to see if your configuration file is correct by sourcing it. You shouldn't see any errors.

source  /opt/glite/yaim/etc/gilda/site-info.def

IDEA! If your configuration file is in the correct format, you may also want to test if you have defined all the needed variables before you do the physical configuration:

/opt/glite/yaim/bin/ig_yaim -v -s /opt/glite/yaim/etc/gilda/site-info.def -n ig_CE_torque -n BDII_site

HELP To finally configure the services execute:

/opt/glite/yaim/bin/ig_yaim -c -s /opt/glite/yaim/etc/gilda/site-info.def -n ig_CE_torque -n BDII_site

IDEA! NOTE:

# The -c parameter specifies to configure
# The -s /opt/... specifies where to find the site-info.def file
# The -n ig_CE_torque specifies to configure as a CE node with torque
# The -n BDII_site specifies to configure as a Site BDII Collector node

# More than one -n node-types can be specified as seen above

Testing

Testing BDII

HELP Check if the local GRIS and the site BDII are running on CE and are publishing the right information (CPU, site name and so on)
ldapsearch -x -h YOUR_CE_HOSTNAME -p 2170 -b mds-vo-name=resource,o=grid

IDEA! The result should look something like this:

# extended LDIF
#
# LDAPv3
# base <mds-vo-name=resource,o=grid> with scope sub
# filter: (objectclass=*)
# requesting: ALL
#
# resource, grid dn: Mds-Vo-name=resource,o=grid objectClass: GlueTop
...
...
...
...
# search result search: 2 result: 0 Success

HELP Now search for your specific Site Name

ldapsearch -x -h YOUR_CE_HOSTNAME -p 2170 -b mds-vo-name=YOUR_SITE_NAME,o=grid

IDEA! The results should look something like:

# extended LDIF
#
# LDAPv3
# base <mds-vo-name=YOUR_SITE_NAME,o=grid> with scope sub
# filter: (objectclass=*)
# requesting: ALL
#
...
...
...
...
# search result search: 2 result: 0 Success
# numResponses: 39
# numEntries: 38

Testing Torque

Log in as a normal non-root user.

HELP Create a simple test job

cat > test.sh  <<EOF
#!/bin/bash
sleep 20     #(its useful to see the job status)
hostname -f
EOF

HELP Submit the test job to the local queue

qsub -q short test.sh

Fast Installation Script

The following shell script is provided to show the steps that will be taken to install the CE gLite service.

HELP Use this script at own risk.

#######Set Variables
HOST_TYPE=CE
SITE_INFO_FILE=/etc/site-info.def               #Where the site-info.def file should reside
INSTALL_HOST=vm.sagrid.ac.za                    #The host where to copy the site-info.def file and certificates from
INSTALL_BASE=/installs/Grid/Certs               #The path where to copy the certificates from
CMD_HOSTNAME=/bin/hostname                      #The command to execute to determine the hostname
CMD_NSLOOKUP=/usr/bin/nslookup                  #The command to execute to resolve the ip-address from the DNS

REPOS="ca dag ig jpackage gilda"                #Repos that should be downloaded
CONFIG_CMD="/opt/glite/yaim/bin/ig_yaim"        #The Yaim config command
CONFIG_TEST="-v"                                #The command option to parse, to verify the config file
CONFIG_OPTIONS="-c"                             #The command option to parse, to configure the services
CONFIG_FILE="-s \$SITE_INFO_FILE"               #How to load the site-info.def config file
RECONFIG_CMD="/root/bin/reconfig.grid"          #A file to create to re-execute the config at a later stage
YUM_PACKAGES="ntp xml-commons-resolver11 java-1.5.0-sun-compat jdk lcg-CA"  #Packages to install
REQ_CERTS=1                                     #Is a host certificate requested
EXIT_ON_ERROR=0                                 #Should script exit if an error is encountered

#If this script is executed directly in a bash shell, don't exit the shell on errors.
[ $0 == "/bin/bash" ] || EXIT_ON_ERROR=1

#Set specific variables for different installation types
REPOS="$REPOS glite-lcg_ce_torque"
REQ_VARS="JAVA_LOCATION RB_HOST WMS_HOST LB_HOST PX_HOST HLR_RESOURCE LFC_HOST BDII_HOST CE_HOST TORQUE_SERVER WN_LIST GROUPS_CONF \
USERS_CONF SITE_EMAIL SITE_NAME SITE_LOC SITE_LAT SITE_LONG SITE_WEB SITE_DESC JOB_MANAGER CE_BATCH_SYS TORQUE_SERVER BATCH_SERVER \
BATCH_BIN_DIR BATCH_VERSION CE_CPU_MODEL CE_CPU_VENDOR CE_CPU_SPEED CE_OS CE_OS_RELEASE CE_OS_VERSION CE_MINPHYSMEM CE_MINVIRTMEM \
CE_SMPSIZE CE_SI00 CE_SF00 CE_OUTBOUNDIP CE_INBOUNDIP DPM_HOST SE_LIST BDII_REGIONS BDII_CE_URL BDII_SE_URL BDII_HTTP_URL VOS \
ALL_VOMS QUEUES"
YAIM_OPTIONS="ig_CE_torque ig_BDII_site"
YUM_PACKAGES="$YUM_PACKAGES ig_CE_torque ig_BDII gilda_utils"
FW_UDP_PORTS=""
FW_TCP_PORTS="2119 2170:2173 9002 20000:25000"
REF_SITE="https://twiki.cern.ch/twiki/bin/view/LCG/Site-info_configuration_variables#BDII \
http://twiki.cern.ch/twiki/bin/view/LCG/Site-info_configuration_variables#cream_CE"
POST_CONFIG=""     

#Test to see if the FQDN is set
if [ `$CMD_HOSTNAME -s` == `$CMD_HOSTNAME -f` ]; then
echo "The hostname and the domain needs to be set first"
echo "Set the FQDN in /etc/hosts, /etc/sysconfig/network and execute:"
echo "hostname full-hostname.full-domain"
[ $EXIT_ON_ERROR -eq 0 ] || exit 1
fi

#Get the ip-address from the DNS server
IP_FROM_DNS=$($CMD_NSLOOKUP `$CMD_HOSTNAME -f`|grep Name -A 1|grep -v Name|sed "s/Address://g"|sed 's/[ \t]*//' )

if [ "$IP_FROM_DNS" != `$CMD_HOSTNAME -i` ]; then
echo "According to your DNS:"
echo "The IP Address for `$CMD_HOSTNAME -f` should be $IP_FROM_DNS"
echo "The IP Address is currently `$CMD_HOSTNAME -i`"
[ $EXIT_ON_ERROR -eq 0 ] || exit 1
else
echo "Running on `hostname -f`"
fi

######Set the Certificates' permissions
if [ $REQ_CERTS -ne 0 ]; then
mkdir /etc/grid-security

if [ -e /etc/grid-security/hostcert.pem ]; then
chmod 644 /etc/grid-security/hostcert.pem
chmod 400 /etc/grid-security/hostkey.pem
else
echo "Certificates could not be found under /etc/grid-security"
echo "Please coppy your hostkey.pem and hostcert.pem to /etc/grid-security"
[ $EXIT_ON_ERROR -eq 0 ] || exit 1
fi
else
echo "No Certificates needed for this host"
fi
###################################################################
echo
echo "Downloading the Repositories"
echo

for name in $REPOS
do
wget http://grid-it.cnaf.infn.it/mrepo/repos/$name.repo -O /etc/yum.repos.d/$name.repo
done

#Download certain Repos from different host
GILDA_REPOS="ca gilda"
for name in $GILDA_REPOS
do
wget  http://grid018.ct.infn.it/mrepo/repos/$name.repo -O /etc/yum.repos.d/$name.repo
done
###################################################################
echo
echo "Installing Packages:"
echo $YUM_PACKAGES
echo

INSTALL_DONE=""
INSTALL_FAILED=""

for package in $YUM_PACKAGES
do
echo
echo "Trying to install $package:"
echo
yum -y install $package
if [ $? -ne 0 ]; then
echo
echo "Failed to install package $package"
echo
INSTALL_FAILED="$INSTALL_FAILED $package"
else
INSTALL_DONE="$INSTALL_DONE $package"
fi
done
echo
echo "Installed Packages: $INSTALL_DONE"

if [ "x$INSTALL_FAILED" != "x" ]; then
echo "Failed to Install: $INSTALL_FAILED"
[ $EXIT_ON_ERROR -eq 0 ] || exit 1
fi
###################################################################
echo
echo "Configuring the IPTables Firewall"
echo

/sbin/iptables -D RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited

if [ "x$FW_TCP_PORTS" != "x" ]; then
for port in $FW_TCP_PORTS
do
/sbin/iptables -D RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport $port -j ACCEPT
/sbin/iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport $port -j ACCEPT
/sbin/iptables -D RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --sport $port -j ACCEPT
/sbin/iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --sport $port -j ACCEPT
done
fi

if [ "x$FW_UDP_PORTS" != "x" ]; then
for port in $FW_UDP_PORTS
do
/sbin/iptables -D RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport $port -j ACCEPT
/sbin/iptables -A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport $port -j ACCEPT
/sbin/iptables -D RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --sport $port -j ACCEPT
/sbin/iptables -A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --sport $port -j ACCEPT
done
fi

iptables -A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited

/sbin/iptables-save > /etc/sysconfig/iptables

/etc/init.d/iptables restart
###################################################################
echo
echo "Retrieving site-info.def file"
echo

ln -s $SITE_INFO_FILE /etc
chown root: $SITE_INFO_FILE
chmod 640 $SITE_INFO_FILE

if ! [ -r $SITE_INFO_FILE ]; then
echo "The file $SITE_INFO_FILE was not found or is not readable"
[ $EXIT_ON_ERROR -eq 0 ] || exit 2
fi
###################################################################
echo
echo  "Testing the site configuration file $SITE_INFO_FILE:"
echo

source $SITE_INFO_FILE >> /dev/null

if [ $? -ne 0 ]; then
echo "Invalid syntax in file: '$SITE_INFO_FILE'"
[ $EXIT_ON_ERROR -eq 0 ] || exit 2
fi

#Verify that required variables are set
error_var=""
for var in $REQ_VARS
do
temp_var=$(cat $SITE_INFO_FILE|grep "^[\W]*$var=.*$")
if [ "x$temp_var" == "x" ]; then
error_var="$error_var $var"
fi
done

if [ "x$error_var" != "x" ]; then
echo "The following variables should be set in $SITE_INFO_FILE"
echo "$error_var"|sed "s/\W/\n/g"
[ $EXIT_ON_ERROR -eq 0 ] || exit 3
fi
###################################################################
echo
echo "Creating $RECONFIG_CMD to reconfigure node when needed"
echo

#Set the command line options for the yaim configure script
for option in $YAIM_OPTIONS
do
OPTIONS="$OPTIONS -n $option"
done

mkdir -p `dirname $RECONFIG_CMD`
cat > $RECONFIG_CMD << EOF
#!/bin/bash

##For more information see:
# $REF_SITE

SITE_INFO_FILE=$SITE_INFO_FILE
source \$SITE_INFO_FILE
RESULT=\$?

if [ \$RESULT -eq 0 ]; then
$CONFIG_CMD $CONFIG_TEST $CONFIG_FILE $OPTIONS
RESULT=\$?
fi

if [ \$RESULT -eq 0 ]; then
$CONFIG_CMD $CONFIG_OPTIONS $CONFIG_FILE $OPTIONS
else
echo "The file-syntax in '\$SITE_INFO_FILE' is invalid"
echo "Script can not continue"
exit 1
fi

exit 0
EOF
chmod 750 $RECONFIG_CMD
###################################################################
echo
echo "Copying MPI-services to $(dirname $SITE_INFO_FILE)"
echo

if [ -e /opt/glite/yaim/examples/siteinfo/services ]; then
/bin/cp -pvr /opt/glite/yaim/examples/siteinfo/services $(dirname $SITE_INFO_FILE)
fi
###################################################################
echo
echo "Configuring User and Group files"
echo

/bin/cp -f /opt/glite/yaim/examples/ig-groups.conf /opt/glite/yaim/etc/gilda/
/bin/cp -f /opt/glite/yaim/examples/ig-users.conf /opt/glite/yaim/etc/gilda/

cat /opt/glite/yaim/etc/gilda/gilda_ig-users.conf >> /opt/glite/yaim/etc/gilda/ig-users.conf
cat /opt/glite/yaim/etc/gilda/gilda_ig-groups.conf >> /opt/glite/yaim/etc/gilda/ig-groups.conf
###################################################################
echo ##############################################################
echo
echo "Done configuring the host as a $HOST_TYPE node"
echo
echo "The host is now ready to be configured."
echo "To configure the host execute:"
echo ". $RECONFIG_CMD"
[ $EXIT_ON_ERROR -eq 0 ] || exit 0

-- VanEckA - 19 Aug 2008

Topic revision: r14 - 03 Jun 2009 - 08:55:19 - VanEckA
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback