grosley

**grosley**
	grosley
Location:	pmtpa
Status
Overall:	This device has been decommissioned.
Icinga?:	host status services status
Hardware
Software

gosley was the failover host for aluminium. See Grosley-Aluminium-Migration for notes from switchover in November 2011.

Services

Various fundraising services, including:

civicrm.wikimedia.org (Apache, PHP & CiviCRM)
fundraising.wikimedia.org
Non-web accessible fundraising scripts can be found in /opt/fundraising_misc.

Hudson

Grosley has Hudson CI installed to facilitate the periodic running of maintenance scripts. It's flexibility, particularly for logging/capturing output, is why we are preferring it over cron. Hudson CI ver. 1.368 running. It is accessible at: http://grosley.wikimedia.org:8080/

Potential issues

Hitting ext3 subdir limit in builds directories

Every time a Hudson job runs, a new directory gets created on Hudson where information about that particular build gets stored. Build dirs get created in:

/var/lib/hudson/jobs/<job name>/builds

Particularly for frequently running jobs (eg queue consumption which runs every 5 minutes), we run the risk of hitting the ext3 limit on subirectories (32000). When this limit gets reached, build jobs will fail silently until the issue gets resolved

Prevention

We have prepared a Nagios script which we are hoping to have monitor the job directories. Warning thresholds should be set at 75% of subdirectory limit and critical at 85%.

Resolution

As the limit gets approached, build directories should be archived to storage3 and then cleaned out:

# tar czf hudson_builds_<build name>_<starting date>-<ending date>.tar.gz /var/lib/hudson/jobs/<build name>/builds
# scp hudson_builds_<build name>_<starting date>-<ending date>.tar.gz user@storage3.pmtpa.wmnet:/archive/fundraising/hudson_builds/
# rm -rf /var/lib/hudson/jobs/<build name>/builds/*

Mail setup

grosley runs a simple Exim setup, which can send out e-mail directly to the outside world using its dnslookup router and remote_smtp transport. It's also configured to accept mail for *@donate.wikimedia.org, and deliver this mail into a Maildir box in /var/mail/civimail, using the civimail router and civimail_delivery transport. This is basically a simpler version of the configuration described in the Mail article.

Dovecot is installed for IMAP access by CiviMail. Its configuration is almost default, but it's configured to listen using non-SSL IMAP on the loopback interface only, and to retrieve mail using Maildir from /var/mail/user. Authentication is done using PAM and system users, so a system user civimail has been added.

Configuration changes

localhost IPs (v4 and v6) added to relay_from_hosts in exim4.conf --David Strauss 02:37, 7 May 2009 (UTC)[reply]
Truncated /var/log/exim4/paniclog, which had an old configuration syntax error notice in it --David Strauss 02:38, 7 May 2009 (UTC)[reply]