Tool:Event Metrics

Toolforge tools
Event Metrics
Website	https://eventmetrics.wmflabs.org/
Keywords	statistics, analytics, events
Author(s)	MusikAnimal, Samwilson, MaxSem, Mooeypoo
Maintainer(s)	MusikAnimal, Samwilson, HMonroy, Dmaza (View all)
Source code	GitHub
License	GNU General Public License 3.0 or later
Issues	Open tasks · Report a bug

This page documents the production and staging instances of Event Metrics. Some things in this guide still use the old name "Grant Metrics", such as references to the Toolforge account.

Note to maintainers: we don't backup any server configuration, so please document everything here.

Contact

The maintainers can be emailed at tools.grantmetricstools.wmflabs.org (note that this means that the maintainers of the VPS account and the Toolforge account need to be in sync).

Production

Production Event Metrics is hosted on on a Wikimedia VPS instance. To log into the server, make sure you've been added as a maintainer to the eventmetrics project. Then set up SSH and connect via ssh eventmetrics-prod02.eventmetrics.eqiad1.wikimedia.cloud and go to the /var/www directory. Not quite everything in this directory is in the Git repository.

Logs are written to /var/www/var/logs/prod.log, but only during a request where an error or high-priority log entry was made. This is why you'll see debug-level log entries in prod.log. You might also need to check /var/log/apache2/error.log for Apache-level errors.

OAuth consumer: Event Metrics 1.0

Database is s53550__grantmetrics on tools.labsdb, we're connecting as user s53550.

Web server configuration is all in /etc/apache2/sites-available/eventmetrics.conf.

There's a /var/www/deploy.sh script that runs every 10 minutes (from www-data's crontab) and if required updates to the latest release. The output of this is mailed to the maintainers.

Building a new instance

First create a new instance running on debian-jessie. Any production node should be at least a m1.medium with 4 GB of RAM, but for the main app server a m1.large is probably best.

Once the instance has been spawned, SSH in and follow these steps:

Install PHP and Apache, along with some dependencies.

sudo apt-get update
sudo apt-get install -y apache2
sudo apt-get install -y php php-cli php-common php-curl php-json php-apcu
sudo apt-get install -y php-mysql php-intl php-xml php-mbstring libapache2-mod-php
sudo apt-get install -y zip unzip php-zip php-bz2
sudo a2dismod mpm_event && sudo a2enmod mpm_prefork && sudo a2enmod php

Install Node and npm.

sudo apt-get install -y nodejs
sudo apt-get install -y npm

Symlink nodejs to node, so the uglify-js plugin runs the right command.
```
sudo ln -s /usr/bin/nodejs /usr/bin/node
```
We need to upgrade npm once more
```
sudo npm install -g npm@next
```
Install composer by following these instructions, but make sure to install to the /usr/local/bin directory and with the filename composer, e.g.:
```
sudo php composer-setup.php --install-dir=/usr/local/bin --filename=composer
```

Clone the repository, first removing the html directory created by Apache.

cd /var/www && sudo rm -rf html
sudo git clone https://github.com/wikimedia/eventmetrics.git .

Become the root user with sudo su root
Run sudo composer install and fill in the necessary parameters. For most options you can use the default. Moving forward we will not run composer as root or with sudo, but via the webserver (see step #13 below).

Create the deploy script at /var/www/deploy.sh with the following:

#!/bin/bash

cd /var/www

git fetch --quiet origin 2>&1

## Find the highest and current tags
HIGHEST_TAG=$(git tag | sort --version-sort | tail --lines 1)
CURRENT_TAG=$(git describe --tags)

## Exit and say nothing if we're already at the highest tag.
if [[ $CURRENT_TAG == $HIGHEST_TAG ]]; then
    # The following line can be temporarily uncommented as-needed
    # to force www-data to clear the production cache:
    # ./bin/console cache:clear --env prod
    exit 0
fi

## If there's an update, pull and install it.
git checkout $HIGHEST_TAG
export SYMFONY_ENV=prod
/usr/local/bin/composer install --no-dev --optimize-autoloader
./bin/console cache:clear --env prod
./bin/console doctrine:migrations:migrate --env prod --no-interaction

Make sure the scripts are executable, and that all the files in the repo are owned by www-data.
```
sudo chmod 744 deploy.sh
sudo chown -R www-data:www-data .
```

Create the web server configuration file at /etc/apache2/sites-available/eventmetrics.conf with the following:

<VirtualHost *:80>
        # Wikimetrics was sunset and should redirect to Event Metrics.
        ServerName metrics.wmflabs.org
        RedirectMatch 301 (.*) https://eventmetrics.wmflabs.org
</VirtualHost>
<VirtualHost *:80>
        DocumentRoot /var/www/public
        ServerName eventmetrics.wmflabs.org

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        <Directory /var/www/public/>
             Options Indexes FollowSymLinks
             AllowOverride All
             Require all granted
        </Directory>

        # Deny access to known spiders.
        SetEnvIfNoCase User-Agent "(uCrawler|Baiduspider|CCBot|scrapy\.org|kinshoobot|YisouSpider|Sogou web spider|yandex\.com/bots|twitterbot|TweetmemeBot|SeznamBot|datasift\.com/bot|Googlebot|Yahoo! Slurp|Python-urllib|BehloolBot|MJ12bot|SemrushBot)" bad_bot

        <Directory /var/www/>
                Options Indexes FollowSymLinks
                AllowOverride None
                Require all granted
                Deny from env=bad_bot
        </Directory>

        ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
        <Directory /usr/lib/cgi-bin/>
                Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
                Require all granted
        </Directory>

        SetEnvIf Request_URI "^/robots\.txt$" dontlog
        
        RewriteEngine On
        RewriteCond %{HTTP:X-Forwarded-Proto} !https
        RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R=301,L]
</VirtualHost>

Enable the mod-rewrite Apache module, and enable the web server configuration.

sudo a2enmod rewrite
sudo a2ensite eventmetrics
sudo service apache2 reload

Setup the crontab to run the deploy script every 10 minutes, and spawn unstarted jobs every minute (this will respect database quota). We'll do this under the www-data user.
```
sudo crontab -e -u www-data
```
Then add this to the bottom of the crontab:
```
MAILTO=tools.grantmetrics@tools.wmflabs.org
*/10 * * * * /var/www/deploy.sh
* * * * * php /var/www/bin/console app:spawn-jobs >/dev/null
```
Wait for the email indicating composer ran successfully. If all goes well, you need only to gracefully (re)start apache, which is how you should always restart apache on production:
```
sudo apachectl -k graceful
```

Staging

SSH to eventmetrics-dev01.eventmetrics.eqiad1.wikimedia.cloud (see notes above in #Production about getting access).

Database is s53713__grantmetrics on tools.labsdb, we're connecting as user s53713.

OAuth consumer: Grant Metrics - staging (note that it's redirecting through grantmetrics-test.toolforge.org)

The code on the staging server is kept up to date with the master branch by running the following cronjob under the www-data user:

*/6 * * * * /var/www/html/vendor/wikimedia/toolforge-bundle/bin/deploy.sh dev /var/www/html/

Setting up the staging server is the same as production, except you would use a smaller box (m1.small), and use the above deploy script instead of the one that goes off of tags. You also need to update the ServerName in eventmetrics.conf accordingly.

Testing other branches on staging

In general staging should always be up-to-date with master. If you do need to test something in this environment, you can checkout another branch, but you will need to also grant permissions back to www-data:

sudo git fetch origin
sudo git checkout -t origin/my-branch
sudo ./bin/console cache:clear --env prod
sudo chown -R www-data:www-data .

You can also run sudo composer install if needed, but ideally this wouldn't include database or major config changes that will need to be undone when going back to master.