Page MenuHomePhabricator

Puppet compiler running out of space on compiler1002.puppet-diffs.eqiad.wmflabs
Closed, ResolvedPublic

Description

I'm getting puppet compiler running out of space error:
https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler-test/671/console

[ 2021-02-02T10:49:25 ] CRITICAL: Build run failed: [Errno 28] No space left on device: '/srv/jenkins-workspace/puppet-compiler/671'

And here as well https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler-test/670/console

compiler1002.puppet-diffs.eqiad.wmflabs is full

Event Timeline

hashar renamed this task from Puppet compiler running out of space to Puppet compiler running out of space on compiler1002.puppet-diffs.eqiad.wmflabs.Feb 2 2021, 10:57 AM
hashar updated the task description. (Show Details)

The instance compiler1002.puppet-diffs.eqiad.wmflabs is full :/

df -h -t ext4
Filesystem                          Size  Used Avail Use% Mounted on
/dev/vda2                            19G  4.8G   13G  27% /
/dev/mapper/vd-second--local--disk   60G   57G  184K 100% /srv

/srv/postgres is 5482 MB

The compiler output is written to /srv/jenkins-workspace/puppet-compiler/output which is garbage collected after 31 days via a cronjob.

However there are a bunch of obsolete builds under /srv/jenkins-workspace/puppet-compiler/

hashar claimed this task.

I have deleted bunch of old builds under /srv/jenkins-workspace/puppet-compiler