deployment-logstash04, deployment-logstash05 and deployment-logstash06 root partitions are all full.
In particular, deployment-logstash04's root filesystem fills up on a daily basis.
This means logstash on beta is not working.
In the short-term, we should clean them up.
In the long-term, it would be good to make logstash on beta more reliable.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | colewhite | T211984 Logstash in beta fails periodically | |||
| Resolved | colewhite | T233134 logstash-beta.wmflabs.org does not receive any mediawiki events | |||
| Resolved | BUG REPORT | colewhite | T288989 beta logstash servers run out of disk space |
In the long-term, it would be good to make logstash on beta more reliable.
That seems to be tracked on T233134: logstash-beta.wmflabs.org does not receive any mediawiki events ... or maybe T211984: Logstash in beta fails periodically
@thcipriani is there an ETA on this ticket? Would it be possible to prioritize this? It is impacting our testing for SecurePoll which is high priority because of the ongoing board elections.
As a short term stopgap I've cleaned daemon.log manually on deployment-logstash0[456] (same thing done on all)
deployment-logstash04:/var/log# du -shc /var/log/daemon.log* 3.5G /var/log/daemon.log 8.6G /var/log/daemon.log.1 141M /var/log/daemon.log.2.gz 33M /var/log/daemon.log.3.gz 13G total deployment-logstash04:/var/log# > /var/log/daemon.log deployment-logstash04:/var/log# > /var/log/daemon.log.1
Longer term looks like there is excessive gc logging enabled
our bit here should be good to go. please re-add observability if you require additional help
Hello. I need help with the issue of deployment-logstash04's root filesystem filling up on a daily basis. I asked a question on T233134: logstash-beta.wmflabs.org does not receive any mediawiki events as well. This (T288989) ticket's description aligns with my problem more directly than T233134's.
Today @Majavah turned off deployment-logstash04, deployment-logstash05, and deployment-logstash06.