limit the impact of heavy/large graphite queries
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fgiunchedi
	Oct 27 2015, 3:35 PM

Description

as discovered in a recent graphite outage, heavy/large queries can occupy all uwsgi workers, resulting in 502s. we should seek how to limit the impact of such queries, ideally with timeouts at the graphite-web level

Details

	Subject	Repo	Branch	Lines +/-
	graphite: uwsgi workers: set timeouts + max RSS	operations/puppet	production	+32 -13

Customize query in gerrit

Related Objects

Mentioned In: T155876: Increased load on graphite1003, carbon-cache not autorestarting when killed by OOM
T155872: graphite1003 short of available RAM
Mentioned Here: T155872: graphite1003 short of available RAM

Event Timeline

fgiunchedi created this task.Oct 27 2015, 3:35 PM

fgiunchedi raised the priority of this task from to Medium.

fgiunchedi updated the task description. (Show Details)

fgiunchedi added projects: SRE, observability.

fgiunchedi subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 27 2015, 3:35 PM

See also T155872: graphite1003 short of available RAM for a case where heavy queries were not impacting uwsgi but carbon-cache instead using a lot of memory.

Updated wikitech Graphite troubleshooting on how to identify such queries.

fgiunchedi merged tasks: T155872: graphite1003 short of available RAM, T155876: Increased load on graphite1003, carbon-cache not autorestarting when killed by OOM.Jan 27 2017, 10:33 AM

fgiunchedi added subscribers: Volans, • ema, Stashbot, gerritbot.

fgiunchedi mentioned this in T155876: Increased load on graphite1003, carbon-cache not autorestarting when killed by OOM.Jan 27 2017, 10:36 AM

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 8:23 PM

Change 494620 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] add uwsgi worker timeouts + max RSS for graphite

https://gerrit.wikimedia.org/r/494620

gerritbot added a project: Patch-For-Review.Mar 5 2019, 11:20 PM

Change 494620 merged by CDanis:
[operations/puppet@production] graphite: uwsgi workers: set timeouts + max RSS