Maniphest T211904

Set up data storage to collect loosely structured data from CI
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	• mmodell
	Dec 13 2018, 5:35 PM

Tags

Referenced Files

None

Subscribers

Description

The Release-Engineering-Team have decided to start collecting as much data as possible out of CI, Code Health, Incidents, etc. To that end, we need to provision some storage for this data.

We don't expect the data to be very large because we will filter and summarize up-front rather than storing bulk logs or artifacts. I would expect the data to be quite a bit less than 10 gigabytes.

After a couple of discussions within the team, we are leaning towards using ElasticSearch for the back-end since it has powerful query capabilities that will facilitate easy retrieval in a usable format for reporting and analysis.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T216085 Release Engineering Data Collection and Retention (aka Data³)
		Declined		None	T53447 Store CI builds output outside Jenkins (e.g. static storage)
		Declined		None	T114998 Provide Swift object store(s) for the labs projects
		Declined		None	T78705 Send Jenkins build log and results to ElasticSearch
		Declined		None	T211904 Set up data storage to collect loosely structured data from CI

Event Timeline

• mmodell created this task.Dec 13 2018, 5:35 PM

Restricted Application added a project: Discovery-Search. · View Herald TranscriptDec 13 2018, 5:35 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

@EBernhardson: Would it be reasonable to store this data on the search cluster? We thought to ask for your blessing to do so, in order to avoid setting up a separate elasticsearch cluster for this tiny use-case. So I guess the question is whether you think it's reasonable and won't be a burden on the Discovery-Search team.

For the CI logs and tests result, we have the old T78705 which has a bunch of context. It is a subset of this T211904 task.

@mmodell EBernhardson is out through the end of the year and really needs to weigh in on this

greg edited projects, added Release-Engineering-Team (Kanban); removed Release-Engineering-Team.Dec 19 2018, 2:08 AM

• LarsWirzenius subscribed.Dec 19 2018, 3:09 PM

@EBjune: Thanks for the heads-up. I definitely want EBernhardson to weigh in. This is just exploratory work and it can wait.

hashar added a parent task: T53447: Store CI builds output outside Jenkins (e.g. static storage).Jan 1 2019, 5:48 PM

• EBjune triaged this task as Medium priority.Jan 3 2019, 6:11 PM

EBernhardson moved this task from needs triage to watching / waiting on the Discovery-Search board.Jan 10 2019, 5:49 PM

Jrbranaa added a parent task: T216085: Release Engineering Data Collection and Retention (aka Data³).Feb 13 2019, 11:07 PM

greg added a project: Release-Engineering-Team-TODO.Jul 1 2019, 9:29 PM

greg moved this task from Should be empty (use Release-Engineering-Team) to Soon-ish on the Release-Engineering-Team-TODO board.Jul 1 2019, 9:30 PM

greg removed a project: Release-Engineering-Team (Kanban).Jul 1 2019, 9:31 PM

greg moved this task from Soon-ish to Next on the Release-Engineering-Team-TODO board.Jul 6 2019, 4:59 AM

greg added a project: Release-Engineering-Team (CI & Testing services).Aug 1 2019, 11:12 PM

• LarsWirzenius unsubscribed.Feb 20 2020, 3:30 PM

• mmodell claimed this task.Sep 16 2020, 5:47 PM

• mmodell edited projects, added Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)); removed Release-Engineering-Team-TODO, Discovery-Search.

• mmodell moved this task from INBOX to New Work on the Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)) board.

• mmodell edited projects, added Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)); removed Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)).Oct 14 2020, 5:21 PM

• mmodell moved this task from INBOX to New Work on the Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)) board.Oct 14 2020, 5:34 PM

• mmodell raised the priority of this task from Medium to High.Nov 4 2020, 6:27 PM

thcipriani removed a project: Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)).Apr 20 2021, 12:21 AM

thcipriani added a project: Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)).Apr 20 2021, 12:34 AM

thcipriani removed a project: Release-Engineering-Team (CI & Testing services).Apr 20 2021, 1:09 AM

thcipriani edited projects, added Release-Engineering-Team (Doing); removed Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)).Apr 20 2021, 4:26 AM

thcipriani removed • mmodell as the assignee of this task.Jul 27 2021, 5:47 PM

thcipriani lowered the priority of this task from High to Low.

thcipriani edited projects, added Release-Engineering-Team (Next); removed Release-Engineering-Team (Doing).

• mmodell moved this task from Next to Priority Backlog 📥 on the Release-Engineering-Team board.Oct 6 2021, 4:32 PM

• mmodell edited projects, added Release-Engineering-Team (Priority Backlog 📥); removed Release-Engineering-Team (Next).

Closing this one again since we have no plan to work on it.

Aklapper removed a subscriber: Release-Engineering-Team.May 16 2023, 1:13 PM