README.txt - operations/software/elasticsearch/plugins - Gitiles

 Debian package to bundle elasticsearch plugins needed for search.

 I. PREPARE A NEW VERSION
 ------------------------
 When adding a new plugin or upgrading to a new elastic version the developer
 needs to:
  - update debian/rules to set the new elastic/build version
  - update debian/plugin_urls.lst to add new plugins or change custom versions
    make sure the GPG key ID (separated by comma at the end of a line) matches
    the one used for signing the release, and that this key has been published
  - update debian/changelog to indicate what's new (`dch -i`).

 Then they need to generate the debian/sha256sums file:

 $ ./debian/rules prepare_commit

 This command will execute the bash script debian/checkout.bash. This script
 will:
  - download all plugins listed in debian/plugin_urls.lst
  - verify gpg signature if one is provided
  - check for elasticsearch version
  - assemble the plugin files to debian/blobs
  - generate a new debian/sha256sums

 If debian/blobs looks good the developer can git commit -a && git review
 (NOTE that debian/blobs is never uploaded to gerrit)

 II. REVIEW A PATCH
 -----------------
 The process relies on the fact that at least two engineers will fetch the
 plugins from their source. The debian/sha256sums file will serve as a verification
 to make sure that the build is repeatable and that no binary blob was uploaded
 maliciously.

 To review they must simply run:

 $ git review -d gerrit_id
 $ ./debian/rules verify_commit

 It will perform exactly the same operations as prepare_commit except that the
 debian/sha256sums file is not generated. The sums are validated.
 They can then verify that debian/blobs looks good.

 If everything is OK the reviewer can +2 the patch.

 III. BUILD THE DEBIAN PACKAGE
 ---------------------------
 An ops engineer can then build the package.
 First they need to fetch the blobs on their local machine prior to uploading to
 the active package build server:

 $ ./debian/rules prepare_build

 It'll download all the files and make all the verifications again.
 If everything looks good they can scp the whole dir to the active package build server and then build
 the package with
 $ DIST=bullseye-wikimedia pdebuild

 NOTE: the sha256sums are verified again.

 The package can then be uploaded to the apt repository.

 IV. BLOBS VERIFICATION
 ----------------------

 In the end the blobs will be download 3 times (at least 2 if ops == reviewer):
  1. The gerrit uploader
  2. The reviewer
  3. The ops

 GPG signatures will be checked 3 times (at least 2 if ops == reviewer):
  1. The gerrit uploader
  2. The reviewer
  3. The ops

 The sha256sums file is generated only one time per patch by the gerrit uploader.
 It then verified 3 times:
  1. The reviewer
  2. The ops on the prepare_build command
  3. On the active package build server by dpkg-buildpackage

 V. FILES
 ---------
 debian/rules            Makefile for dh and custom build steps
 debian/sha256sums       List of all sha256 sums that were generated by the last prepare_commit
 debian/plugin_urls.lst  List of plugins to install, the format is as follow:
 debian/changelog        Debian changelog file
 debian/checkout.bash    Bash script to fetch&verify plugins from their source

 VI. FORMATS
 -----------
 plugin_urls.lst is a flat file with one line per plugin formatted as follow:
 URL,GPG_KEY
 e.g.
 https://artifacts.elastic.co/downloads/elasticsearch-plugins/analysis-icu/analysis-icu-$ELASTICSEARCH_VERSION.zip,D27D666CD88E42B4
 Use none when the zip is not signed.
 $ELASTICSEARCH_VERSION will resolve to the version defined in debian/rules,
 it's useful for core plugins.

 VII. DEVELOPER IMAGES
 ---------------------
 When shipping a new version of the plugins you might want to also update the
 images used by developers that work with CirrusSearch and Elasticsearch. The
 project is named cirrussearch-elasticsearch-image
 (https://gitlab.wikimedia.org/repos/search-platform/cirrussearch-elasticsearch-image).
	Debian package to bundle elasticsearch plugins needed for search.

	I. PREPARE A NEW VERSION
	------------------------
	When adding a new plugin or upgrading to a new elastic version the developer
	needs to:
	- update debian/rules to set the new elastic/build version
	- update debian/plugin_urls.lst to add new plugins or change custom versions
	make sure the GPG key ID (separated by comma at the end of a line) matches
	the one used for signing the release, and that this key has been published
	- update debian/changelog to indicate what's new (`dch -i`).

	Then they need to generate the debian/sha256sums file:

	$ ./debian/rules prepare_commit

	This command will execute the bash script debian/checkout.bash. This script
	will:
	- download all plugins listed in debian/plugin_urls.lst
	- verify gpg signature if one is provided
	- check for elasticsearch version
	- assemble the plugin files to debian/blobs
	- generate a new debian/sha256sums

	If debian/blobs looks good the developer can git commit -a && git review
	(NOTE that debian/blobs is never uploaded to gerrit)

	II. REVIEW A PATCH
	-----------------
	The process relies on the fact that at least two engineers will fetch the
	plugins from their source. The debian/sha256sums file will serve as a verification
	to make sure that the build is repeatable and that no binary blob was uploaded
	maliciously.

	To review they must simply run:

	$ git review -d gerrit_id
	$ ./debian/rules verify_commit

	It will perform exactly the same operations as prepare_commit except that the
	debian/sha256sums file is not generated. The sums are validated.
	They can then verify that debian/blobs looks good.

	If everything is OK the reviewer can +2 the patch.

	III. BUILD THE DEBIAN PACKAGE
	---------------------------
	An ops engineer can then build the package.
	First they need to fetch the blobs on their local machine prior to uploading to
	the active package build server:

	$ ./debian/rules prepare_build

	It'll download all the files and make all the verifications again.
	If everything looks good they can scp the whole dir to the active package build server and then build
	the package with
	$ DIST=bullseye-wikimedia pdebuild

	NOTE: the sha256sums are verified again.

	The package can then be uploaded to the apt repository.

	IV. BLOBS VERIFICATION
	----------------------

	In the end the blobs will be download 3 times (at least 2 if ops == reviewer):
	1. The gerrit uploader
	2. The reviewer
	3. The ops

	GPG signatures will be checked 3 times (at least 2 if ops == reviewer):
	1. The gerrit uploader
	2. The reviewer
	3. The ops

	The sha256sums file is generated only one time per patch by the gerrit uploader.
	It then verified 3 times:
	1. The reviewer
	2. The ops on the prepare_build command
	3. On the active package build server by dpkg-buildpackage

	V. FILES
	---------
	debian/rules Makefile for dh and custom build steps
	debian/sha256sums List of all sha256 sums that were generated by the last prepare_commit
	debian/plugin_urls.lst List of plugins to install, the format is as follow:
	debian/changelog Debian changelog file
	debian/checkout.bash Bash script to fetch&verify plugins from their source

	VI. FORMATS
	-----------
	plugin_urls.lst is a flat file with one line per plugin formatted as follow:
	URL,GPG_KEY
	e.g.
	https://artifacts.elastic.co/downloads/elasticsearch-plugins/analysis-icu/analysis-icu-$ELASTICSEARCH_VERSION.zip,D27D666CD88E42B4
	Use none when the zip is not signed.
	$ELASTICSEARCH_VERSION will resolve to the version defined in debian/rules,
	it's useful for core plugins.

	VII. DEVELOPER IMAGES
	---------------------
	When shipping a new version of the plugins you might want to also update the
	images used by developers that work with CirrusSearch and Elasticsearch. The
	project is named cirrussearch-elasticsearch-image
	(https://gitlab.wikimedia.org/repos/search-platform/cirrussearch-elasticsearch-image).