Page MenuHomePhabricator
Create Task
Maniphest T274306

docker-pkg: "certificate verify failed: unable to get local issuer certificate" for docker-registry.discovery.wmnet when publishing dev-images from contint2001
Closed, ResolvedPublic
Actions

Description

Tried to publish some dev-images changes for T262976 with the usual tox -e fabric -- deploy_docker. Got:

[contint.wikimedia.org] run: /srv/deployment/docker-pkg/venv/bin/docker-pkg --info -c /etc/docker-pkg/dev-images.yaml build /srv/dev-images/dockerfiles | tee dev-images-build.log
[contint.wikimedia.org] out: 2021-02-09 21:08:18,966 [docker-pkg-build] INFO - Processing the dockerfile template in /srv/dev-images/dockerfiles/stretch-php74-jobrunner (builder.py:294)
[contint.wikimedia.org] out: 2021-02-09 21:08:18,966 [docker-pkg-build] INFO - Processing the dockerfile template in /srv/dev-images/dockerfiles/stretch (builder.py:294)
[contint.wikimedia.org] out: 2021-02-09 21:08:18,969 [docker-pkg-build] INFO - Processing the dockerfile template in /srv/dev-images/dockerfiles/stretch-php72-webserver (builder.py:294)
[contint.wikimedia.org] out: 2021-02-09 21:08:18,971 [docker-pkg-build] INFO - Processing the dockerfile template in /srv/dev-images/dockerfiles/buster (builder.py:294)
[contint.wikimedia.org] out: 2021-02-09 21:08:18,972 [docker-pkg-build] INFO - Processing the dockerfile template in /srv/dev-images/dockerfiles/stretch-php72-fpm-apache2-xdebug (builder.py:294)
[contint.wikimedia.org] out: 2021-02-09 21:08:18,973 [docker-pkg-build] INFO - Processing the dockerfile template in /srv/dev-images/dockerfiles/stretch-php72-fpm-apache2 (builder.py:294)
[contint.wikimedia.org] out: 2021-02-09 21:08:18,975 [docker-pkg-build] INFO - Processing the dockerfile template in /srv/dev-images/dockerfiles/elasticsearch (builder.py:294)
[contint.wikimedia.org] out: 2021-02-09 21:08:18,977 [docker-pkg-build] INFO - Processing the dockerfile template in /srv/dev-images/dockerfiles/stretch-php74-fpm (builder.py:294)
[contint.wikimedia.org] out: 2021-02-09 21:08:19,060 [docker-pkg-build] ERROR - Could not load image in /srv/dev-images/dockerfiles/elasticsearch: HTTPSConnectionPool(host='docker-registry.discovery.wmnet', port=443): Max retries exceeded with url: /v2/dev/stretch-elasticsearch/manifests/0.0.1 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)'))) (builder.py:298)
[contint.wikimedia.org] out: Traceback (most recent call last):
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 594, in urlopen
[contint.wikimedia.org] out:     self._prepare_proxy(conn)
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 805, in _prepare_proxy
[contint.wikimedia.org] out:     conn.connect()
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/urllib3/connection.py", line 344, in connect
[contint.wikimedia.org] out:     ssl_context=context)
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 344, in ssl_wrap_socket
[contint.wikimedia.org] out:     return context.wrap_socket(sock, server_hostname=server_hostname)
[contint.wikimedia.org] out:   File "/usr/lib/python3.7/ssl.py", line 412, in wrap_socket
[contint.wikimedia.org] out:     session=session
[contint.wikimedia.org] out:   File "/usr/lib/python3.7/ssl.py", line 853, in _create
[contint.wikimedia.org] out:     self.do_handshake()
[contint.wikimedia.org] out:   File "/usr/lib/python3.7/ssl.py", line 1117, in do_handshake
[contint.wikimedia.org] out:     self._sslobj.do_handshake()
[contint.wikimedia.org] out: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)
[contint.wikimedia.org] out: 
[contint.wikimedia.org] out: During handling of the above exception, another exception occurred:
[contint.wikimedia.org] out: 
[contint.wikimedia.org] out: Traceback (most recent call last):
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
[contint.wikimedia.org] out:     timeout=timeout
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
[contint.wikimedia.org] out:     _stacktrace=sys.exc_info()[2])
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 398, in increment
[contint.wikimedia.org] out:     raise MaxRetryError(_pool, url, error or ResponseError(cause))
[contint.wikimedia.org] out: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='docker-registry.discovery.wmnet', port=443): Max retries exceeded with url: /v2/dev/stretch-elasticsearch/manifests/0.0.1 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)')))
[contint.wikimedia.org] out: 
[contint.wikimedia.org] out: During handling of the above exception, another exception occurred:
[contint.wikimedia.org] out: 
[contint.wikimedia.org] out: Traceback (most recent call last):
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/docker_pkg/builder.py", line 296, in _process_dockerfile_template
[contint.wikimedia.org] out:     return ImageFSM(root, self.client, self.config, self.nocache, self.pull)
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/docker_pkg/builder.py", line 60, in __init__
[contint.wikimedia.org] out:     if self._is_published():
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/docker_pkg/builder.py", line 121, in _is_published
[contint.wikimedia.org] out:     resp = requests.get(manifest_url, proxies=proxies)
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/requests/api.py", line 75, in get
[contint.wikimedia.org] out:     return request('get', url, params=params, **kwargs)
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/requests/api.py", line 60, in request
[contint.wikimedia.org] out:     return session.request(method=method, url=url, **kwargs)
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
[contint.wikimedia.org] out:     resp = self.send(prep, **send_kwargs)
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
[contint.wikimedia.org] out:     r = adapter.send(request, **kwargs)
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/requests/adapters.py", line 514, in send
[contint.wikimedia.org] out:     raise SSLError(e, request=request)
[contint.wikimedia.org] out: requests.exceptions.SSLError: HTTPSConnectionPool(host='docker-registry.discovery.wmnet', port=443): Max retries exceeded with url: /v2/dev/stretch-elasticsearch/manifests/0.0.1 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)')))
[contint.wikimedia.org] out: 2021-02-09 21:08:19,072 [docker-pkg-build] ERROR - Could not load image in /srv/dev-images/dockerfiles/buster: HTTPSConnectionPool(host='docker-registry.discovery.wmnet', port=443): Max retries exceeded with url: /v2/dev/buster/manifests/0.0.1-1 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)'))) (builder.py:298)
[contint.wikimedia.org] out: Traceback (most recent call last):
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 594, in urlopen
[contint.wikimedia.org] out:     self._prepare_proxy(conn)
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 805, in _prepare_proxy
[contint.wikimedia.org] out:     conn.connect()
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/urllib3/connection.py", line 344, in connect
[contint.wikimedia.org] out:     ssl_context=context)
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 344, in ssl_wrap_socket
[contint.wikimedia.org] out:     return context.wrap_socket(sock, server_hostname=server_hostname)
[contint.wikimedia.org] out:   File "/usr/lib/python3.7/ssl.py", line 412, in wrap_socket
[contint.wikimedia.org] out:     session=session
[contint.wikimedia.org] out:   File "/usr/lib/python3.7/ssl.py", line 853, in _create
[contint.wikimedia.org] out:     self.do_handshake()
[contint.wikimedia.org] out:   File "/usr/lib/python3.7/ssl.py", line 1117, in do_handshake
[contint.wikimedia.org] out:     self._sslobj.do_handshake()
[contint.wikimedia.org] out: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)
...

...it goes on in that vein for quite a while. Unclear if this is failing for other uses of docker-pkg than dev-images.

Details

SubjectRepoBranchLines +/-
docker-pkg: add ca_bundle configurationoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects

Event Timeline

brennen created this task.Feb 9 2021, 9:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 9 2021, 9:38 PM
brennen triaged this task as High priority.Feb 9 2021, 9:39 PM
brennen added a project: User-brennen.
brennen moved this task from Backlog to Radar on the User-brennen board.
brennen added a comment.Feb 9 2021, 10:23 PM

Same result with:

brennen@contint2001:/tmp$ /srv/deployment/docker-pkg/venv/bin/docker-pkg --info -c /etc/docker-pkg/dev-images.yaml build /srv/dev-images/dockerfiles | tee -a dev-images-build.log

curl for one of the URLs in question seems fine:

brennen@contint2001:/tmp$ curl -I https://docker-registry.discovery.wmnet/v2/dev/stretch-php74-jobrunner/manifests/2.0.0
HTTP/1.1 200 OK
Server: nginx/1.13.6
Date: Tue, 09 Feb 2021 22:14:35 GMT
Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws
Content-Length: 15653
Connection: keep-alive
Docker-Content-Digest: sha256:77231172b886294941a985fc85a7833cf38cc4702cb33b26348a5cfa7743c6e4
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:77231172b886294941a985fc85a7833cf38cc4702cb33b26348a5cfa7743c6e4"
X-Content-Type-Options: nosniff
Docker-Distribution-Api-Version: registry/2.0
Vary: Accept

Seems docker-pkg specific?

brennen edited projects, added serviceops; removed SRE.Feb 10 2021, 8:52 PM
Stashbot added a comment.Feb 10 2021, 8:59 PM

Mentioned in SAL (#wikimedia-releng) [2021-02-10T20:59:28Z] <brennen> Attempting one more update from dev-images docker-pkg on contint2001 for T274306

brennen added a subscriber: Joe.Feb 10 2021, 9:00 PM

@Joe any thoughts here?

JMeybohm subscribed.Feb 11 2021, 2:57 PM

For production-images we specify the ca_bundle for python requests to use via 'ca_bundle' in the config.yaml, see: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/templates/docker/production-images-config.yaml.erb#8

gerritbot added a comment.Feb 11 2021, 3:04 PM

Change 663588 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/puppet@production] docker-pkg: add ca_bundle configuration

https://gerrit.wikimedia.org/r/663588

gerritbot added a project: Patch-For-Review.Feb 11 2021, 3:04 PM
brennen added a comment.Feb 11 2021, 5:57 PM

For production-images we specify the ca_bundle for python requests to use via 'ca_bundle' in the config.yaml

A ha - thanks! Should have occurred to me to diff the config here with one used elsewhere.

gerritbot added a comment.Feb 12 2021, 9:15 AM

Change 663588 merged by JMeybohm:
[operations/puppet@production] docker-pkg: add ca_bundle configuration

https://gerrit.wikimedia.org/r/663588

JMeybohm added a comment.Feb 12 2021, 9:16 AM

@brennen let me know (maybe by closing this :-)) if the patch worked out for you.

Maintenance_bot removed a project: Patch-For-Review.Feb 12 2021, 10:10 AM
brennen closed this task as Resolved.Feb 12 2021, 5:21 PM
brennen claimed this task.

All good - thanks again!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL