So after figuring out a bunch of things, it seems that even with using an oauth grant from actual metawiki, we (naturally) get capital letters from oauth and the most recent version of jupyterhub (and possibly k8s) is choking on the capitals. They get converted into the hex representations of the letters. That *could* still somehow be affected by running in Minikube, but it seems unlikely.

Overall, this is how Jupyterhub normally works: jupyter--45-56-413-2e0-5f-28bot-29 1/1 Running 0 133m. Capitals get converted to hex. So why is that such a problem on our local env...

So I worked a bit on this but didn't finish it, at the end I've noticed @yuvipanda ran into this a few months ago https://github.com/berkeley-dsep-infra/datahub/pull/2264/files

Using a similar strategy we can freely modify the pods https://github.com/toolforge/paws/commit/746dab22205931aa20b18efc12244d4356a4058c

There are still a few things that are not quite ironed out, including a weird error The 'cookie_secret' trait of a JupyterHub instance must be a bytes object, but a value of 'ca4f7a84fddc5bfe81c74b765d6a8e1763b674b93c5c20a04e8e6f6a3a72f621' <class 'str'> was specified. That perhaps might indicate we need to rethink the way the hub image is built.

@Chicocvenancio thank you for looking into the jupyter tag. Is there any timing on when we might have a patch to try?

@mdipietro The commit I made there should be enough to fix the label issue. Not sure about other issues, such as the hub image build.