Help:Troubleshooting Toolforge
This page contains information about common issues you may encounter while working with Toolforge.
Permission denied when running a script
Make sure the script is running from your Tool Account, rather than your user account.
User preferences can't be accessed in the replicas
User preferences are considered private information and can't be accessed in the replicas.
Error message: must be installed for pthread_cancel to work?
If you encounter this while trying to run a job on the grid engine, the job needs more memory.
See Allocating additional memory for more information.
Error message: No such tool
You receive the following error message after creating a new Tool Account:
$ become <TOOL NAME> become: no such tool '<TOOL NAME>'
Error message: You are not a member of the group tools
You receive the following error after joining a Tool Account:
$ become <TOOL NAME> You are not a member of the group tools.<TOOL NAME>.Any existing member of the tool's group can add you to that.
If you are already logged in via SSH when you created the a new tool, log out, and log in again to activate your new permissions.
Error message: Trying to ssh into getting a " Permission denied (publickey,hostbased)".
If you are running ssh from the command line, make sure that the command is ssh​, where "username" is your username (and not the name of the tool you're attempting to log into).
Error message: Warning: the ECDSA host key for '' differs from the key for the IP address ''
This means the key has been updated. Try refreshing it with :ssh-keygen -R <ip>
public_html does not work in home directory
Users cannot have a public_html folder to themselves. The only web-accessible directories are under /data/project/<toolname>/public_html/​.
To have a URL such as http://<username>​, you must create a tool called <username>, which will create a folder called /data/project/<username>/public_html/.
Nobody--except you, the user--will *ever* be given access to your home or its files. Allowing public services to run from a home directory means that their management could not be shared or taken over if they end up abandoned, defeating the purpose.
Can't detect Cloud VPS or the project
Every Cloud VPS instance contains the file: /etc/wmflabs-project. The contents of the of the file will indicate whether the project is "tools" for the Toolforge, or "toolsbeta" for the experimental Toolforge.
Tool requires a package that is not currently installed in Toolforge
You might not be the only one missing that package. Please submit a ticket in Phabricator and ask the admins to install it project-wide. If they have reasons not to do so you can always install software locally / just for yourself.
Note that for python packages, we recommend that you use a virtualenv local to your tool.
Only request a global package if it is difficult to install via pip (numpy, scipy, etc). For example to install the python package internetarchive run
virtualenv ~/env source $HOME​/env/bin/activate echo "source $HOME​/env/bin/activate" >> .bashrc pip install internetarchive
See also Help:Toolforge/Python application stub on how to get a sample flask python application installed onto the tools-project.
Slow connection
Your connection may be slower depending on your location.
If you are connecting from Europe, try using Mosh. To connect, use mosh -a (-a to force predictive echo) (instead of ssh
No "instances"
Developers working in Toolforge do not have to create or set up virtual machines (i.e., Cloud VPS "instances"), because the Toolforge project admins create and manage them. The term may appear in global documentation on Wikitech but is not applicable to Toolforge.
Bastion is slow
When your connection is fine but the Toolforge bastion (, tools-sgebastion-07) is slow or unresponsive, it is likely due to NFS lag. All users of the Toolforge bastion share the same limited connection to NFS. Tasks such as downloading large files or doing processing directly on the bastion host create a high volume of traffic that saturates the connection to NFS. Interactive tools like your shell, cd, and ls must then wait longer to send and receive their own traffic.
To prevent NFS lag, do not run any tasks that require heavy disk IO directly from the bastion. Instead, run them from the Grid or inside a webservice shell. Interactive jobs that require disk IO can also be carried out on the development bastion (, but they may still impact others there.
Toolforge administrators will kill processes running from the bastion that are causing excessive NFS lag.
Report issues with Toolforge
Phabricator is used for issue and work tracking across Wikimedia technical projects.
All issues in Phabricator are available for the public to view. However, to use Phabricator to report issues, you must create a developer account.
When reporting issues, remember
Reporting an issue with a specific Tool
1) Find maintainers to notify and a Phabricator project to associate.
As an example, you can find details for the admin Tool (which powers the default page for at the toolsadmin page for it. Any tool should provide information at a similar URL: ''<tool name>''
2) Create a task in Phabricator.
There are general Wikimedia documents on the needed instructions and recommended content for reporting issues.
This can use this template to populate a task form.
Communication and support
We communicate and provide support through several primary channels. Please reach out with questions and to join the conversation.
Communicate with us
ConnectBest for
Phabricator Workboard#Cloud-ServicesTask tracking and bug reporting
IRC Channel#wikimedia-cloud​connect
Telegram bridge
mattermost bridge
General discussion and support
Mailing Listcloud@Information about ongoing initiatives, general discussion and support
Announcement emailscloud-announce@Information about critical changes (all messages mirrored to cloud@)
News wiki pageNewsInformation about major near-term plans
Cloud Services BlogClouds & UnicornsLearning more details about some of our work
Wikimedia Technical Blogtechblog.wikimedia.orgNews and stories from the Wikimedia technical movement
See Also
Last edited on 27 August 2021, at 22:00
Content is available under CC BY-SA 3.0 unless otherwise noted.
Privacy policy
Terms of Use
HomeRandomLog inSettingsDonateAbout WikitechDisclaimers