Oh, another piece of info: WMF traffic frontends set a timeout for all connections of 15 minutes. This causes connected SSE clients to reconnect every 15 minutes. The disconnect should decrement the client IP. But, if there are enough connections from the same IP, I think the reconnect would be more likely to end up at a worker that is already at the limit for that IP.

Or, could we just avoid rate limiting Cloud VPS / Toolforge IPs in EventStreams code? Or at least increase the limit by a lot for those IPs?

If you really wanted to know for sure, I suppose you'd have to just ask everyone. There are 155 people listed (hmm, more than I thought before I checked).

In T363290#9742949, @matmarex wrote:

I feel like people mostly use it for notifications, and don't intend to actually review all changes they signed up for.

In T363290#9739118, @hashar wrote:

On the counterpart, if the listed reviewers are added to the CC field, they would not be put in the attention set which kind of defeat the purpose of the Gerrit reviewer bot.

Mentioned in SAL (#wikimedia-cloud) [2024-04-24T18:37:10Z] <wmbot~anticomposite@tools-bastion-13> Deploy 2f528f2..c8354da (T329327)

In T329327#9739317, @aborrero wrote:

I confirm this is not the case. The destination address of stream.wikimedia.org, as seen from within Toolforge, is exempt from the egress NAT on Cloud VPS. This means EventStreams gets to see the source IP address of the Toolforge kubernetes worker node.

However, there could be multiple pods within the same Toolforge k8s worker node connecting to the same endpoint, thus consuming the available slots.

This already happened, see T363296: toolforge: explore options to introduce egress network quotas.

This is probably not helpful, but EventStreams naive IP based local rate limiting is pretty dumb. If there was a smarter more global solution in WMF prod (maybe there is these days), we'd much prefer to use that.

In T329327#9738585, @bd808 wrote:

Based on this explanation of the rate limiting implementation I am very much wondering if EventStreams is seeing all traffic from Cloud VPS as coming from a single IP, specifically 185.15.56.1 (nat.cloudgw.eqiad1.wikimediacloud.org). If so, EventStreams would be mostly unusable by Toolforge tools and other Cloud VPS projects with potentially hundreds of tools fighting over 16 slots.

On the counterpart, if the listed reviewers are added to the CC field, they would not be put in the attention set which kind of defeat the purpose of the Gerrit reviewer bot.

I've created a set of instructions for how to use Toolhub to find and review tool documentation, and then use the Tool Docs Guide to review and improve docs as part of this Hackathon project: https://www.mediawiki.org/wiki/Doc_Your_Tool:_Creating_user-friendly_documentation/Instructions

solved by restarting the server. Next time, please open a NEW ticket for NEW problems. you can always link back to a ticket like this simply by mentioning it in the new ticket.

https://wikimap.toolforge.org, thank you!

I did update it last week, ill take a look later tonight.

Did Nominatim loose it's CORS header? Can no longer request from other toolforge tool.

Updates on the progress made during Wishathon (March 2024) by @Soda and @Gopavasanth :

@Liz: Quarry and bots are not AFD?

@Soda @Gopavasanth Since both of you worked on this tool during the Wishathon, could you please share more updates about the progress on this task and any relevant links? Thank you!

Status update:

• Finished post-review changes and removed draft status from the new Tool Docs guide; it's officially published!
• Added links and updated related content on
  • https://www.mediawiki.org/wiki/Doc_Your_Tool:_Creating_user-friendly_documentation
  • https://www.mediawiki.org/wiki/Documentation/Toolkit
  • https://wikitech.wikimedia.org/wiki/Help:Toolforge/Developing_successful_tools
• Posted on the Talk pages for each of the tools I used as examples, to let them know that they were featured and also to mention the Hackathon project ([[ https://meta.wikimedia.org/wiki/Talk:Citation_Hunt#Your_tool_docs_are_featured_as_an_example_in_the_new_Tool_Docs_Guide%21 | example ]]of what I posted on each Talk page).

Hi, is there any chance of MATCH&SPLIT being reintegrated into Wikisource soon? Many contributors (including myself) use it extensively to ease the process of matching pre-formated text to scans and we're desperately hoping to be able to be able to use it again! Many thanks in advance!

This tool would not scale to today's Wikidata size

Created a quick first draft of instructions for how to claim phab tasks and work on docs during the hackathon: https://www.mediawiki.org/wiki/Doc_Your_Tool:_Creating_user-friendly_documentation/Instructions

Draft of Tool Docs guide is now ready finalized at https://www.mediawiki.org/wiki/Documentation/Tool_docs. I welcome any edits and feedback on it (feel free to use the Talk page or comment on this task).

In case anyone is using this prototype tool, a cursory deprecation notice should be given.

This is stalled until Itamar comes back – the only maintainers are Itamar and Michael (who no longer works at WMDE, sadly), so the rest of the wmde-wd-team can’t do anything.

@bd808 I find that there are some repeated files in the folder. That was my attempt to deduplicate them in order to make the upload process more efficient. I will just let the upload script work smoothly. Sorry for any troubles caused.

I found and killed a sha1sum process running on tools-sgebastion-10.tools.eqiad1.wikimedia.cloud a few minutes ago that seemed to be attempting to compute the sha1 of all 533G of PDFs here in one go. That is not something that can reasonably happen on a shared bastion server. Please rethink your process.

Metadata of these files are not well-structured. I was managing to tidy them up so that template fields can be filled properly and appropriate categories can be created, which makes me frustrated. This is the reason why I leave them there so long.

@Hoi The files are still there, I'm guessing you did not have the time? Are you encountering any errors?

I wasn't able to get one line per wiki, but https://wsstats.toolforge.org/stats/all/alltime exists now which gives a overview of all wikisources.

In T319965#9633459, @Tpt wrote:

@Soda Amazing! Thank you! A user request from French Wikisource: Would it be possible to add to the website plots for all the wikisources (a line per wiki)? And the data in a table format instead of charts (I guess just allowing to download a csv is fine)?

@Soda Amazing! Thank you! A user request from French Wikisource: Would it be possible to add to the website plots for all the wikisources (a line per wiki)? And the data in a table format instead of charts (I guess just allowing to download a csv is fine)?