Not an operations thing, just a general site config.

But generally speaking, the only wiki we've ever allowed SMW on was wikitech, and that's for different reasons.

Pretty sure this is automatically declined per T10390

FYI, installing SMW on the cluster really isn't "as easy as that". There's more to it. Plus security review for SMW and dependencies\. And we're running 1.8 for wikitech.

I'm not sure if the security review is more or less of an issue for a private wiki

What do we need to get SMW installed for a small private wiki (chapcom.wikimedia.org)?

Can we get some feedback on this please?
We want to start tracking stuff in the wiki and build some automated tools, which need the structured data in the wiki.

You'll need to convince people to get the rejection on T10390 reverted, at least. Then you'll need it to be properly security reviewed.

I don't understand. We alread use SWM on wikitech wiki, why can't we just do the same for the affcom wiki then? Where is the difference?

labsconsole (now wikitech) was much more separate from production at the time it got SMW installed. chapcomwiki being private seems like another good reason to require it.

Well, in the end we simply need the features and (since we lost quite some time since this discussion has started) we need them soon. We have a huge tech department, server admins and software developers, so we just need to get it done.

You cannot reasonably expect Wikimedia's developers to just drop everything and rush over here to start preparing an extension for deployment because you said so. You're a small (and private!) wiki, and haven't even provided your use case.

It's not even clear this would be allowed.

Well, we are a volunteer body who serve the movement, as volunteers, and provide a service.
The techs are paid staff intended to provide their tech service as well. Of course I don't expect "drop everything", but so far we have made a resolution that we need to introduce certain workflows and the best and easiest way doing so is using SMW, as that already exists, is in use by the WMF and runs on top of our existing working platform. That was on January 29th, so far we have lost one month doing nothing, we don't even have an ETA and no real reason not to just do it. That is not an acceptable service level to me, especially as the whole install takes five minutes - so we wasted more time discussing than to just fix the actual issue.

The AffCom has reviewed its policies and needs to (re-)introduce certain workflows and management systems. We need to collect and maintain information of the orgs we deal with and trigger workflows based on deadlines or certain regular dates, eg. the renewal process for WUGs. Obviously using SMW would be crucial as it would give us the database features (forms, entities) to manage the information and the logic to implement the workflows needed.
I wonder why I need to excuse myself to do the (volunteer) work I have committed to and that has been approved by the committee to be needed to be done, while a simple request for help is being killed here.

Hi Manuel,

Sorry for the delay and confusion here. I'll talk with some people here to determine if and how we can do this for AffCom. I can't make promises (there is a long standing decision not to use SMW on production wikis, and wikitech was only an exception, not a path creator) but I'll let you know asap.

Thanks a lot Greg for your response. That certainly brings some brightness into this discussion :-)

I don't know the rationals behind such a general decision of not using SMW (I run around 60 instances of it), I am just convinced that using SMW more broadly would have huge benefits, especially for committees like the AffCom. Maybe this request can facilitate some general changes (to SMW or wherever the underlying problems are) so we can make that step of being able to allow SMW in the future.

I know that SMW is used in several public and private sites, ranging from austrian authorities to the NASA, so I wonder if they are all stupid not seing the issues you see or if they done something we don't know about, or if we have such a special infrastructure (it is special for sure, but that much of an issue?).

SMW has had an unusually high number of security issues, so I would definitely advise against it for any wiki that contains private information.

What's the affcom wiki dbname? Is it private?

If we need to install smw, the extensions will need a fair amount of review and work.

Regarding the decision not to use SMW on WMF production wikis: See https://phabricator.wikimedia.org/T10390#132245

I can't stop you from bringing up that discussion again, but I personally don't see the decisions being any different this time, especially since Wikidata is addressing much of the use-cases on many wikis.

What this task is requesting is a special exemption from the rule of not installing SMW on WMF prod wikis. If you want to start the "should WMF allow SMW?" conversation, please do so on a different ticket or mailing list thread.

Of course I don't expect "drop everything", but so far we have made a resolution that we need to introduce certain workflows and the best and easiest way doing so is using SMW, as that already exists, is in use by the WMF and runs on top of our existing working platform. That was on January 29th, so far we have lost one month doing nothing, we don't even have an ETA and no real reason not to just do it. That is not an acceptable service level to me, especially as the whole install takes five minutes - so we wasted more time discussing than to just fix the actual issue.

This ticket is less than a month old. Do you know that compares to many other extension deployment requests?
Do not pretend this is a simple, mindless, <5 minute deployment that requires no thought.
The process described at the top of this ticket is not even how deployments work on the WMF production cluster anyway (update.php? composer?)

Additionally, I noticed that the "SMW Bundle" requested appears to include much more than what is already on wikitech - so yes, some sort of security review is definitely required.

In T88748#1078553, @csteipp wrote:

What's the affcom wiki dbname? Is it private?

chapcomwiki, domain is chapcom.wikimedia.org - yes, it's private.

Hmm, having Wikibase could be an alternative. It would need some help from Wikidata pros to achieve what we planned to do with SMW.

What we need:

• forms with specific fields for all types of affiliates, to store / maintain information about all orgs
• queries / list views creating an overview of exising orgs in the wiki - with seperate views based on status and dates - such as affiliated, applications, denied applications, open requests ordered by date (tasklist) etc.
• some templates with a logic (parser functions?) that trigger events based on dates (eg. when WUG renewal date is reached)

I don't know if / how that is doable with Wikibase. Is there some way to create queries that create lists or even other result types?

In T88748#1078636, @80686 wrote:

What we need:

• forms with specific fields for all types of affiliates, to store / maintain information about all orgs
• queries / list views creating an overview of exising orgs in the wiki - with seperate views based on status and dates - such as affiliated, applications, denied applications, open requests ordered by date (tasklist) etc.
• some templates with a logic (parser functions?) that trigger events based on dates (eg. when WUG renewal date is reached)

All of this should be doable with Scribunto/Lua (IMO) and maybe a gadget to make the editor a bit nicer.

Ok, problem here: SMW is all simple MW syntax, there are a bunch of examples and experience around.

I have no clue how to use Lua or Scribunto. I am happy to learn it but I certainly need help. This should all be up running until the next AffCom meeting (May 14). With SMW no problem, already know what to do. With Wikibase / Lua / Scribunto: No idea. So is there anyone helping me with that?

Lua and forms don't have much to do with each other. If the use case is simple, perhaps https://www.mediawiki.org/wiki/Extension:Cargo + SemanticForms is an option? Usage is similar for the end user and there's a much smaller codebase to review.

T64114: Disable new bug posting for some semantic extensions (moving to GitHub) would also need to be reversed.

SMW has had an unusually high number of security issues, so I would definitely advise against it for any wiki that contains private information.

I'm stunned by such statemens the WMF (or an employee on behalf of WMF) makes about SMW without citing the potential list of issues.

Given that the Municipal and Provincial Archives of Vienna (MA8) [0] hosts a SMW/Wiki and needed an external review with the result of "All of the used software components were security checked by the Information and Communications Technology (ICT) department of the City (MA 14)", I wonder about your assertion.

If you have a list of issues you see or like see to be addressed, the community would welcome it but making a statement without reference has the smell of public discreditation.

[0] http://semantic-mediawiki.org/wiki/Vienna_History_Wiki

I too would like some clarification. In particular, compared to what is the number of security issues so unusually high? Other MediaWiki extensions? Also, is this about relative or absolute amounts? And is this about SMW, or SMW and all extensions to it?

@greg, @csteipp, it seems clear that SMW isn't going to be installed on WMF-controlled sites. It is my understanding that SMW's reliance on the job queue and the performance hit that can cause on a large site is the big reason WMF has deemed it necessary to create other tools for use cases that SMW handles. (Of course, I'm willing to be corrected here since my understanding of all this is very poor.)

What is less clear is what actual security problems you've identified in SMW that have gone unresolved.

Given that orgs like CERT, NASA, and major corps are using SMW, it would help to identify the problems so that they can be fixed or their use of SMW can be re-evaluated and perhaps be replaced with the presumably more secure combination of Wikidata, Lua and JS,.

Since this is stalled and there is nothing for RelEng to do here, I'm removing our team's project.

In T88748#1179721, @mwjames wrote:

SMW has had an unusually high number of security issues, so I would definitely advise against it for any wiki that contains private information.

I'm stunned by such statemens the WMF (or an employee on behalf of WMF) makes about SMW without citing the potential list of issues.

Given that the Municipal and Provincial Archives of Vienna (MA8) [0] hosts a SMW/Wiki and needed an external review with the result of "All of the used software components were security checked by the Information and Communications Technology (ICT) department of the City (MA 14)", I wonder about your assertion.

If you have a list of issues you see or like see to be addressed, the community would welcome it but making a statement without reference has the smell of public discreditation.

@mwjames, it was not by intention to discredit SMW generally. There have been 3 closed security issues in SMW. That's more than some extensions, and less than others. I'd be very interested to see the results of the security audit performed by your city. They have probably examined it more thoroughly than I have.

I made that statement after I had triaged several bug reports from a bounty program where all of the reports were issues in SMW. Most of the issues have ended up being because the admin was confused about how to upgrade to a version of SMW that fixed all the known security issues, so my comment should have been, "Anecdotally, I'm seeing more issues reported against SMW than the default extensions distributed with the tarball, when it's been in scope for a bug bounty program, but that may be due to other factors".

In T88748#1183977, @MarkAHershberger wrote:

Given that orgs like CERT, NASA, and major corps are using SMW, it would help to identify the problems so that they can be fixed or their use of SMW can be re-evaluated and perhaps be replaced with the presumably more secure combination of Wikidata, Lua and JS,.

Mark, those organizations should do their own risk assessments. I'm not trying to say that SMW is worse than Wikidata+Lua+JS in combination. Those extensions have had more security issues fixed than SMW, although they have likely had more scrutiny as well. Each organization should run the minimum amount of code possible, and turn off any unessesary functions.

I am apposed to adding SMW to the existing Wikidata+Lua+etc on our cluster, because it increases our attack surface.

In T88748#1184340, @csteipp wrote:

I am apposed to adding SMW to the existing Wikidata+Lua+etc on our cluster, because it increases our attack surface.

This is reasonable. Thank you for this clarification.

In T88748#1184277, @csteipp wrote:

I made that statement after I had triaged several bug reports from a bounty program where all of the reports were issues in SMW.

Ugh... I think I saw a couple of those. Bug bounty programs seem like a good way for the offering org to be informed that it is time to update or reconfigure the software that they're using on their website, even if they aren't responsible for writing it.

This task started with a request for implementing a specific solution to a problem that has been vaguely defined.

In T88748#1078477, @80686 wrote:

The AffCom has reviewed its policies and needs to (re-)introduce certain workflows and management systems. We need to collect and maintain information of the orgs we deal with and trigger workflows based on deadlines or certain regular dates, eg. the renewal process for WUGs.

There are two basic questions:

• What good tools can solve this problem?
• Who will maintain the selected tool(s)?

Some options might be self-hosted, some hosted by third parties. Some options might have to be maintained by the Wikimedia Foundation, some options might allow to be maintained by the AffCom / other volunteers themselves (in Labs, somewhere else) or be maintained by third parties.

The discussion here goes around the Wikimedia Fundation teams not willing to take the responsibility of hosting SemanticMediaWiki in production, for the reasons explained. Are there other alternatives?

In T88748#1078778, @Nemo_bis wrote:

Lua and forms don't have much to do with each other. If the use case is simple, perhaps https://www.mediawiki.org/wiki/Extension:Cargo + SemanticForms is an option? Usage is similar for the end user and there's a much smaller codebase to review.

much smaller codebase

Line count is not everything. Cyclomatic complexity and so on are probably more relevant