While one would expect that such a crucial broken feature should be fixed within days, in Wikimedia environment even 9 months is not enough and I am quite sure that it will not be too difficult to conquer even 1 year boundary for WM tech team. The reason can be found at https://lists.gnu.org/archive/html/lilypond-devel/2020-10/msg00092.html , quoting Tim Starling:

I discussed a plan for rectifying it with Han-Wen, and suggested that we could contribute funding towards fixing it. However, I was not able to get approval for funding it.

In T257066#7013930, @Jan.Kamenicek wrote:

While one would expect that such a crucial broken feature should be fixed within days, in Wikimedia environment even 9 months is not enough and I am quite sure that it will not be too difficult to conquer even 1 year boundary for WM tech team. The reason can be found at https://lists.gnu.org/archive/html/lilypond-devel/2020-10/msg00092.html , quoting Tim Starling:

I discussed a plan for rectifying it with Han-Wen, and suggested that we could contribute funding towards fixing it. However, I was not able to get approval for funding it.

Having said Wikimedia Italia has some relevant partnerships with music archives under Wikipedia-compatible licenses and it seems they are interested in trying to help to resuscitate Score, funding the development of Shellbox to make it happen,

Is this request for funding still valid?

We have a budget for it now, but nobody to actually do it. But the plan is to re-enable LilyPond execution anyway once Shellbox setup is complete.

Change 703489 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/mediawiki-config@master] Re-enable Score using Shellbox on testwiki

https://gerrit.wikimedia.org/r/703489

Change 703790 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/mediawiki-config@master] Enable Score using Shellbox on testwiki

https://gerrit.wikimedia.org/r/703790

My tentative plan is to re-enable Score on test.wikipedia.org on Monday and leave it available for testing for the week. We still have a bit of work to do on monitoring, creating dashboards, etc.

After that, I'd like to re-enable it on 3-5 medium-ish wikis that would like to volunteer to help test for a week or two.

And if that goes well we can enable it everywhere. Note that we will *not* be re-enabling it on private wikis, loginwiki or votewiki (higher security wikis) and have no plan to do so. If you need it on one of those wikis please reach out/ask and we can discuss alternative options.

In T257066#7202898, @Legoktm wrote:

My tentative plan is to re-enable Score on test.wikipedia.org on Monday and leave it available for testing for the week. We still have a bit of work to do on monitoring, creating dashboards, etc.

After that, I'd like to re-enable it on 3-5 medium-ish wikis that would like to volunteer to help test for a week or two.

And if that goes well we can enable it everywhere. Note that we will *not* be re-enabling it on private wikis, loginwiki or votewiki (higher security wikis) and have no plan to do so. If you need it on one of those wikis please reach out/ask and we can discuss alternative options.

The Security-Team chatted about this within our clinic meeting today. Thanks for all of this planning and work - it sounds good to us. Please let us know if we can help with this strategy in any way.

@Legoktm some at enwikisource are pretty desperate to get this functionality back, so I imagine if I proposed at the enWS Scriptorium that we volunteer it would be supported. Would it be OK if I did that?

In T257066#7206368, @Inductiveload wrote:

@Legoktm some at enwikisource are pretty desperate to get this functionality back, so I imagine if I proposed at the enWS Scriptorium that we volunteer it would be supported. Would it be OK if I did that?

enwikisource is a bit on the larger side (see https://noc.wikimedia.org/conf/highlight.php?file=dblists/medium.dblist and https://noc.wikimedia.org/conf/highlight.php?file=dblists/large.dblist) but if we don't have enough medium wikis happy to bump it up.

If you want to start a discussion go for it :) For volunteering I ask that:

• Someone or people will verify that whatever specific Score functionality your wiki uses works in the new setup
• Keep an eye out for people reporting issues in your forums and forward them to Phabricator and cc me

In theory the main risk is that pages with scores (especially larger or multiple ones) might timeout and not be able to be saved. But that'll be a problem eventually anyways, so I think it would be nice to know ahead of time

We are using some tools in plwikisource that require the raw mode. The tools are intended to choose among various width scores (when rescaling a window) and for merging multiple scores. Will they work? The most "complex" lilypond feature they need is to define and use own variables (as \commands).
We are ready to test this, if possible.

OK, I have proposed it: https://en.wikisource.org/wiki/Wikisource:Scriptorium#Proposal:_Volunteer_to_test_the_reintroduction_of_the_Score_extension

Change 704149 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/mediawiki-config@master] Disable Score on private wikis

https://gerrit.wikimedia.org/r/704149

Change 703790 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable Score using Shellbox on testwiki

https://gerrit.wikimedia.org/r/703790

Mentioned in SAL (#wikimedia-operations) [2021-07-12T18:12:16Z] <legoktm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Enable Score using Shellbox on testwiki (T257066) (duration: 00m 58s)

In T257066#7206576, @Ankry wrote:

We are using some tools in plwikisource that require the raw mode. The tools are intended to choose among various width scores (when rescaling a window) and for merging multiple scores. Will they work? The most "complex" lilypond feature they need is to define and use own variables (as \commands).

Score is now enabled on test.wikipedia.org so I would suggest trying out those specific features there just to make sure they still work. Also, https://test.wikipedia.org/wiki/Score is the test page I check, so feel free to add an example that uses variables/commands/raw there.

Hard to do. It is a piece of LUA and requires proofreage namespaces. It would need not trivial porting to test there.
It works through {{#tag:score|...}} and LUA-generated code.

@Ankry can you provide a link to an example of a page using these complex scores?

https://pl.wikisource.org/wiki/Pro%C5%9Bba_dziewcz%C4%99cia
Transcluded using ProofreadPage and LUA from
pages:
https://pl.wikisource.org/wiki/Strona:Pro%C5%9Bba_dziewcz%C4%99cia.djvu/3
https://pl.wikisource.org/wiki/Strona:Pro%C5%9Bba_dziewcz%C4%99cia.djvu/4
https://pl.wikisource.org/wiki/Strona:Pro%C5%9Bba_dziewcz%C4%99cia.djvu/5
https://pl.wikisource.org/wiki/Strona:Pro%C5%9Bba_dziewcz%C4%99cia.djvu/6
https://pl.wikisource.org/wiki/Strona:Pro%C5%9Bba_dziewcz%C4%99cia.djvu/7
LUA modules used are:
https://pl.wikisource.org/wiki/Modu%C5%82:Mscore
https://pl.wikisource.org/wiki/Modu%C5%82:Mscore_resizable
+ few templates

In T257066#7206860, @Ankry wrote:

https://pl.wikisource.org/wiki/Strona:Pro%C5%9Bba_dziewcz%C4%99cia.djvu/3

I used Special:ExpandTemplates to get the underlying generated <score> tags for that page and saved them on https://test.wikipedia.org/wiki/Score/plwikisource where it looks like they all work. If it's fine with you, I'll add plwikisource to the list of early testing wikis for next week.

In T257066#7206899, @Legoktm wrote:

In T257066#7206860, @Ankry wrote:

https://pl.wikisource.org/wiki/Strona:Pro%C5%9Bba_dziewcz%C4%99cia.djvu/3

I used Special:ExpandTemplates to get the underlying generated <score> tags for that page and saved them on https://test.wikipedia.org/wiki/Score/plwikisource where it looks like they all work.

Well, it is not eactly the final code generated by LUA, but, I think, the final one would not need more features.

If it's fine with you, I'll add plwikisource to the list of early testing wikis for next week.

Yes, it would be nice.

In T257066#7206936, @Ankry wrote:

In T257066#7206899, @Legoktm wrote:

In T257066#7206860, @Ankry wrote:

https://pl.wikisource.org/wiki/Strona:Pro%C5%9Bba_dziewcz%C4%99cia.djvu/3

I used Special:ExpandTemplates to get the underlying generated <score> tags for that page and saved them on https://test.wikipedia.org/wiki/Score/plwikisource where it looks like they all work.

Well, it is not eactly the final code generated by LUA, but, I think, the final one would not need more features.

The final one also seems to work:
https://test.wikipedia.org/wiki/Score/plwikisource/2

I tried out the long-awaited return of Score on testwiki and it seems good. It's... surprising the OS features you can invoke in Lilypond but I don't understand them enough to seriously attack the Shellbox's sandboxing.

I got intermittent "Exception caught: Shellbox server returned status code 503 (Service Unavailable?)" errors while previewing, and some pages with Score have the T245377 Heisenbug with the TMH player.

Change 706020 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/mediawiki-config@master] Enable Score via Shellbox on enwikisource and plwikisource

https://gerrit.wikimedia.org/r/706020

Change 704149 merged by jenkins-bot:

[operations/mediawiki-config@master] Uninstall Score on private wikis

https://gerrit.wikimedia.org/r/704149

Change 706020 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable Score via Shellbox on enwikisource and plwikisource

https://gerrit.wikimedia.org/r/706020

Mentioned in SAL (#wikimedia-operations) [2021-07-21T23:15:42Z] <legoktm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Enable Score on enwikisource, plwikisource. Disable on all private/lockeddown wikis (T257066) (duration: 01m 03s)

@Ankry @Inductiveload it's enabled now. Please don't immediately go and mass-purge every page using Score but please test out specific cases and let it spread gradually. There is a known issue (thanks to @Skierpage for testing and reporting) that saving large/big pages with scores is returning 503 errors, I'm still looking into that.

First testing results:

• The old scores are working (likely due to being pre-generated with old setup)
• Scores containing any of the following commands cannot be modified (altering scores containing them triggers an error):

\paper { #(set-paper-size "a3") }
\override TupletBracket #'direction = #UP

"a3" is an example value. While the first can likely be avoided (as the default "a4" paper size can be used in most cases), impossibility to change direction of some features may be a problem for some complex scores. Eg. any white-space altering of this score: https://test.wikipedia.org/wiki/Score/plwikisource/3 triggers errors. Any hints?

Things that are broken in safe mode should be put in a separate task. I think in the long term, improving rather than disabling safe mode is the way to go. But in the case of #UP, I made a merge request for that which might fix that issue. If it's not deployed already, we should probably deploy it.

Adding a cache-busting "." to the title of https://en.wikisource.org/wiki/Abide_with_Me_(Illustrated_Victorian_Songbook) causes a parse failure:

https://test.wikipedia.org/wiki/Score/enwikisource/1

In T257066#7229715, @Inductiveload wrote:

Adding a cache-busting "." to the title of https://en.wikisource.org/wiki/Abide_with_Me_(Illustrated_Victorian_Songbook) causes a parse failure:

https://test.wikipedia.org/wiki/Score/enwikisource/1

This case is similar to the above (safe mode limitation). Removing

#(layout-set-staff-size 16)

from the \layout section makes it working.

In T257066#7229688, @tstarling wrote:

Things that are broken in safe mode should be put in a separate task. I think in the long term, improving rather than disabling safe mode is the way to go. But in the case of #UP, I made a merge request for that which might fix that issue. If it's not deployed already, we should probably deploy it.

After discussing with Tim, I'm going to just package the latest stable upstream version, 2.22.1 which includes that fix, https://gitlab.com/lilypond/lilypond/-/merge_requests/285 and the one for CVE-2020-17353. And as a bonus it includes SVG support too.

I created T287212 while Legoktm was already fixing it 😄

OK, we're now running lilypond 2.22.0 which should make some more things available in safe mode. Null edits should trigger re-renders.

https://test.wikipedia.org/wiki/Score/enwikisource/1 works now.

https://test.wikipedia.org/wiki/Score/plwikisource/3 isn't working for other reasons besides #UP I think, so that probably merits a dedicated task?

Hi everyone, I had an eye on here for a contributor in german wikibooks. I kept him informed about the progress and he published his tests there: https://test.wikipedia.org/w/index.php?title=Score&type=revision&diff=477408&oldid=477110

I for myself tested a little and stumbled across a problem while browsing the history. I'm sorry if this is not the right spot for this info. Feel free to move it, where it belongs. From time to time while browsing the history of a score-page the following error message occured (just checked right now on above mentioned page, it happens if you browse diffs too)

Best regards and thanks for your effort.

The error message I got was:

[d4440a45-5c93-428d-bb4a-05cfe847b8dc] /w/index.php?title=Score&direction=next&oldid=340358 Shellbox\ShellboxError: Shellbox server returned status code 503

Backtrace:

from /srv/mediawiki/php-1.37.0-wmf.15/vendor/wikimedia/shellbox/src/Client.php(183)
#0 /srv/mediawiki/php-1.37.0-wmf.15/vendor/wikimedia/shellbox/src/Command/RemoteBoxedExecutor.php(80): Shellbox\Client->sendRequest(string, array, array, array)
#1 /srv/mediawiki/php-1.37.0-wmf.15/vendor/wikimedia/shellbox/src/Command/BoxedExecutor.php(20): Shellbox\Command\RemoteBoxedExecutor->executeValid(Shellbox\Command\BoxedCommand)
#2 /srv/mediawiki/php-1.37.0-wmf.15/vendor/wikimedia/shellbox/src/Command/BoxedCommand.php(183): Shellbox\Command\BoxedExecutor->execute(Shellbox\Command\BoxedCommand)
#3 /srv/mediawiki/php-1.37.0-wmf.15/extensions/Score/includes/Score.php(150): Shellbox\Command\BoxedCommand->execute()
#4 /srv/mediawiki/php-1.37.0-wmf.15/extensions/Score/includes/Score.php(128): Score::fetchLilypondVersion()
#5 /srv/mediawiki/php-1.37.0-wmf.15/includes/libs/objectcache/BagOStuff.php(208): Score::{closure}(integer)
#6 /srv/mediawiki/php-1.37.0-wmf.15/extensions/Score/includes/Score.php(129): BagOStuff->getWithSetCallback(string, integer, Closure)
#7 /srv/mediawiki/php-1.37.0-wmf.15/extensions/Score/includes/Score.php(391): Score::getLilypondVersion()
#8 /srv/mediawiki/php-1.37.0-wmf.15/extensions/Score/includes/Score.php(262): Score::renderScore(string, array, Parser)
#9 /srv/mediawiki/php-1.37.0-wmf.15/includes/parser/Parser.php(3959): Score::render(string, array, Parser, PPFrame_Hash)
#10 /srv/mediawiki/php-1.37.0-wmf.15/includes/parser/PPFrame_Hash.php(341): Parser->extensionSubstitution(array, PPFrame_Hash)
#11 /srv/mediawiki/php-1.37.0-wmf.15/includes/parser/Parser.php(2916): PPFrame_Hash->expand(PPNode_Hash_Tree, integer)
#12 /srv/mediawiki/php-1.37.0-wmf.15/includes/parser/Parser.php(1584): Parser->replaceVariables(string)
#13 /srv/mediawiki/php-1.37.0-wmf.15/includes/parser/Parser.php(645): Parser->internalParse(string)
#14 /srv/mediawiki/php-1.37.0-wmf.15/includes/content/WikitextContent.php(375): Parser->parse(string, Title, ParserOptions, boolean, boolean, integer)
#15 /srv/mediawiki/php-1.37.0-wmf.15/includes/content/AbstractContent.php(591): WikitextContent->fillParserOutput(Title, integer, ParserOptions, boolean, ParserOutput)
#16 /srv/mediawiki/php-1.37.0-wmf.15/includes/Revision/RenderedRevision.php(266): AbstractContent->getParserOutput(Title, integer, ParserOptions, boolean)
#17 /srv/mediawiki/php-1.37.0-wmf.15/includes/Revision/RenderedRevision.php(235): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(WikitextContent, boolean)
#18 /srv/mediawiki/php-1.37.0-wmf.15/includes/Revision/RevisionRenderer.php(217): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string, array)
#19 /srv/mediawiki/php-1.37.0-wmf.15/includes/Revision/RevisionRenderer.php(154): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#20 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(MediaWiki\Revision\RenderedRevision, array)
#21 /srv/mediawiki/php-1.37.0-wmf.15/includes/Revision/RenderedRevision.php(197): call_user_func(Closure, MediaWiki\Revision\RenderedRevision, array)
#22 /srv/mediawiki/php-1.37.0-wmf.15/includes/poolcounter/PoolWorkArticleView.php(137): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#23 /srv/mediawiki/php-1.37.0-wmf.15/includes/poolcounter/PoolCounterWork.php(162): PoolWorkArticleView->doWork()
#24 /srv/mediawiki/php-1.37.0-wmf.15/includes/page/ParserOutputAccess.php(281): PoolCounterWork->execute()
#25 /srv/mediawiki/php-1.37.0-wmf.15/includes/page/Article.php(747): MediaWiki\Page\ParserOutputAccess->getParserOutput(WikiPage, ParserOptions, MediaWiki\Revision\RevisionStoreRecord, integer)
#26 /srv/mediawiki/php-1.37.0-wmf.15/includes/page/Article.php(559): Article->generateContentOutput(User, ParserOptions, integer, OutputPage, array)
#27 /srv/mediawiki/php-1.37.0-wmf.15/includes/actions/ViewAction.php(74): Article->view()
#28 /srv/mediawiki/php-1.37.0-wmf.15/includes/MediaWiki.php(538): ViewAction->show()
#29 /srv/mediawiki/php-1.37.0-wmf.15/includes/MediaWiki.php(320): MediaWiki->performAction(Article, Title)
#30 /srv/mediawiki/php-1.37.0-wmf.15/includes/MediaWiki.php(925): MediaWiki->performRequest()
#31 /srv/mediawiki/php-1.37.0-wmf.15/includes/MediaWiki.php(559): MediaWiki->main()
#32 /srv/mediawiki/php-1.37.0-wmf.15/index.php(53): MediaWiki->run()
#33 /srv/mediawiki/php-1.37.0-wmf.15/index.php(46): wfIndexMain()
#34 /srv/mediawiki/w/index.php(3): require(string)
#35 {main}

Another error-message was:

[4db28ebe-6ef8-4788-a6d1-dc8dffe0b882] 2021-07-22 10:18:04: Fatal exception of type "Shellbox\ShellboxError"

In T257066#7233684, @HirnSpuk wrote:

Hi everyone, I had an eye on here for a contributor in german wikibooks. I kept him informed about the progress and he published his tests there: https://test.wikipedia.org/w/index.php?title=Score&type=revision&diff=477408&oldid=477110

I for myself tested a little and stumbled across a problem while browsing the history. I'm sorry if this is not the right spot for this info. Feel free to move it, where it belongs. From time to time while browsing the history of a score-page the following error message occured (just checked right now on above mentioned page, it happens if you browse diffs too)

Best regards and thanks for your effort.

The error message I got was:

[d4440a45-5c93-428d-bb4a-05cfe847b8dc] /w/index.php?title=Score&direction=next&oldid=340358 Shellbox\ShellboxError: Shellbox server returned status code 503

Thanks for testing. This issue is known, but I filed a dedicated ticket for it since other people have mentioned it to me too: T287288: Score ocassionally gets a 503 response from Shellbox.

In T257066#7233536, @Legoktm wrote:

OK, we're now running lilypond 2.22.0 which should make some more things available in safe mode. Null edits should trigger re-renders.

I do hope you really meant 2.22.1 as that is the stable release for this version

In T257066#7233536, @Legoktm wrote:

https://test.wikipedia.org/wiki/Score/plwikisource/3 isn't working for other reasons besides #UP I think, so that probably merits a dedicated task?

This seems to be due to syntax changes in lilypond 2.22 vs. 2.18. On-wiki changes are needed here.
In my opinion this could be considered fixed.

In T257066#7234490, @Beeswaxcandle wrote:

In T257066#7233536, @Legoktm wrote:

OK, we're now running lilypond 2.22.0 which should make some more things available in safe mode. Null edits should trigger re-renders.

I do hope you really meant 2.22.1 as that is the stable release for this version

No, I meant 2.22.0, which is the version currently in buster-backports, see Special:Version on any wiki to get the lilypond version. I did attempt to get 2.22.1 packaged, but it was more complex than I expected it to be so I punted on it (see T287212#7233183).

In T257066#7233536, @Legoktm wrote:

https://test.wikipedia.org/wiki/Score/plwikisource/3 isn't working for other reasons besides #UP I think, so that probably merits a dedicated task?

IIRC I did look at the prospects for \paper in safe mode, but it's much more complicated than exposing a few constants. A layout in LilyPond is a module with an unsafe Guile environment embedded in it. It'll be easier to fix if we do that proposed contract job of making all modules be safe. Or the syntax for paper size declarations could be simplified, making it both easier to implement and easier to use.

Safe-mode also prevents usage of "include" in the lilypond (necessary for some stuff like gregorian chant, which would be mildly useful for a project on Wikisource which had been interrupted due to this). I'm not sure if it's possible to fix this. Please advise if you have anything on this. I can provide an exact example of valid lilypond code that currently doesn't work on-wiki on request.

I'd like to enable Score on one or two more projects of any size for a bit more testing before doing a full rollout next week. @Tpt would fr.wikisource be interested in that (Inductiveload suggested I ask you :))?

In T257066#7257726, @RandomCanadian wrote:

Safe-mode also prevents usage of "include" in the lilypond (necessary for some stuff like gregorian chant, which would be mildly useful for a project on Wikisource which had been interrupted due to this). I'm not sure if it's possible to fix this. Please advise if you have anything on this. I can provide an exact example of valid lilypond code that currently doesn't work on-wiki on request.

Can you file a dedicated task in MediaWiki-extensions-Score for this please? And include the lilypond code that doesn't work.

Change 713726 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/mediawiki-config@master] Re-enable Score with Shellbox on most public wikis

https://gerrit.wikimedia.org/r/713726

Change 713726 merged by jenkins-bot:

[operations/mediawiki-config@master] Re-enable Score with Shellbox on most public wikis

https://gerrit.wikimedia.org/r/713726

Mentioned in SAL (#wikimedia-operations) [2021-08-19T16:49:12Z] <legoktm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Re-enable Score with Shellbox on most public wikis (T257066) (duration: 01m 08s)

Should this be announced in the next Tech News edition?
Please confirm if that is correct, and if this draft-wording is suitable (or amend as needed):

[in the "Recent changes" section...]
The [[Special:MyLanguage/Extension:Score | Score]] extension has been re-enabled on public wikis. The security issue has been fixed. [link here]

Thanks!

In T257066#7295473, @Quiddity wrote:

Should this be announced in the next Tech News edition?
Please confirm if that is correct, and if this draft-wording is suitable (or amend as needed):

[in the "Recent changes" section...]
The [[Special:MyLanguage/Extension:Score | Score]] extension has been re-enabled on public wikis. The security issue has been fixed. [link here]

Thanks!

Done: https://meta.wikimedia.org/w/index.php?title=Tech%2FNews%2F2021%2F34&type=revision&diff=21908187&oldid=21907984

I'm also going to be publishing a retroactive incident report and declassifying most of the private tasks later today/tomorrow.

It would be nice if someone could put together a page tracking all the functionality that no longer works in safe mode so editors can identify what needs to be removed and it provides a work list for any lilypond developers who want to help us out.

I've only noticed one issue so far, Scores written in ABC are failing: T289298: Scores written in ABC failing with "PCRE regular expression replacement failed". I'll deploy the fix for it shortly.

Many thanks for your efforts! I'm glad we're having this feature running again.

Just a thank you to Tim and Lego for working on this for all that time. I know its been quite a bit of work.