User:Magnus Manske/quick statements2
< User:Magnus Manske

This is an informal brainstorming page to collect ideas for a version 2 of QuickStatements (aka QS). The underlying assumption here is a complete rewrite of the tool from scratch, allowing backwards compatibility to the current version without being limited by it. All constructive suggestions are welcome, no promises as to implementation! See also current QS issues on bitbucket, includes feature requests, which should probably be copied here to get a better overview.
PRE-ALPHA VERSION IS UP FOR TESTING
I have started implementing a new version here. It should be able to perform all the version 1 commands (see "import commands"). Co-developers welcome!
CAUTION! The code is mostly untested and under heavy development. It might nuke Wikidata and steal your lunch. Try it on the Sandbox item (Wikidata Sandbox (Q4115189)) only, for now.
Initialising
Interface
Instead of writing: Qx|Q17|Q801||Qa|Q17|Q801||Qb|Q17|Q801||Qc|Q17|Q801.... ==> Qx,Qa,Qb|Q17|Q801. Possible? Thanks.Mikey641 (talk) 14:57, 2 April 2017 (UTC)
Bugfixes for V1
Requirements over current version
Reconciliation
Having imported some bibliographic items I miss support of reconciliation (matching strings to items). How about this: the character ? before a string value should indicate that this value must manually be reconciliated. Example:
LAST P50 ?"Larry Wall" LAST P123 ?"O'Reilly"
The user interface should ask to select an item via dropdown/autosuggest, e.g. Larry Wall (Q92597) and O’Reilly Media (Q1065097). Suggestion may be improved by limiting the list to items of the type required by property, e.g. author (P50) has property value type constraint human (Q5). But even without this optimization reconciliation would be great help.
Optional support of language to select from:
LAST P17 ?de:"Deutschland"
-- JakobVoss (talk) 08:01, 17 August 2017 (UTC)
Avoid duplication
Could the tool give a warning when someone tries to create a new entry which already exists in Wikidata? I especially think in the context of publications where the same title may already indicate that this item may already exist. --Zuphilip (talk) 12:14, 8 June 2017 (UTC)
To do so, quickstatements would need to query selected LAST ... statements whether there is already an item with this statement. How do we indicate which statement should be used to check for duplicates? Maybe check for properties with distinct-values constraint (Q21502410)? -- JakobVoss (talk) 08:01, 17 August 2017 (UTC)
Thoughts on Code and implementation
Model
First attempt at modelling the data flow.
Internal JSON format
Each command is a JSON object. Qxxx is an item, Pxxx is a property, PQxxx is either. {datavalue} is a JSON object representing a datavalue in Wikidata API notation.
Adding a statement
 Done
{"action":"add","what":"statement","item":"PQxxx","property":"Pxxx","value":{datavalue}}
Adding a qualifier
 Done
{"action":"add","what":"qualifier","item":"PQxxx","property":"Pxxx","value":{datavalue},"qualifier":{datavalue_qualifier}}
requires either a statement as above, or a statement ID as "id":"...". {datavalue_qualifier} is the datavalue of the qualifier, in Wikidata API notation.
Adding sources
 Done
{"action":"add","what":"qualifier","item":"PQxxx","property":"Pxxx","value":{datavalue},"sources":[{datavalue_source,...}]}
requires either a statement as above, or a statement ID as "id":"...". [{datavalue_source}] is an array of datavalue of the sources, in Wikidata API notation.
Adding a label/description/alias
 Done
{"action":"add","what":"label","item":"PQxxx","language":"ISO code","value":"The new label"}
what is label, description, or alias; the latter adds, the others replace
Adding a sitelink
 Done
{"action":"add","what":"sitelink","item":"PQxxx","site":"enwiki","value":"Wiki page name"}
Remove statement/sources/qualifier
Use "action":"remove" on any of the above to remove. A value may be "any" to remove all instances.
Create a new item
 Done
{"action":"create","type":"item"}
type can be item or property
Documentation for v2
While the first version of the tool had a good explanation, the UI for QuickStatements2 does not offer any hint at how we are supposed to create a command. How about a short example showing how to write and run a sample command? The only way I can find to enter a command is the "Import from v1" option, I can't believe it is the only way, right? Or is it? Thank you! Syced (talk) 09:20, 7 September 2017 (UTC)
"Import from v1" has the only documentation I can find. The tool is also backward compatible with older version which is well documented. --Jarekt (talk) 11:41, 7 September 2017 (UTC)
Same question as Syced, is "Import from v1" the only option? (it's the only one that I used).
Jarekt for the documentation for the v1, it's quite good but it's improvable. For instance, it doesn't mention subtleties like the calendar for dates, the ranking or badges (I guess it's not possible but ideally, it should be clearly said). In the end, I think that a specific documentation for the v2 is necessary and maybe a central for asking "How can I do this" would be welcome (instead of asking on the Project chat, on talk pages, on Twitter or on IRC here the discussion are losted).
Cdlt, VIGNERON (talk) 17:46, 3 October 2017 (UTC)
May be we should create some un-official project page Help:Quick statements where we would copy current V1 documentation and start expanding it, with what we know about V2. --Jarekt (talk) 17:55, 3 October 2017 (UTC)
I started a documentation page: Help:QuickStatements‚Äč. Feel free to enrich it, thanks! Syced (talk) 04:05, 4 October 2017 (UTC)
Syced, Great. I am working on expanding it. --Jarekt (talk) 13:21, 4 October 2017 (UTC)
Thanks! I agree https://tools.wmflabs.org/wikidata-todo/quick_statements.php is gentler on new users. I had forgotten the exact syntax so I had no clue what to do on the new version, but after I checked the old documentation it's working great. :) Nemo 08:23, 17 October 2017 (UTC)
New location of the QuickStatements tool Geertivp (talk) 21:49, 20 May 2020 (UTC)
Upload report for batch uploads
Thank you for creating such a useful tool. I am using this tool for writing descriptions in Malayalam language. I have some suggestions for improving the tool :
I want to back this request by Netha Hussain. I think there is indeed a need for a better report feature for uploads via Quicstatements1&2. Has this already been considered? Getting the following overviews would really help keeping Wikidata clean:
@Magnus Manske: I would really like to know your opinion on that. Thank you! - Alina data (talk) 14:26, 21 February 2018 (UTC)
Removing a property does not seem to work
I am trying to remove a property from an item.
First I clicked "Import commands" then "Version 1 format" then I entered the following, which contains a proper TAB character:
-Q5332409 P571
Upon clicking "Import", nothing happens. Just "No data available in table".
Second I read in the documentation that Remove is a V2-only feature, so I decided to try the V2 syntax, which is apparently:
Q5332409|P571
There is nowhere to import this command, the Import command menu is only about V1, strangely. So I tried running it via URL: https://tools.wmflabs.org/quickstatements/#v2=-Q5332409%7CP571 but that does not work either, only "No data available in table" is shown.
Has anyone managed to remove a property? If yes, how?
Thank you! Syced (talk) 06:20, 6 November 2017 (UTC)
Syced I was just removing some properties with lines like -Q33169363|P373|"Aoike-chŇć, Nagoya" and it worked just fine. My guess is that you still need the value of the property you are removing. Which makes sense in case there are several. That is also what the documentation suggest. --Jarekt (talk) 18:36, 6 November 2017 (UTC)
Add multiple sources in one claim
Hi. I'd like to add two sources in one claim, but in the new interface of QuickStatements V2, I cannot do that. The below code works fine in older interface:
LAST|P5205|14|S143|Q33109119||LAST|P5205|14|S854|"‚Äčhttp://kanji.jitenon.jp/kanjid/1895.html‚Äč"
, but in the new one, it makes two duplicate claims (‚Äčhttps://www.wikidata.org/w/index.php?title=Q55406214&oldid=707356668‚Äč). This phenomenon doesn't happen when it does not entail an item creation (‚Äčhttps://www.wikidata.org/w/index.php?title=Q12174424&type=revision&diff=707251396&oldid=707250957‚Äč). --Okkn (talk) 06:31, 7 July 2018 (UTC)
Ability to export QS batch to tab/csv
As a use case ... my batch has completed. It has 89 errors. I want to download the errors in the same (tab, csv) format as was used to upload them, so that I can work through them to fix issues. (Sure I could work from the error list in the QS interface, but I cannot, for instance, track my progress as I wade through issues, and I have 30 batches each with ~50 errors per batch & am bear of small brain.) So. I'm thinking it would be good to be able to extract the contents of a batch, perhaps suffixed by a #status. thx --Tagishsimon (talk) 08:12, 17 July 2018 (UTC)
Would also be useful to be able to view errors for a QS temporary 'browser-based' batch when viewing the batch through the discuss/revert link - cf. https://tools.wmflabs.org/editgroups/b/QSv2T/1533259904195/ - albeit I appreciate right now you likely do not store the data? Errors tend to be the most interesting thing for me (playing with labels & descriptions) as they often point to duplicate items. But I lack the time, now, to fix those from my last batch, and lack the discipline to keep a browser tab open until I do have time, days or weeks later. --Tagishsimon (talk) 11:50, 3 August 2018 (UTC)
Petscan to QS
Hello Magnus,
I don't understand why this Petscan https://petscan.wmflabs.org/?psid=5146689&al_commands=-P50%3AQ8254925%0AP50%3AQ4233718%0A when I click on "Start QS" and run, gives errors for removal of erroneous value, while still adding the new value. In Petscan, it runs OK, so it's not a syntax problem.
Can you please explain what I do wrong ? Thanks for you help. Hsarrazin (talk) 23:42, 24 July 2018 (UTC)
PS : I noticed that on the REMOVE action, the QID lacks Q at the begining. Could this be the cause ? Hsarrazin (talk) 13:03, 25 July 2018 (UTC)
I saw today that it is now fixed... Thanks ! \o/
Can't submit literals in CSV format?
Hi @Magnus Manske: thanks for this great tool!
But how can we submit literals (strings) using the new CSV format? Its documentation Help:QuickStatements#CSV_file_syntax says "The double quotes for string values seem to interfere with CSV syntax. Empirically four double quotes before and one after the string have been found to work" but
Is it possible for you to use TSV as the basic format, and then hopefully quotes will make it through unmangled? Otherwise the utility of the CSV format is very limited, and we'll have to stick to the v1 format --Vladimir Alexiev (talk) 16:06, 19 November 2018 (UTC)
QuickStatements for Commons
Tracked in Phabricator
Task T181062
Structured data gets rolled out on Commons, for now only with labes. Would it be possible to enable QuickStatements for Commons? (Ping for @Keegan (WMF): who knows about eventual technical issues) --GPSLeo (talk) 09:43, 11 January 2019 (UTC)
Added the related Phabricator ticket. Some functionalities are currently possible, but it's still a pain that you need to know the M numbers of the files you want to edit (i.e. you can't just feed a list of filenames into QS). Spinster ūüí¨ 08:46, 22 November 2019 (UTC)
Removing a property does not seem to work
I am trying to remove a property from various items. I have tried several formats, but I can't remove P571. For example,
-Q2644688 P571 1942
-Q2644688 P571 1942-01-01T00:00:00Z
-Q2644688 P571 1942-01-01T00:00:00Z/09
-Q2644688|P571|1942
-Q2644688|P571|1942-01-01T00:00:00Z
-Q2644688|P571|1942-01-01T00:00:00Z/09
--&beer&love (talk) 13:54, 7 October 2019 (UTC)
@&beer&love: The format is:
-Q2644688 P571 +1942-00-00T00:00:00Z/9
Note 00 for month and day when precision is year, and leading +. I removed a few of the inception (P571) statements using this format, e.g. on Frank Zappa (Q127330). -LiberatorG (talk) 07:09, 8 October 2019 (UTC)
Thanks. --&beer&love (talk)
Hitting wall in removing P:1552 from category using Petscan.
Hi there, I'm hitting a wall on this and seeking help, I've filed a bug on Phabricator, and wanted to give you a heads up here.

Thanks Magnus! Shameran81 (talk) 22:58, 12 November 2019 (UTC)
Adding a single qualifier multiple times
Hi Magnus. I am currently importing data from past censuses in Macedonia that could be later easily re-used in templates across Wikipedia articles (see WikiProject Macedonia for more details). I know that the tool can not add a single qualifier multiple times, which visually makes it a bit confusing that there is only one value with multiple census years (see at Evla (Q3041202) for instance). Do you know any other way (other than manual editing) this could be visually improved? Thank you.--Kiril Simeonovski (talk) 07:09, 28 February 2020 (UTC)
Bug in line 16
Showing a (running) batch, like for instance https://tools.wmflabs.org/quickstatements/#/batch/26133‚Äč, an error occurs:
TypeError: j.datavalue is null
in line 16
if ( typeof j.datavalue != 'undefined' && j.datavalue.type == 'wikibase-entityid' ) to_cache[j.datavalue.value.id] = 1 ;
Thanks! Bargioni (talk) 21:19, 1 March 2020 (UTC)
Is there a bug logging in?
After hitting log in mediawiki, I get "Error retrieving token: mwoauthdatastore-bad-verifier" ? Bouzinac (talk) 22:30, 15 March 2020 (UTC)
Other problem logging in
I have aother transient error during login; a workaround is to retry the Oauth login once again.
Error retrieving token: mwoauthdatastore-request-token-not-found
Possibly related to previous error?
Geertivp (talk) 10:26, 28 March 2020 (UTC)
TSV copy/paste import not implemented
New requirement: import from TSV file via copy/paste.
I am preparing batches using Excel. Excel copy/paste by default generate TSV format. Then I have to copy/paste first to notepad to replace all TABs to "," via find/replace all. This step would be avoided when Quickstatements2 would also allow to straight import TSV format files in addition to CSV format. Geertivp (talk) 10:26, 28 March 2020 (UTC)
50% error rate solved after repeated retry
I am running a well structured batch, but 50% of the transactions fail without any good reason. When I reset the errors and retry the batch (without changing the input file) the transactions are correctly executed eventually after several retries.
Transient server (overload) errors? (local/remote) Network instablility? Geertivp (talk) 10:26, 28 March 2020 (UTC)
I have exactly the same issue. Has happened repeatedly, it is very annoying and , in many cases, it ends up selectively failing to introduce core information. Any thoughts on how to solve it? @Geertivp: did you have any progress on that? TiagoLubiana (talk) 17:33, 11 April 2020 (UTC)
As I indicated above, the only workaround I currently have is, after the current batch terminated, to reset the errors, and re-execute the remaining transactions. After repeating this exercise several times, the complete batch was eventually completed. Very annoying. Geertivp (talk) 10:25, 12 April 2020 (UTC)
Sometimes it is completed but says there are errors, and sometimes it makes the same edit twice. Batch 30306, now completed, has "100% (4448) of 4647 done, 199 errors" after several attempts, and although some were because I forgot to check for duplicates (the list included Q21891735 and Q26380071 twice, and Q26505508 four times), it looks like most of the "errors" are attempts to retry removing statements that were removed the first time but had not been reported as successful. Also https://tools.wmflabs.org/editgroups/b/QSv2/30306/ says 4679 edits, so some edits were performed twice by Quickstatements - at least 12 of the 4647 were not done, so if the number is correct there are 44 of these. I found one of them, Q26612148, [1] and addition of P131 there failed the first time, then when retrying it was added twice. Peter James (talk) 18:18, 19 April 2020 (UTC)
Interactive session not responding
I am running an interactive session. It is not responsive any more. I need to click (possibly multiple times) on STOP/Start to resume processing. Job is eventually completed, but this is kind of embarassing.
Kind of timeout? Geertivp (talk) 10:38, 28 March 2020 (UTC)
Might be related to next problem. Geertivp (talk) 10:25, 12 April 2020 (UTC)
Extremely low execution speed
It appears in Western Europe, that the only time of day that the transactions are executed at 1 transactions per/s is (early) in the morning when the USA is sleeping and Asia is going to bed. Geertivp (talk) 10:25, 12 April 2020 (UTC)
Impossible to switch back and forth to offline batch mode once interactive mode is started
I would like to have to possibility to start first to execute a batch interactively, and then switch to offline batch mode, once the jobs seems to be running OK. Currently, once I start online, I cannot transfer the batch to offline. I need to stay connected all the time and have no option to do so, except as to abandon the current bath, and resubmit the remaining transaction in offline batch mode.
The other way round would be to switch to online batch mode again to monitor each transaction as it executes. You could compare this functionality with the &, fg, and bg commands of Linux where you have the possibility to switch between foreground and backgrond process execution mode. Geertivp (talk) 10:25, 12 April 2020 (UTC)
Adding multiple statements with identical properties & values, but different qualifiers
Be great if this were possible - currently trying to find a way to add multiple terms of office in the same role in bulk. The only workaround I've found thus far would be to create new items and merge, but that seems... messy. Can also see uses for individuals winning an award multiple times, etc. Lirazelf (talk) 12:48, 14 April 2020 (UTC)
Process URL encoded values
When assiging a Commons Category, Wikidata Query Download does not properly encode UTF-8 URL strings. It leaves the URL encoding untouched. Example in V1 format:
Q15630294 P373 "Andr%C3%A9_Dumas"
This does not work with QuickStatements when assigning a Commons Category, as the %C3%A9 should be converted to "é" and the _ (underscore) to " " (space). It would be wonderful when Wikidata Query Download, or QuickStatements would convert the URL encoded string automatically into "André Dumas". For more information, see d:User:Geertivp/training/Wikidata Query/Commons Creator page without Commons category. Geertivp (talk) 19:25, 18 April 2020 (UTC)
I have found a work-around by using schema:name to extract the proper target Category, that can be processes without further conversion in QuickStatements V1 syntax. Geertivp (talk) 20:55, 18 April 2020 (UTC)
Troubleshooting
Hello, I've tried to input that QS file but 1/2 of the statements come out into error. Retry and retry and some statements still in error. Is there a reason? Thanks The troubled batch : https://quickstatements.toolforge.org/#/batch/36531 Bouzinac (talk) 13:22, 6 June 2020 (UTC)
Batches not being seen
I've now run three batches by hand, using the v2 interface at https://quickstatements.toolforge.org/#/batch . (The first and third were successful; the second failed, for trivial and now-fixed reasons.) Unfortunately, these are not turning up on my "last batches" list. Presumably as a result of this, it looks like I can't make any API-based submissions, as I get the message "Problem generating OAuth signature; user '{}' needs to have submitted a batch namually at least once before" from the API. Can anybody help? -- The Anomebot 3 (talk) 14:00, 23 June 2020 (UTC) (User:The Anome, currently logged in as my bot)
Update: looks like the earlier batches didn't get logged because they didn't have a name. Hasn't fixed my login problem yet, but it's one step forward. -- The Anomebot 3 (talk) 14:25, 23 June 2020 (UTC)
OK, I've thrown a big batch in manually, and it seems to have disappeared without trace. Back to my first approach of talking directly to the API. -- The Anomebot 3 (talk) 15:52, 24 June 2020 (UTC)
Errors adding identifiers
Batch 39056 consists of 8 edits; errors on four the first time, two were successful on second attempt but the two edits that add identifiers always fail with "No success flag set in API result". Peter James (talk) 14:42, 29 July 2020 (UTC)
How to troubleshoot
Hi Magnus Manske Thanks so much for QS tool! I would love to optimize this tool for uploading our collections at the Smithsonian! I am new to Wikidata and very new to QS for bulk creating new items. I have been testing QS to establish workflow for our library team which has just started to test QS bulk load.
When I followed instructions for commands (V1 and CSF), QS system seemed to read the data fine but hanging after clicking <Run>. I have received some gracious pointers when I reported on the RUN_takes_a_long_time Talk page with screenshots. But the QS still not loading the data. Is this beyond the syntax issue that are not obvious to me? Is there anything else I need to adjust.
Incidentally, like the previous user, somehow I am also not able to see the previous loads via <Your last batches>. Thank you very much for your help! jshieh (talk) 14:53, 30 July 2020 (UTC)
CREATE LAST Len "Morris Blackburn" LAST P31 Q5 LAST P21 Q6581097 LAST P27 Q30 LAST P569 +1902-00-00T00:00:00Z/9 LAST P570 +1979-00-00T00:00:00Z/9 LAST P106 Q1028181 CREATE LAST Len "Berthold Schmutzhart" LAST P31 Q5 LAST P21 Q6581097 LAST P27 Q30 LAST P106 Q1028181

jshieh, Magnus did not replied on this talk page since 2016, so you might need to get help from other users. I would suggest:
  1. trying small batches to test the tool. In big runs do not run batches bigger than 25k "simple" statements (like the ones above).
  2. use "Run" not "Run in the background" as they have different behaviors
  3. Your above statements look good except that they are duplicates of Morris Blackburn (Q20860828) and Berthold Schmutzhart (Q98447422). And creating duplicate records should not be happening.
I am using the tool quite a lot (see https://quickstatements.toolforge.org/#/batches/JarektBot or https://quickstatements.toolforge.org/#/batches/Jarekt‚Äč) and can try to help if you have issues with it. Jarekt (talk) 03:16, 24 September 2020 (UTC)
QuickStatements API cannot write date statements?
Copied from Help_talk:QuickStatements
Following up on this earlier note, I've tracked down the problem to the QuickStatements API and dates. It seems specifically that the QuickStatements API cannot write date statements. I've tried with and without quotes (%22) around the date. See the examples below (I've blanked out my token; Find yours here if you want to test):
Can anyone spot a workaround (please reply at Help_talk:QuickStatements‚Äč)? It seems a pretty important feature to be missing. T.Shafee(evo&evo) (talk) 00:17, 1 June 2021 (UTC)
Removing Lexeme forms grammatical features
Hi, I need to replace about 1500 grammatical features in 500 lexemes. My plan is to remove the wrong grammatical features, then add the correct one. My initial batch look like:
-L576680 L576680-F1 Q1317831
but it gave error. I guess this means the feature hasn't been implemented yet?
Is there any other way to do this? I tried to ask in Wikidata:Bot_requests#Request_to_change_lexeme_forms'_grammatical_features_(2021-07-08)‚Äč, but no reply for 3 months now.
If there's no other way, then I'd have to ask to delete those 500 lexemes and just upload a new batch...
Thanks! Bennylin (talk) 15:04, 1 October 2021 (UTC)
Last edited on 1 October 2021, at 15:08
Wikidata
All structured data from the main, Property, Lexeme, and EntitySchema namespaces is available under the Creative Commons CC0 License; text in the other namespaces is available under the Creative Commons Attribution-ShareAlike License; additional terms may apply. By using this site, you agree to the Terms of Use and Privacy Policy.
Privacy policy
Terms of Use
Desktop
HomeRandom Nearby Log in Settings DonateAbout WikidataDisclaimers
WatchEdit