Obsolete:RCStream

From Wikitech
Revision as of 19:31, 8 March 2015 by Petrb (talk | contribs) (→‎Other client libraries: add note)

RCStream is a simple server daemon that broadcasts activity from MediaWiki wikis using the Socket.IO protocol. An instance runs at stream.wikimedia.org. It subscribes to the Redis feed of recent changes on WMF's production cluster, and publishes this on the endpoint 'stream.wikimedia.org/rc'. As a web developer, one can tap the stream using JavaScript. As an app developer, one can use a suitable client library for your platform.

Note: As of 2015-01 this uses version 0.9 of the Socket.IO protocol, not 1.0 (phab:T68232). 0.9 documentation is here and here.

API

RCStream provides a simple API for subscribing to the recent changes feed of a set of Wikimedia wikis. After connecting you emit a 'subscribe' custom event, specifying the wikis you wish to subscribe to:

  • a single hostname such as 'nl.wikimedia.org'
  • an array of hostnames
  • hostnames matching a shell-style pattern such as '*.wikivoyage.org'
  • all wikis by subscribing to the special topic name '*'.

You then receive 'change' custom events whose data is an RCFeed structure containing the type of change, the title of the page, the new revision number, etc.

The RCStream server also responds at http://stream.wikimedia.org/rcstream_status with a simple text message; check this if you do not receive any events.

Client

Consuming RCStream is a cleaner approach than parsing the change messages generated on irc.wikimedia.org channels by IRCD. The WMF production cluster continues to generate both, see $wgRCFeeds in CommonSettings.php.

JavaScript

// Requires socket.io-client 0.9.x :
// browser code can load a minified Socket.IO JavaScript library;
// standalone code can install via 'npm install socket.io-client@0.9.1'.

var io = require( 'socket.io-client' );
var socket = io.connect( 'stream.wikimedia.org/rc' );

socket.on( 'connect', function () {
     socket.emit( 'subscribe', 'commons.wikimedia.org' );
} );

socket.on( 'change', function ( data ) {
    console.log( data.title );
} );

Python

#!/usr/bin/python
# -*- coding: utf-8 -*-
import socketIO_client

class WikiNamespace(socketIO_client.BaseNamespace):
    def on_change(self, change):
        print '%(user)s edited %(title)s' % change

    def on_connect(self):
        self.emit('subscribe', 'commons.wikimedia.org')


socketIO = socketIO_client.SocketIO('stream.wikimedia.org', 80)
socketIO.define(WikiNamespace, '/rc')

socketIO.wait()

If you needed to convert the structure to XML, you could import dicttoxml and then in on_change() do something like: 

        xml = dicttoxml.dicttoxml(change)
        dom = dicttoxml.parseString(xml)
        print dom.toprettyxml()

Other client libraries

Note: In case you are using other programming languages than JavaScript or Python, consider using XmlRcs instead, most of socket.IO libraries for low level languages are obsolete and not working.

See also

Source code: rcstream
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Obsolete:RCStream&oldid=147444"