Checking in wikipedia-logger, improved version of wikipedia-hdfs

As the functionality is not really specific to HDFS, renamed the project to wikipedia-logger. switched to cronolog, which is presumably more reliable and clean. HDFS archiving is not re-implemented yet.
internetarchive · Dec 22, 2018 · faa8bdc · faa8bdc
1 parent d8c6dec
commit faa8bdc
Show file tree

Hide file tree

Showing 2 changed files with 30 additions and 0 deletions.
diff --git a/wikipedia/logger/start-logger.sh b/wikipedia/logger/start-logger.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+#
+HERE=$(cd $(dirname $0); pwd)
+
+: ${KAFKA_SERVERS:=crawl-db07:9092,crawl-db02:9092}
+: ${KAFKA_BIN:=$HERE/kafka/bin}
+: ${DATADIR:=$HERE/data}
+: ${CONSUMER_GROUP:=wikipedia-logger}
+
+topic=$1
+case "$topic" in
+  wiki-irc|wiki-links) ;;
+  '') echo Usage: $0 '{wiki-irc|wiki-links}' >&2; exit;;
+  *) echo invalid topic: $topic >&2; exit;;
+esac
+
+CONSUMER_OPTS=(
+  --bootstrap-server $KAFKA_SERVERS
+  --topic $topic
+  --consumer-property group.id=$CONSUMER_GROUP
+)
+$KAFKA_BIN/kafka-console-consumer.sh "${CONSUMER_OPTS[@]}" | \
+    cronolog $DATADIR/$topic-%Y-%m-%d.log --symlink $DATADIR/$topic.log
diff --git a/wikipedia/logger/wikipedia-logger.ini b/wikipedia/logger/wikipedia-logger.ini
@@ -0,0 +1,7 @@
+[program:wikipedia-logger-irc]
+directory=/1/crawling/wikipedia-logger
+command=/1/crawling/wikipedia-logger/start-logger.sh wiki-irc
+
+[program:wikipedia-logger-links]
+directory=/1/crawling/wikipedia-logger
+command=/1/crawling/wikipedia-logger/start-logger.sh wiki-links