A live Wikipedia edit firehose, bridged into Kafka
Every edit made to any Wikipedia page, anywhere in the world, arrives over a public live feed. This project listens to that feed and publishes every single edit onto a Kafka topic — so any number of independent systems can consume it, at their own pace, without ever touching each other.
Why bridge a Wikipedia feed into Kafka at all?
Wikimedia already broadcasts every edit as it happens. The interesting problem isn't getting the data — it's what happens once several different systems want to react to it, at different speeds, without getting in each other's way.
One system counts edits per language
It only cares about a running tally. It can be slow, batchy, and lag behind by minutes — that's fine, it just reads the topic at its own pace.
Another watches for vandalism
It needs to react within seconds. It reads the exact same stream of edits, independently, with no coordination with the counting system.
A third just archives everything
It writes every edit to cold storage, forever. If it falls a day behind, it simply catches up — Kafka kept every record.
None of them talk to each other
They don't call each other's APIs, they don't share a database row, and none of them can slow another one down. If one goes offline and comes back an hour later, it resumes exactly where it left off — because Kafka remembers the offset, not the consumer.
The actual engineering argument
You could have the SSE handler write straight into a database table. Here's specifically what that gives up.
Decoupling
The producer knows nothing about who's downstream, or how many consumers there are. New consumers can be added later with zero changes to the pipeline.
Replay from an offset
A consumer isn't reading "the current state" — it's reading a durable, ordered log it can rewind. Fix a bug, reset the offset, reprocess history.
Durability
Records are written to disk and replicated across brokers (when a real cluster is configured), so a crashed consumer doesn't lose data — it just isn't there to read it yet.
Back-pressure absorption
The Wikimedia feed doesn't wait for a slow consumer. Kafka absorbs the burst; a slower downstream system just falls behind on the topic instead of blocking the producer.
Horizontal scale by partition
A topic is split into partitions that can be consumed in parallel by a consumer group — the unit this page's animation makes visible.
From a public SSE feed to a Kafka topic
The real class names and the real topic name, exactly as they appear in kafka-producer-wikimedia.
Watch a Wikipedia edit move through the pipeline
Sample edits flow from the SSE connection, through the handler, into the producer, onto a partition, and out to a consumer group. Toggle keys and consumers to see two of Kafka's core mechanics happen in front of you.
Illustrative animation using sample event data — this page does not connect to the live Wikimedia stream.
No key: records round-robin across all three partitions. Turn on the key toggle to see the same wiki domain always land on the same partition.
The real code, and why it looks like that
Excerpts straight from kafka-producer-wikimedia, trimmed for length but otherwise unchanged.
1. Building the producer's Properties
Properties props = new Properties();
props.setProperty("bootstrap.servers", bootstrapServers);
props.setProperty("key.serializer", StringSerializer.class.getName());
props.setProperty("value.serializer", StringSerializer.class.getName());Kafka clients are configured through a plain Properties map — no framework, no annotations. Key and value are both serialized as strings here, because a Wikimedia change event arrives as a JSON string and is forwarded as-is.
2. Constructing the KafkaProducer
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
String topic = "wikimedia.recentchange";
EventHandler eventhandler = new WikiMediaChangeHandler(producer, topic);One producer instance is created for the lifetime of the stream and handed to the SSE handler, which is the only thing that will ever call send() on it.
3. Opening the SSE connection with EventSource.Builder
String url = "https://stream.wikimedia.org/v2/stream/recentchange";
EventSource.Builder builder = new EventSource.Builder(eventhandler, URI.create(url));
EventSource eventSource = builder.build();
eventSource.start();The LaunchDarkly EventSource client (built on OkHttp) owns the reconnecting HTTP connection to Wikimedia's public endpoint and dispatches every event to the handler — the actual network plumbing never appears in application code.
4. onMessage — an asynchronous send
@Override
public void onMessage(String s, MessageEvent messageEvent){
LOGGER.info("Received message : {}", messageEvent.getData());
//asynchronus
producer.send(new ProducerRecord<>(topic, messageEvent.getData()));
}producer.send() returns immediately — it hands the record to the producer's internal buffer and a background I/O thread does the actual network write. That's what lets a single callback thread keep up with a firehose of edits without ever blocking on Kafka.
5. onClosed — closing flushes what's buffered
@Override
public void onClosed() {
producer.close();
}KafkaProducer.close() blocks until every record still sitting in the internal buffer has been sent (it behaves like an implicit flush()). Because sends are async, this is the moment that guarantees nothing in flight is silently dropped when the SSE connection ends.
kafka-examples: a deliberate lab of core mechanics
Six focused main() classes, each isolating exactly one producer or consumer concept — built while working out how the pipeline above should behave.
ProducerDemo
Builds the minimal Properties, sends one ProducerRecord to notification-email-queue, then calls flush() and close(). The shape every other producer example builds on.
ProducerDemoWIthCallback
Sends batches of records in a loop, attaching a callback to each send(). On success, the callback logs the RecordMetadata — topic, partition, offset, timestamp; on failure, it logs the exception. This is how you find out what actually happened to a record you fired off asynchronously.
ProducerDemoWIthCallbackKeys
Same pattern, but every record now carries a key ("id_" + i). The callback logs the key next to the partition it landed on — proving that a given key always maps to the same partition, the exact mechanism this page's key toggle visualizes.
ConsumerDemo
Subscribes to notification-email-queue in group test-group, then loops forever calling consumer.poll(1000) and logging each record. Consumers don't get pushed data — they pull it in bounded batches. This version has no shutdown handling: killing the process just kills it mid-poll.
ConsumerDemoWithShutDown
Adds a JVM shutdown hook that calls consumer.wakeup() and joins the main thread; the poll loop catches the resulting WakeupException and closes the consumer in a finally block. wakeup() is the only thread-safe way to interrupt a blocked poll() — without it, a killed consumer just vanishes mid-batch instead of leaving the group cleanly.
ConsumerDemoWithCooperative
Identical to the shutdown-safe consumer, with one line added: partition.assignment.strategy set to CooperativeStickyAssignor. With Kafka's eager default, every consumer in the group revokes all its partitions and stops for the duration of any rebalance. Cooperative-sticky only revokes the specific partitions that need to move — everyone else keeps consuming, which is what the add/remove-consumer toggle above illustrates.
Skills the code actually evidences
- Streaming ingestion from an external SSE source (OkHttp + LaunchDarkly EventSource) into a long-running JVM process
- Kafka producer configuration and string serialization, wired through
java.util.Properties - Partitioning and ordering semantics — round-robin versus key-based assignment, and the ordering guarantee a key actually buys you
- Consumer groups, the poll loop, and rebalance strategies including cooperative-sticky assignment
- Graceful shutdown of a long-lived consumer via
wakeup()and a JVM shutdown hook - A multi-module Gradle project using the Kotlin DSL, with the
applicationplugin wiring a runnable main class
What this is, and what it isn't
It's a focused ingestion pipeline plus a learning lab of producer/consumer mechanics. It is not a production deployment, and it doesn't pretend to be.
Known gaps, stated plainly
- The broker address is hardcoded (
127.0.0.1:9092in the pipeline,localhost:9092in the examples) — point it at your own broker before running - No schema registry — events are forwarded as raw JSON strings via
StringSerializer, with no enforced or evolvable contract - No dead-letter topic or retry policy — a send failure is only logged
- No consumer reads
wikimedia.recentchangein this repository — the pipeline is producer-only - No delivery-guarantee tuning configured explicitly (
acks,enable.idempotence,retries) — the producer runs on Kafka's client defaults, not a deliberately chosen guarantee
What I'd add next
- Avro (or JSON Schema) with a schema registry, so the topic has an enforced, evolvable contract instead of raw strings
- An idempotent producer with
acks=alland bounded retries, for a deliberate exactly-once-per-partition guarantee - A consumer — plain or Kafka Streams — doing per-language or per-wiki aggregation off
wikimedia.recentchange - A containerised broker via Docker Compose, so the whole pipeline runs with one command instead of a hand-run broker
Prerequisites and commands
- JDK 25 (the
kafka-producer-wikimediamodule targets it via the Gradle Java toolchain) - A running Kafka broker, reachable at the address hardcoded in
WikiMediaChangesProducer.java— updatebootstrapServersto point at your own broker first
./gradlew :kafka-producer-wikimedia:runOpens an SSE connection to Wikimedia's recent-changes stream and publishes every event to wikimedia.recentchange.
./gradlew buildBuilds both modules. The kafka-examples classes have no application plugin configured — run them directly from your IDE (right-click a class → Run), as the README suggests.