Why Build Your Own?
Sometimes existing solutions don’t fit. We needed a scheduler that could:
- Handle 100k+ jobs per minute
- Tolerate node failures without job loss
- Support complex dependency graphs between jobs
Existing tools like Celery or Temporal were either too opinionated or too heavy for our use case.
The Architecture
We chose a single-leader architecture over consensus-per-operation. The trade-off: simpler implementation at the cost of a brief unavailability window during leader election.
Leader Election with Raft
We implemented a simplified Raft variant for leader election. The key insight from constraint-propagation applies here: reduce the state space by making strong guarantees early.
enum NodeState {
Follower { voted_for: Option<NodeId>, leader: Option<NodeId> },
Candidate { votes_received: HashSet<NodeId>, term: u64 },
Leader { next_index: HashMap<NodeId, u64> },
}
impl Node {
fn on_election_timeout(&mut self) {
self.state = NodeState::Candidate {
votes_received: HashSet::from([self.id]),
term: self.current_term + 1,
};
self.broadcast_vote_request();
}
}
Consistency Trade-offs
We operate under the CAP theorem constraints. Our choice:
In practice, this means jobs may be executed at least once during network partitions. We handle this by requiring all jobs to be idempotent.
If your jobs are not idempotent, you must use the exactly-once mode which adds ~15ms latency per job due to the two-phase commit protocol.
Graceful Degradation
When a worker dies, its in-flight jobs enter a “suspect” state:
- Wait for the heartbeat timeout ()
- Mark jobs as
SUSPECT - Wait for the recovery window ()
- Re-enqueue unacknowledged jobs
This prevents thundering herd problems while still recovering quickly.
See also: unix-philosophy for why we kept each component small and focused.