Hot-Swapping Cron Schedules with MongoDB Change Streams

If your background jobs run on cron expressions baked into code or environment variables, then changing when a job runs means a redeploy. Move a report from 9am to 8am, and you are shipping a build, running a pipeline, and restarting a process to change one number. It is slow, it is risky out of proportion to the change, and it discourages the small tuning that keeps a system healthy. The fix is to store schedules in the database and have the scheduler react to edits in real time. MongoDB change streams make the reaction immediate. This post covers the design, the reconnection handling that makes it reliable, and when it is more machinery than you need.

The problem with schedules in code

A hardcoded schedule couples two things that change at completely different rates and for completely different reasons. The job's logic changes when the work changes, which is an engineering decision that belongs in a build. The job's timing changes when operational needs change, a report should land before a meeting, a heavy task should move off peak hours, and that is an operations decision that has nothing to do with the code. Binding them means every timing tweak drags the full weight of a code deploy behind it.

The cost is not just the deploy time. It is the chilling effect. When changing a schedule is a production deploy, people stop doing it. Schedules drift out of alignment with actual needs because nobody wants to ship a build to move a job by an hour. The schedule should be data you can edit, not code you have to release.

Schedules as data

The first move is to put the schedules in a collection, one document per job, with the cron expression as a field.

// One document per scheduled job. The cron expression is data, not code.
type Schedule struct {
	ID        string    `bson:"_id"`        // e.g. "daily_report"
	CronExpr  string    `bson:"cron_expr"`  // e.g. "0 9 * * *"
	UpdatedAt time.Time `bson:"updated_at"`
}

Now changing a schedule is a single database update, no deploy involved:

db.schedules.updateOne(
  { _id: "daily_report" },
  { $set: { cron_expr: "0 8 * * *", updated_at: new Date() } }
)

But data alone is not enough. Something has to notice the change and reconfigure the running scheduler. You have two ways to make that happen, and the difference between them is the whole point of this post.

Polling versus reacting

The simple approach is to poll: every minute, the scheduler re-reads the collection and rebuilds its timers if anything changed. This works, and for many systems it is entirely adequate. Its weaknesses are latency and waste. A change takes up to a full poll interval to take effect, and the vast majority of polls find nothing changed and were pure overhead. Shorten the interval to cut the latency and you increase the waste. It is a dial with no good setting, only less-bad ones.

Change streams remove the dial. Instead of asking "did anything change?" on a timer, the scheduler subscribes to a live feed and is told the instant something does. A schedule edit propagates in near real time, and when nothing changes, the scheduler does nothing and costs nothing. You get lower latency and less work at the same time, which is unusual enough to be worth the extra care the mechanism needs.

What a change stream is

A change stream is a live, ordered feed of the writes happening to a collection. Under the hood it reads the same replication log the database uses to keep replica set members in sync, so it sees inserts, updates, and deletes as they are committed, in order, with no polling. Your daemon opens the stream once and then blocks, receiving an event each time a schedule document changes.

// Watch the schedules collection. Block until an edit arrives, then apply it.
stream, err := coll.Watch(ctx, mongo.Pipeline{})
if err != nil {
	return fmt.Errorf("open change stream: %w", err)
}
defer stream.Close(ctx)

for stream.Next(ctx) {
	var event struct {
		FullDocument Schedule `bson:"fullDocument"`
	}
	if err := stream.Decode(&event); err != nil {
		logger.Warn("decode change event", logger.Err(err))
		continue
	}
	scheduler.Reload(event.FullDocument) // rebuild this job's timer
}

When an operator runs that one-line update, the daemon receives the event within moments and rebuilds only the affected timer. No redeploy, no restart, no poll interval to wait out. The schedule is now genuinely live data.

Change streams require a replica set, because they read the replication log, which a standalone server does not maintain. Any production MongoDB deployment is already a replica set, so this is rarely a new requirement, but it is worth knowing before you reach for the feature on a local single-node setup where it will not be available.

The part that makes it reliable: resume tokens

A naive change stream has a gap. If the connection drops, from a network blip, a failover, a brief restart, and you simply reopen the stream, you resume from now and silently miss every change that happened while you were disconnected. A schedule edited during a thirty-second reconnect would be lost, and the daemon would keep running the old timing while the database says otherwise. That divergence is exactly the bug this whole design was meant to prevent.

Change streams solve it with a resume token. Every event carries a token marking its position in the feed. Persist the latest token you have processed, and after a disconnect, reopen the stream starting from that token instead of from now. The database replays every change you missed, in order, and you catch up exactly where you left off.

// Resume from the last processed position so a reconnect misses nothing.
opts := options.ChangeStream()
if resumeToken != nil {
	opts.SetResumeAfter(resumeToken)
}
stream, err := coll.Watch(ctx, mongo.Pipeline{}, opts)
// ... after handling each event, persist stream.ResumeToken() durably

The discipline is to store the token after processing each event, durably enough to survive the daemon restarting. On reconnect, load it and resume from it. With that in place, the stream is not just fast, it is gap-free across the disconnects that are inevitable in any long-running consumer. The token is what turns a convenient feature into a reliable one.

One caveat: resume tokens are only valid while the change still exists in the replication log, which has a finite size. If the daemon is down long enough that the oldest unprocessed change has already rolled off the log, the token becomes invalid and the stream cannot resume from it. Handle that case by detecting the invalid-resume error and falling back to a full reload of the collection, which re-syncs the scheduler to the current truth. It is the same reconcile-from-scratch path you would want on first startup anyway.

Applying the change without dropping a beat

Receiving the event is only half the job. Applying it has to be careful, because you are mutating a running scheduler. Rebuild the timer for the changed job specifically, rather than tearing down and recreating every timer, so an edit to one schedule does not disturb the others or risk a double fire during the swap. Cancel the old timer for that job id, install the new one from the updated cron expression, and leave every other job untouched. Keep the reconfiguration idempotent, so that applying the same event twice, which resume-from-token can cause at a reconnect boundary, lands the scheduler in the same state as applying it once. That idempotency is what makes the at-least-once nature of the resumed stream safe.

When to keep it simple

Change streams are the right tool when schedule changes are frequent enough that redeploys are a real drag, or when the latency of a change taking effect actually matters. If your schedules almost never change, the operational win is small, and a plain restart-to-reload, or a slow poll, is less to build and less to break. Do not add a live event consumer, resume-token persistence, and invalid-token fallback to a system where someone edits a schedule twice a year.

The deeper principle generalizes past cron. Any configuration that changes for operational reasons rather than code reasons wants to live in a data store the running system watches, so that changing it is an edit rather than a release. Schedules are a clean example because the win is so concrete, a timing change that used to be a deploy becomes a one-line update, but the same pattern fits feature flags, routing rules, and rate limits. Store the config as data, watch it with a change stream, and handle reconnects with a resume token. That trio is what makes live configuration trustworthy instead of merely convenient.