Field note: scheduled jobs need catch-up rules

A cron expression says when work should start under normal conditions. It does not say what to do when that time passes without a start.

After an outage, should the system execute every missed minute, only the latest daily report, one reconciliation covering the whole gap, or nothing without operator approval? Starting all missed work can be as damaging as skipping it.

The Kubernetes CronJob documentation exposes startingDeadlineSeconds, concurrency policies, suspended schedules, time zones, and limits on how the controller handles missed start times.

My field note puts the missing product contract beside those controls: define the data window, consequence of lateness, safe replay shape, duplication guard, observable skip state, and human repair path.

Scheduled work is reliable only when the team has designed the hours in which it did not run.

Name the intended fire time

Every run should retain the schedule time it represents, timezone, schedule version, and data window instead of using actual container start as its only identity.

I would pressure-test that decision with four questions:

• Which instant was expected?
• Which timezone owns the rule?
• What data window belongs to it?
• Can schedule changes reinterpret history?

The failure mode here is keying work only by the moment a worker happened to start. In billing, reports, reminders, cleanup, reconciliation, imports, model refreshes, backups, retention, notifications, and operational automation where a scheduler outage, clock issue, long prior run, deployment, pause, or concurrency policy can skip expected executions, that can hide the exact boundary a reviewer or teammate needs to understand. My working artifact would be an intended-run identity schema. I want it close enough to the implementation that it can change the work, not created afterward to decorate the story.

The result I would look for is reproducible detection of missing and duplicate windows. That is a narrower claim than saying the whole system improved, but it is also one I can verify and defend.

In practice, I would put an intended-run identity schema beside the question “Which instant was expected?” before the first implementation review. The next pass would use “Which timezone owns the rule?” to test the boundary, then “What data window belongs to it?” to expose the state most likely to be missed. I would keep “Can schedule changes reinterpret history?” for the release check because it asks whether the decision still holds outside the ideal path. The work is ready to move when the artifact can explain the choice and the observed result supports reproducible detection of missing and duplicate windows.

Define the lateness deadline

The useful window should come from product consequence—freshness, payment cutoff, legal retention, user communication, or downstream dependency—not an arbitrary scheduler default.

The practical review starts here:

• When does the result lose value?
• Can late output harm users?
• Which downstream deadline exists?
• Who approves an override?

Those questions keep allowing unlimited catch-up because completion seems better than absence from becoming the default. I would capture the decision in a per-job starting-deadline rule, then use it while the work is still cheap to change. For scheduled work whose absence and delay are handled intentionally, the artifact should make ownership, constraint, and next action visible without requiring a private explanation.

Success would look like late work executed only while its promise remains valid. If I cannot point to that evidence, I have a direction, not a finished decision.

The implementation move is to make a per-job starting-deadline rule part of the working surface. I would use it to answer “When does the result lose value?” while scope is still flexible, and “Can late output harm users?” before code or content becomes expensive to unwind. During QA, “Which downstream deadline exists?” and “Who approves an override?” become concrete checks rather than discussion prompts. That sequence turns scheduled work whose absence and delay are handled intentionally into something the team can operate and gives me a specific outcome to report: late work executed only while its promise remains valid.

Choose catch-up semantics

Skip all, run latest, coalesce a range, replay every occurrence, or require manual approval are distinct policies and should be selected per job.

Before implementation, I would answer:

• Does every interval matter?
• Can windows be combined?
• Can a newer result supersede old work?
• What scale follows a long outage?

The artifact is a catch-up policy enum with rationale. Its job is to expose the tradeoff early enough that design, engineering, support, or product can disagree with something concrete. The common trap is letting the scheduler's global behavior decide product meaning; it moves uncertainty downstream and makes the final interface carry a problem the system never resolved.

For me, the useful receipt is recovery matched to the operation's temporal semantics. That connects a missed-run policy that defines schedule time, lateness deadline, catch-up window, concurrency, idempotency, replay order, user consequence, manual repair, and evidence to an observable result instead of a process claim.

I would test this with one typical case and one boundary case. The typical case should make “Does every interval matter?” easy to answer. The boundary should force a decision about “Can windows be combined?” and “Can a newer result supersede old work?.” I would record both in a catch-up policy enum with rationale, including the part that stayed unresolved after the first pass. The final check, “What scale follows a long outage?,” is where the artifact earns its place: it either supports recovery matched to the operation's temporal semantics, or it shows exactly why another iteration is needed.

Set concurrency intentionally

Allow, Forbid, and Replace-like behavior changes duplication, overlap, lost progress, ordering, and load; long-running jobs need an explicit decision.

I would use these prompts during the working review:

• May runs overlap?
• Can a new run cancel an old one?
• Does ordering matter?
• What if runtime exceeds cadence?

If the team slips into choosing Forbid without noticing it creates missed starts, the product can still look complete while its operating rule stays ambiguous. I would make a concurrency-policy decision table the shared reference and keep it small enough to update as evidence changes.

The standard is known behavior when execution duration crosses schedule. That tells me whether the decision helped the product, not merely whether the document was completed.

The working sequence is small: draft a concurrency-policy decision table, review it against “May runs overlap?,” implement the narrowest useful path, and then return with evidence for “Can a new run cancel an old one?.” I would use “Does ordering matter?” to inspect product consequence and “What if runtime exceeds cadence?” to decide whether the result is stable enough to ship. This keeps choosing Forbid without noticing it creates missed starts visible as a known risk and makes known behavior when execution duration crosses schedule the release receipt rather than a hopeful conclusion.

Make each window idempotent

An intended time or data-window key should make retries, controller duplication, and manual replay converge on one consequential outcome.

I would pressure-test that decision with four questions:

• What identifies the logical run?
• Where is uniqueness enforced?
• Can partial output be resumed?
• Do downstream calls accept the key?

The failure mode here is using pod or attempt ID as business identity. In billing, reports, reminders, cleanup, reconciliation, imports, model refreshes, backups, retention, notifications, and operational automation where a scheduler outage, clock issue, long prior run, deployment, pause, or concurrency policy can skip expected executions, that can hide the exact boundary a reviewer or teammate needs to understand. My working artifact would be a scheduled-operation idempotency contract. I want it close enough to the implementation that it can change the work, not created afterward to decorate the story.

The result I would look for is one durable outcome per intended window. That is a narrower claim than saying the whole system improved, but it is also one I can verify and defend.

In practice, I would put a scheduled-operation idempotency contract beside the question “What identifies the logical run?” before the first implementation review. The next pass would use “Where is uniqueness enforced?” to test the boundary, then “Can partial output be resumed?” to expose the state most likely to be missed. I would keep “Do downstream calls accept the key?” for the release check because it asks whether the decision still holds outside the ideal path. The work is ready to move when the artifact can explain the choice and the observed result supports one durable outcome per intended window.

Bound backlog recovery

A long outage can produce a catch-up storm, so replay needs capacity budget, ordering, batching, priority, dependency checks, and pause control.

The practical review starts here:

• How many runs may start?
• Which window goes first?
• Can work be coalesced?
• What protects online traffic?

Those questions keep releasing every missed run simultaneously after recovery from becoming the default. I would capture the decision in a catch-up capacity plan, then use it while the work is still cheap to change. For scheduled work whose absence and delay are handled intentionally, the artifact should make ownership, constraint, and next action visible without requiring a private explanation.

Success would look like restored completeness without a second outage. If I cannot point to that evidence, I have a direction, not a finished decision.

The implementation move is to make a catch-up capacity plan part of the working surface. I would use it to answer “How many runs may start?” while scope is still flexible, and “Which window goes first?” before code or content becomes expensive to unwind. During QA, “Can work be coalesced?” and “What protects online traffic?” become concrete checks rather than discussion prompts. That sequence turns scheduled work whose absence and delay are handled intentionally into something the team can operate and gives me a specific outcome to report: restored completeness without a second outage.

Represent skipped work durably

A skipped run is an outcome with reason, intended time, consequence, owner, and resolution—not the absence of a Job object or log line.

Before implementation, I would answer:

• Why was it skipped?
• Was the deadline exceeded?
• Does a user-facing state change?
• Who closes the gap?

The artifact is a skipped-run status record. Its job is to expose the tradeoff early enough that design, engineering, support, or product can disagree with something concrete. The common trap is assuming no alert means no scheduled work was expected; it moves uncertainty downstream and makes the final interface carry a problem the system never resolved.

For me, the useful receipt is auditable negative space in automation. That connects a missed-run policy that defines schedule time, lateness deadline, catch-up window, concurrency, idempotency, replay order, user consequence, manual repair, and evidence to an observable result instead of a process claim.

I would test this with one typical case and one boundary case. The typical case should make “Why was it skipped?” easy to answer. The boundary should force a decision about “Was the deadline exceeded?” and “Does a user-facing state change?.” I would record both in a skipped-run status record, including the part that stayed unresolved after the first pass. The final check, “Who closes the gap?,” is where the artifact earns its place: it either supports auditable negative space in automation, or it shows exactly why another iteration is needed.

Provide controlled manual replay

Operators need a parameterized command or workflow that validates window, idempotency, current state, load, authorization, and preview before repair.

I would use these prompts during the working review:

• Which window will replay?
• Can it duplicate an outcome?
• What is the estimated scope?
• Who may approve it?

If the team slips into copying a container command and editing dates by hand, the product can still look complete while its operating rule stays ambiguous. I would make a dry-run and replay procedure the shared reference and keep it small enough to update as evidence changes.

The standard is safe repair with the same guarantees as normal execution. That tells me whether the decision helped the product, not merely whether the document was completed.

The working sequence is small: draft a dry-run and replay procedure, review it against “Which window will replay?,” implement the narrowest useful path, and then return with evidence for “Can it duplicate an outcome?.” I would use “What is the estimated scope?” to inspect product consequence and “Who may approve it?” to decide whether the result is stable enough to ship. This keeps copying a container command and editing dates by hand visible as a known risk and makes safe repair with the same guarantees as normal execution the release receipt rather than a hopeful conclusion.

Monitor schedule truth

Last success alone hides skipped windows; expected-versus-recorded runs, lateness, queue age, overlap, deadline misses, clock skew, and downstream freshness reveal actual reliability.

I would pressure-test that decision with four questions:

• Which expected window is absent?
• How late are starts?
• Is the controller healthy?
• Did output reach users?

The failure mode here is alerting only when the latest pod fails. In billing, reports, reminders, cleanup, reconciliation, imports, model refreshes, backups, retention, notifications, and operational automation where a scheduler outage, clock issue, long prior run, deployment, pause, or concurrency policy can skip expected executions, that can hide the exact boundary a reviewer or teammate needs to understand. My working artifact would be a schedule completeness dashboard. I want it close enough to the implementation that it can change the work, not created afterward to decorate the story.

The result I would look for is detection of work that never began. That is a narrower claim than saying the whole system improved, but it is also one I can verify and defend.

In practice, I would put a schedule completeness dashboard beside the question “Which expected window is absent?” before the first implementation review. The next pass would use “How late are starts?” to test the boundary, then “Is the controller healthy?” to expose the state most likely to be missed. I would keep “Did output reach users?” for the release check because it asks whether the decision still holds outside the ideal path. The work is ready to move when the artifact can explain the choice and the observed result supports detection of work that never began.

Drill scheduler outages

QA should pause controllers, suspend and resume jobs, skew clocks, exceed runtime, miss more than the controller limit, duplicate attempts, fail halfway, replay ranges, and verify product state.

The practical review starts here:

• Can missing windows be enumerated?
• Does policy choose correctly?
• Is backlog bounded?
• Can every outcome be reconciled?

Those questions keep testing the cron expression at the expected minute from becoming the default. I would capture the decision in a missed-run failure exercise, then use it while the work is still cheap to change. For scheduled work whose absence and delay are handled intentionally, the artifact should make ownership, constraint, and next action visible without requiring a private explanation.

Success would look like evidence that scheduled automation survives the schedule not firing. If I cannot point to that evidence, I have a direction, not a finished decision.

The implementation move is to make a missed-run failure exercise part of the working surface. I would use it to answer “Can missing windows be enumerated?” while scope is still flexible, and “Does policy choose correctly?” before code or content becomes expensive to unwind. During QA, “Is backlog bounded?” and “Can every outcome be reconciled?” become concrete checks rather than discussion prompts. That sequence turns scheduled work whose absence and delay are handled intentionally into something the team can operate and gives me a specific outcome to report: evidence that scheduled automation survives the schedule not firing.

What I would show in the work

The public version needs evidence from the work itself. For this topic, the first five artifacts I would reach for are:

• an intended-run identity schema
• a per-job starting-deadline rule
• a catch-up policy enum with rationale
• a concurrency-policy decision table
• a scheduled-operation idempotency contract

I would not publish all five at equal weight. One should orient the reader, one should reveal the hardest tradeoff, and one should prove the result. The others can live in a downloadable note or appear as supporting frames. That edit matters because a missed-run policy that defines schedule time, lateness deadline, catch-up window, concurrency, idempotency, replay order, user consequence, manual repair, and evidence becomes harder to understand when every process detail is treated as equally important.

I would also show one rejected direction. The useful version is specific: which option looked attractive, which constraint made it wrong, and what evidence supported the narrower choice. That gives an engineering manager something real to question and keeps the case study from reading like the final answer was obvious from the beginning.

# intent
daily-ledger / 2026-07-17 / America/Bogota
Data window [00:00,24:00), schedule version 5, expected at 01:00, deadline 03:00, idempotency key ledger:2026-07-17.
# decision
started 02:14 / catch-up=within-deadline
Prior controller outage, no newer run, Replace forbidden, estimated load safe, one operation claims the date window.
# receipt
completed 02:27 / cursor advanced
Rows, checksum, downstream handoff, lateness, source incident, next schedule, and controlled replay command are recorded.

Resource path

The practical follow-up I would build is a scheduled-job contract with timezone, cadence, intended fire time, starting deadline, catch-up policy, concurrency policy, idempotency key, data window, retry, late-result behavior, manual replay, skip reason, monitoring, owner, and disaster drill. I am treating that as a resource backlog item, not pretending the adjacent downloads below are the same artifact. The related cards cover useful pieces of the workflow today; this specific file should only be published when its examples, fields, and instructions are complete.

The first version should stay concise: context, constraint, decision, evidence, owner, and follow-up. Its value would come from helping someone repeat this exact review, not from adding another generic PDF to the site.

Review checklist

The article-specific review questions are:

• Which instant was expected?
• When does the result lose value?
• Does every interval matter?
• May runs overlap?
• What identifies the logical run?
• How many runs may start?
• Why was it skipped?
• Which window will replay?
• Which expected window is absent?
• Can missing windows be enumerated?

I would add two editorial checks before publishing: can a recruiter find the point in the first minute, and can an engineer trace at least one claim to an implementation or production receipt? If either answer is no, the article needs another edit.

Implementation notes

For scheduled work whose absence and delay are handled intentionally, I would write the implementation note before polish. It would name the changed surface, source of truth, owner, failure boundary, and verification path. Those details prevent the principle from floating above the actual code or operational workflow.

The proof signals I care about are specific to this article:

• restored completeness without a second outage
• auditable negative space in automation
• safe repair with the same guarantees as normal execution
• detection of work that never began
• evidence that scheduled automation survives the schedule not firing

I would choose two or three of those signals for the first release rather than instrumenting everything. The strongest pair usually combines one direct behavior check with one operating check: a route and a data query, a keyboard path and a support state, a handler replay and a reconciliation result, or a migration count and a rendered screen.

The follow-up belongs in the note before shipping. It should say what remains temporary, what evidence would trigger another pass, and who owns that decision. That is how the first version stays intentionally narrow without making the boundary invisible.

Case-study packaging

I would structure the case-study version around the four visual lessons already established:

• Recovery begins by converting a missed time into a product decision.
• Lateness changes whether the same work is still useful.
• Different jobs need different catch-up semantics.
• The run receipt starts with intended time, not container start.

The opening frame explains the product pressure. The middle two show the decision moving through the system. The last frame is the receipt: what was checked, what held, and what remained unresolved. That order lets the reader move from product judgment into implementation detail without reconstructing the whole project first.

I would include one caveat tied to billing, reports, reminders, cleanup, reconciliation, imports, model refreshes, backups, retention, notifications, and operational automation where a scheduler outage, clock issue, long prior run, deployment, pause, or concurrency policy can skip expected executions: a data limit, rollout boundary, unsupported state, external dependency, or result that is still directional. A precise caveat makes the evidence easier to trust because it shows where the claim stops.

The final test is whether the page creates a better conversation. If the artifact helps someone ask a sharper question about product judgment, implementation detail, or release proof in a live interview, it belongs in the story.

Interview angle

In an interview, I would explain this through a missed-run policy that defines schedule time, lateness deadline, catch-up window, concurrency, idempotency, replay order, user consequence, manual repair, and evidence. The story should start with the product pressure, then move into the system constraint, the artifact, and the proof. That order keeps the answer grounded. It also gives the interviewer several places to go deeper: data, frontend architecture, design systems, support, migration, accessibility, or release process.

The strongest version of the answer includes a tradeoff. I want to be able to say what I chose, what I left alone, and how I knew the work helped. That is more credible than presenting every project as a clean win.

The hiring signal

A missed-run policy is a hiring signal because it shows I can find the product consequence inside scheduler configuration and connect time, concurrency, idempotency, recovery, and operations.

That is the level I want this site to communicate. The work should show taste, but it should also show operating judgment. It should make me look like someone who can enter a real product system, understand the messy middle, ship the useful version, and leave enough proof for the next person to trust it.