Two bugs from a school ERP, and where validation actually belongs
Three and a half thousand schools sounds like a load problem. It mostly isn't. Schools use an ERP in a pattern so lumpy that averages are useless: admissions land in a six-week window, fee collection spikes on the same three days of the month across the entire customer base, and report cards all get generated in the week before results.
What that concentration really does is statistical. A bug with a one-in-ten-thousand trigger is invisible in testing and happens forty times in a fortnight when the whole country is doing admissions at once. Two of those taught me more about where to put validation than any amount of reading had.
The child who was admitted twice
A parent fills in an online admission form on a phone, on a patchy connection, and taps submit. Nothing appears to happen. They tap it again.
Two admission records, two admission numbers, two fee ledgers, one child. The school notices in about three weeks when the fee reminders go out in duplicate and an angry parent calls the office.
Everything about this is embarrassing in hindsight, but the interesting part is how many layers we'd thought were handling it.
The form disabled the submit button on click. That does nothing when the first request times out at the network layer and the page never transitions, which is exactly the situation a patchy connection produces. Client-side guards protect against impatience. They don't protect against uncertainty, and a parent who doesn't know whether the form went through is being perfectly rational when they resubmit.
The service layer checked for an existing student before inserting. Two requests arriving four hundred milliseconds apart both ran that check, both found nothing, both inserted. A read-then-write in application code is not a constraint, it's an optimistic wish, and under concurrency it fails precisely when the system is busiest.
Which leaves the database, and this is where it got genuinely difficult, because a unique constraint needs a key and there isn't an obvious one. Name and date of birth collide with siblings more often than you'd expect, especially with twins. Parent phone number is shared across siblings by design. Aadhaar isn't always available at admission time and legally can't be mandatory. Every natural key we proposed had a real counterexample somewhere in the customer base.
So we stopped trying to define what makes a student unique and defined what makes a submission unique instead. The form generates an idempotency key when it loads, sends it with the submission, and there's a unique index on it. A resubmission of the same filled form returns the original record rather than creating a second one. The database enforces it, so no amount of retrying, load balancing or double-tapping gets around it.
Then, separately, a soft duplicate detector: on insert, look for existing students with similar name, same date of birth or same guardian phone, and if any turn up, flag the record for the admissions office rather than blocking it. A human decides. That distinction between a hard constraint the system enforces and a soft signal a person acts on is the thing I took away from all of it.
The payroll calculation that lived in three places
The second one is less dramatic and cost far more time.
A teacher joins on the 18th of the month. Payroll needs to pay them a part month. Simple enough, and the school's accountant found that our salary slip, our payroll register and the preview shown in the HR portal gave three different figures for the same person.
Not wildly different. Tens of rupees. Which is worse than wildly different, because a large discrepancy gets reported immediately and a small one gets quietly corrected by hand every month until someone mentions it in passing.
The cause was that the same rule had been implemented three times. The frontend had a pro-rata preview so HR could see the effect before saving. The payroll service had the real calculation. The reporting module, which ran against a different data shape, had its own. All three were written from the same specification by three people, and they diverged on the questions the specification hadn't answered: whether the joining day itself is paid, whether you divide by calendar days or working days, and what a working day is when a school's holiday calendar is configurable per branch.
None of those questions had a right answer. They just needed one answer.
The fix was to delete two of the implementations. One function, in one place, that takes the employee, the period and the school's calendar and returns a breakdown. The frontend preview calls the same endpoint that payroll runs. The report reads what payroll stored rather than recomputing it.
Recomputation is the specific smell. If a number can be derived in more than one place in your codebase, it will eventually be derived differently, and the divergence will surface at the worst possible time to somebody who trusts you with their salary. Store the computed result with the inputs and the version of the rule that produced it, and let everything downstream read it.
That last part mattered when the labour rules changed and the calculation had to change with them. Old payslips still show what they showed. Recomputing history because the rule changed is its own category of disaster.
Where each layer belongs
Which brings me to the actual thesis, arrived at the hard way.
The database holds invariants: things that must never be true regardless of which code path ran, who was logged in, or what the client sent. Uniqueness, foreign keys, non-negative amounts, dates that must precede other dates. If your answer to "how do we prevent this" is a code review habit, it belongs here instead.
The service layer holds rules that need context the database doesn't have. Whether this user can admit a student to this branch. Whether the fee structure applies to this class this year. These are rules, and rules have exceptions, so they live somewhere a human can read them and an exception can be recorded.
The form exists to help the person filling it in. It is not a security boundary and it is not a source of truth. It should catch the typo before submission and say something clear, and nothing downstream should assume it ran.
The rule nobody follows
There's a coda to all this that took me longer to accept.
A school will want to admit a student without a date of birth, because the birth certificate is coming next week and admissions close on Friday. Make the field mandatory with no path around it, and you will not get correct data. You will get 01/01/1900, entered by an office administrator who has a queue of parents in front of her and a deadline that is more real than your validation rule.
We found thousands of those. Every one was a place where our schema said we had a date of birth and we didn't, which is strictly worse than a null, because a null is honest.
So the incomplete admission became a first-class state. Records can be saved with required fields missing, they're marked incomplete, they appear in a list the office works through, and certain operations stay blocked until they're resolved. The validation didn't get weaker. It moved from the moment of entry to the moment it actually mattered, and the data got more truthful as a result.
You can enforce a rule, or you can find out what people are actually doing. Enforcing a rule that fights the workflow gets you neither.