I've been working on production systems for a few years now. These are the things I keep coming back to—learned from real incidents, broken deployments, and 3 AM pages.
1. Boring beats clever
A predictable system that runs without paging anyone at 3 AM is worth more than an elegant architecture that needs a specialist to debug. When something breaks, uptime matters more than architectural purity.
2. Automate the second time
First time I do something manually, I'm learning how it works. Second time, I automate it. Automating too early just encodes assumptions I haven't tested yet.
3. Observability isn't optional
If a system can't explain what it's doing, it's not finished. Metrics, structured logs, traces—these aren't nice-to-haves. They're core deliverables, same as the feature itself.
4. Small changes, often
Large deployments carry large risks. Each change should be small enough to understand in code review, safe enough to deploy on Friday, and possible to rollback within minutes if it breaks.
5. Everything is a trade-off
There's no universal best practice. The right answer depends on team size, budget, timeline, and how much risk you can tolerate. Engineering judgment is knowing which constraints matter most in your situation.
6. Incidents are investments
Every outage shows you a gap you didn't know existed. Blameless post-mortems, actionable follow-ups, shared learning—incidents can be the highest-value engineering work you do if you treat them right.
7. Share what you know
A senior engineer who hoards context is a liability. Document your runbooks, share on-call rotations, write code anyone on the team can deploy. The goal is making yourself unnecessary, not indispensable.
8. Ship it
Analysis paralysis kills more projects than bad decisions. Make the best call with what you know, commit to it, measure the outcome, adjust. Feedback loops matter more than waiting for perfect information.
These change as I learn more. Each incident, retrospective, and architecture review sharpens them. What matters is getting it right repeatedly, under pressure, with a team that's counting on you.