Software Architecture: Interview & Big Picture

Monolith
┌──────────────────────────────┐
│ Order Service                │
│ Payment Service              │
│ Inventory Service            │
│ User Service                 │
│                              │
│ Shared: Database, Cache      │
└──────────────────────────────┘

Microservices
┌─────────────────┐    ┌─────────────────┐
│ Order Service   │    │ Payment Service │
└─────────────────┘    └─────────────────┘
        ↓                      ↓
   Order DB              Payment DB
        ↓                      ↓
    [Event Bus / Message Queue]
        ↓
┌─────────────────┐    ┌─────────────────┐
│Inventory Service│    │ Reporting Service
└─────────────────┘    └─────────────────┘

// Phase 1: Monolith with clear boundary
myapp/
├── order/          // Order context
├── payment/        // Payment context (candidate for extraction)
└── shared/

// Phase 2: Extract payment as library
payment-lib/       // Shared library
myapp/
├── order/
├── payment-client/ // Consumes payment-lib

// Phase 3: Separate service
payment-service/   // Independent service
myapp/
├── order/
├── payment-client/ // Makes gRPC calls to payment-service

// Note: Not: monolith splits cleanly. Usually messy.

Question: Can we handle 10x traffic without rewrite?
---
Monolith: Horizontal scale is hard (must scale whole thing)
Microservices: Can scale hot services (Payment) independently

Metric: Response time / CPU under peak load
Red flag: CPU 100%, response time degrades non-linearly

Question: How hard is it to deploy, monitor, recover from failure?
---
Monolith: Easy (one process, one database)
Microservices: Hard (N databases, N services, N failure modes)

Metric: Mean time to recovery (MTTR)
Red flag: Average incident = 2 hours to resolve

Question: Can each team work independently?
---
Monolith: Bottleneck (all teams in same repo)
Microservices: Autonomy (team owns service)

Metric: Deployment frequency, deployment size
Red flag: Can only deploy Friday afternoon, deploys affect multiple teams

Question: How much $ to run this system?
---
Monolith: 1 DB, 3 instances, 1 cache = low cost
Microservices: 5 DBs, 15 instances, 5 caches, load balancers, service mesh = 10x cost

Metric: Infrastructure cost per request / per user
Red flag: Cost grows faster than revenue

Question: Can we adopt new tech without rewrite?
---
Monolith: Stuck with first framework choice
Microservices: Each service can be different

Metric: Can we experiment with new language/framework?
Red flag: Forced to use 10-year-old framework cuz system is too coupled

# ADR-001: Monolith for MVP

## Status
Accepted

## Context
- Team: 3 engineers
- Timeline: 6 months to MVP
- Budget: Limited

## Decision
We will start with monolithic architecture in Go.

## Rationale
1. Simplicity: Monolith faster to develop, deploy, debug
2. Team size: <5 devs, not need independent scaling yet
3. Consistency: ACID transactions important for order correctness
4. Operational: Single deployment, single database

## Consequences
- Good: Developer productivity high, operations simple
- Bad: Scaling limited to horizontal (add instances), will need refactor later
- Ugly: If domain grows, may become tightly coupled

## Alternatives considered
- Microservices: Overkill at this stage, high operational overhead
- Serverless: Good for certain functions, but complex for persistent data

## Revisit date
2025-Q2 (when traffic > 1000 req/sec or team > 5)

User → Ask: Peak QPS? Delivery latency? Reliability requirement?
Assume:
- 10M notifications / day
- 100K QPS peak
- < 5 sec delivery
- 99.9% reliability
- Channels: Email, SMS, Push

Naive: Service receives notification request → immediately send
Problem: If Email API is slow, blocks other notifications

Solution: Async queue

Request
  ↓
Validation
  ↓
Enqueue (to RabbitMQ / Kafka)
  ↓ (async)
  Workers (Email, SMS, Push)
  ↓
Retry logic (if fails)

- Notification failed: retry (exponential backoff)
- Duplicate: idempotent key (ID)
- Rate limit: token bucket
- Persistence: store in DB before sending (for replay)

- Multi-region: send region-local queue
- Partitioning: by user ID
- Circuit breaker: if Email API down, fail fast

User → Select payment method → Charge → Verify → Confirm
       ↓           ↓              ↓        ↓        ↓
     Validate   Store method  API call   Check    DB save
               temporarily

Must ensure: Order confirmed ⟺ Payment captured
Solution: Saga pattern or 2-phase commit

Saga (recommended):
  1. Order service: create order (PENDING)
  2. Payment service: charge card (via async event)
  3. Order service: if success, change to CONFIRMED; else FAILED
  4. Compensating transaction: if charge fails, release order

If network fails mid-charge, might retry.
Must not double-charge.

Solution: Idempotent key (orderID as key)
- First attempt: charge($100, key=order123) → success
- Retry: charge($100, key=order123) → returns same result (idempotent)

- Never store card details (PCI compliance nightmare)
- Tokenize: Card → Payment Gateway → Token
- Pass token in subsequent requests

User A types "hello"  → Broadcast to User B (< 100ms)
User B types "world"  → Broadcast to User A

Challenge: Concurrent edits

Without: User A changes position 0-5, User B changes position 0-3
→ Conflict! Who wins?

Solution: Operational Transform (OT) or CRDT (Conflict-free Replicated Data Type)

CRDT (simpler to understand):
- Each character gets unique ID (user_id + lamport_clock)
- Insert: character("h", id=user1.001)
- When sync, insert by ID order (always same order on all clients)

┌─────────────────┐    ┌──────────────────┐
│  Browser (User A)     │  Browser (User B)
└──────┬──────────┘    └────────┬─────────┘
       │                        │
       └────────────┬───────────┘
                    ↓
          WebSocket Connection
                    ↓
          ┌──────────────────┐
          │ Collaboration    │
          │ Server (Node.js) │
          └────────┬─────────┘
                   ↓
           ┌───────────────┐
           │ Redis (OT ops)│ (for real-time)
           │ Postgres (doc)│ (for persistence)
           └───────────────┘

- User offline: queue ops locally, sync when online
- Concurrent editing: OT algorithm handles it
- Persistence: save snapshot + ops log
- Undo/Redo: easy with op log

Strong Consistency ←→ Eventual Consistency

Strong: All reads see latest write (ACID)
        E.g., Bank transfer: money visible immediately everywhere

Eventual: Reads may be stale, but converge to latest (BASE)
         E.g., Instagram likes: count may lag by few seconds

Choose based on domain, not just preference.

- Financial systems: money must be accurate always
- Booking system: seat must be allocated atomically
- Inventory: stock count must be accurate
- Technique: Single DB with transactions, or distributed consensus (Raft)

- Social media: likes, comments can lag
- Recommendations: data can be stale
- Reporting: data doesn't need to be real-time
- Technique: Event-driven, async processing, caching

// Strong consistency (monolith + single DB)
BEGIN TRANSACTION
  INSERT INTO orders ...
  UPDATE inventory SET qty = qty - 1 WHERE product_id = ?
COMMIT

// All systems see order + inventory update at same time

// Eventual consistency (event-driven)
1. Order service: INSERT INTO orders → publish OrderCreated event
2. Inventory service: subscribes to OrderCreated → updates inventory async
   (If inventory write fails, retry later)

// Short window where order exists but inventory not updated
// But eventually converges

Phase 1: Identify pain
  → Which service most problematic?
  → Which bounded context should be separate?

Phase 2: Extract seam
  → Create clear interface between this service and rest
  → Add integration tests

Phase 3: Dual-write
  → New code writes to both old + new service
  → New code reads from new service (fallback to old if fails)

Phase 4: Migrate traffic
  → Route 5% traffic to new service
  → Monitor, increase to 100%

Phase 5: Decommission
  → After stable, remove old service + dual-write

Technical Debt ≈ Financial Debt
- You borrow now (ship fast)
- You pay interest later (every change is harder)
- Eventually, you can't pay (system unmaintainable)

// WRONG: Hardcoded logic to ship faster
func ProcessOrder(order *Order) {
    if order.Total > 100 {
        discount = 0.1
    } else if order.Total > 50 {
        discount = 0.05
    }
    // Couple months later: business says "loyalty members get 20% always"
    // Now code is wrong, must refactor
}

// RIGHT: Config-driven
type DiscountPolicy struct {
    Thresholds []struct {
        MinAmount  float64
        Discount   float64
    }
}

// To change: update config, not code

Debt: Code works, but unverified
Cost: Next person touches it, breaks something → cascading failures

Debt: Code exists, but "why"  unknown
Cost: Next person spends 2 days reverse-engineering

1. Clarify Constraints
   - Team size?
   - Timeline?
   - Scale?
   - Budget?

2. List Options
   - Monolith / Microservices
   - SQL / NoSQL
   - Sync / Async
   - Single DC / Multi-region

3. Evaluate Trade-offs
   - Complexity?
   - Cost?
   - Scalability?
   - Operational burden?

4. Make Decision
   - Choose option with best trade-off for constraints
   - Document in ADR (why, not just what)
   - Set revisit date

5. Revisit
   - Quarter / Half-year: is decision still valid?
   - Constraints changed? Refactor incrementally