Question 1

What is a message queue, and which problems of direct service-to-service calls does it solve?

Accepted Answer

A buffer between services that stores messages from a producer until a consumer is ready to process them, enabling asynchronous communication. Direct calls create tight coupling, slow responses, failures that propagate across services, and trouble with traffic spikes. A queue lets services operate independently: better scalability, reliability, fault tolerance and decoupling.

Question 2

Show the order-to-email flow and explain why the order still succeeds if the email service is down.

Accepted Answer

Customer places order → Order service → Message queue → Email service

The order service drops the email request into the queue and returns. The queue holds the message until the email service is available; even if that service is temporarily down, the order placement has already succeeded and the message is safe in the queue.

Question 3

Producer, consumer, publisher, subscriber: who is who?

Accepted Answer

Producer (also called publisher) creates and sends messages to the queue. Consumer (also called subscriber) retrieves and processes them.

Question 4

Synchronous vs asynchronous processing: when do you use each?

Accepted Answer

Synchronous (immediate response required before continuing, e.g. updating inventory after a purchase): financial apps, security-critical operations, database transactions, strict dependencies.

Asynchronous (queued and processed later without blocking, e.g. confirmation emails, triggering delivery): background processing, notifications, analytics pipelines, event-driven architectures, broadcast messaging.

Question 5

Queues are FIFO by default. What are the three processing variants?

Accepted Answer

- Strict order: A failed message halts the queue until it succeeds; nothing behind it is processed
- Unordered: A failed message is skipped and the next one is processed
- Priority queue: Each message carries a priority; higher priority is processed first

Question 6

Explain the publisher-subscriber model.

Accepted Answer

Multiple publishers send messages; multiple subscribers consume them independently; publisher and subscriber never talk directly.

1. Publishers send messages to the queue.
2. The queue stores them.
3. Subscribers consume independently.
4. The queue monitors subscriber health.
5. Failed deliveries are retried automatically.

Question 7

What are the key factors to handle in pub-sub: message order, consumption, poison messages, duplicacy?

Accepted Answer

- Message order: Maintained with FIFO or priority queues depending on implementation
- Message consumption: Subscribers consume independently, so processing order may vary
- Poison message: A message that fails repeatedly due to invalid data, a bad request or a consumer bug
- Duplicacy: Successfully processed messages must be removed so they are not processed twice

Question 8

What is a dead letter queue and why does it exist?

Accepted Answer

A secondary queue that stores messages which still fail after multiple retries, instead of retrying them forever.

Producer → Main queue → Consumer → fails → retry → retry → Dead letter queue

Purpose: store permanently failed messages, keep poison messages from blocking the main queue, support debugging, allow manual inspection and re-processing.

Question 9

Where do message queues fit, and where do they not?

Accepted Answer

Fit: asynchronous operations (emails, notifications, reports), decoupling services (analytics pipelines), load balancing between services, deferred or scheduled work (periodic report generation).

Do not fit: synchronous, low-latency or acknowledgement-dependent workflows where the caller must know the result immediately (payment confirmation, inventory check before checkout).

Question 10

What is a fault, and why classify them?

Accepted Answer

A fault or error is the system failing to perform as expected or deviating from intended behaviour; it hurts reliability, availability, performance and data integrity. In a distributed system failures can occur at any layer (servers, networks, databases, applications), so knowing the categories lets you detect, tolerate and recover: less downtime, more fault tolerance, less data loss.

Question 11

Hardware failures: causes, characteristics, mitigation.

Accepted Answer

Causes: disk full or crashed, server failure, damaged network cable, power supply failure, overheating, database hardware compromise.

Characteristics: random and unpredictable; components wear out; can cause complete outages and data loss without redundancy; usually need replacement or failover.

Mitigation: data replication, server redundancy, RAID storage, backup power (UPS), hardware monitoring, automatic failover.

Question 12

Software failures: causes, characteristics, mitigation.

Accepted Answer

Causes: poor or unoptimized code, unhandled exceptions, missing edge cases, configuration errors, deployment failures, merge conflicts, performance bottlenecks.

Characteristics: usually deterministic and reproducible, often caught in testing, easier to fix than hardware; generally preventable.

Mitigation: code reviews, unit and integration tests, proper exception handling, CI/CD, automated testing, performance optimization, configuration validation.

Question 13

Human errors: examples, characteristics, mitigation.

Accepted Answer

Examples: wrong configuration changes, accidental data deletion, wrong deployment procedure, misconfigured servers, incorrect database updates, permission mismanagement.

Characteristics: hard to predict, cannot be eliminated, can affect many components, usually caused by manual processes.

Mitigation: automate repetitive tasks, role-based access control (RBAC), standard operating procedures, change approval, user-friendly interfaces, training and documentation.

Question 14

List the best practices for fault tolerance.

Accepted Answer

- Design with redundancy and replication.
- Continuously monitor infrastructure and applications.
- Automate deployments and operational tasks.
- Test comprehensively before production releases.
- Maintain robust backup and disaster-recovery strategies.
- Restrict access with role-based permissions.
- Document procedures and train regularly.

Anticipate all three categories and put prevention, detection and recovery in place for each.

Question 15

Monitoring vs observability: what is the difference?

Accepted Answer

- Monitoring: Tracks predefined metrics, logs and alerts | Observability: Understands internal state from metrics, logs and traces
- Monitoring: Answers "Is something wrong?" | Observability: Answers "Why is something wrong?"
- Monitoring: Detects failures and abnormal behaviour | Observability: Diagnoses root causes
- Monitoring: Focuses on known problems | Observability: Surfaces unknown, unexpected problems

Typical monitoring metrics: CPU, memory, API response time, error rate, server availability. Observability matters most in distributed systems where a request crosses many services and must be traced end to end.

Question 16

What are the three outputs observability relies on?

Accepted Answer

Metrics (numbers over time), logs (event records), traces (a request's path across services). Together they let engineers investigate failures, find bottlenecks, trace requests and explain why an issue occurred.

Question 17

What does API monitoring track?

Accepted Answer

1. Throughput: requests processed per period, e.g. 10,000 requests per minute (RPM); shows load, capacity and when to scale.
2. Error monitoring: failed requests by status family (2xx, 3xx, 4xx, 5xx); alert when the error rate exceeds a threshold; catches deployment issues.
3. Health checks: confirm the service is up (HTTP 200 OK); feed load balancers and automated recovery.
4. Latency: time to respond; directly affects user experience; reveals bottlenecks; guides capacity planning.

Question 18

Why use latency percentiles instead of the average?

Accepted Answer

A few very slow requests barely move the average but ruin those users' experience. Percentiles expose them: P90 = 200 ms means 90% of requests finish within 200 ms and 10% take longer; P95 leaves only 5% slower; P99 represents the worst case for almost all users. Production systems monitor percentiles for this reason.

Question 19

What does infrastructure (system) monitoring track?

Accepted Answer

1. CPU usage: spots overloaded systems and when to scale; alerts commonly at 80%.
2. Memory usage: prevents out-of-memory failures, detects leaks; consistently high usage suggests inefficiency.
3. Disk I/O: storage bottlenecks and slow database operations; high utilization raises response times.
4. Network I/O: traffic spikes, bottlenecks, inter-service communication issues; abnormal patterns may mean failures, attacks or misconfiguration.

Question 20

What makes monitoring actually useful? Talk about alerting.

Accepted Answer

Monitoring only pays off when paired with alerting on critical metrics: high CPU, high memory, rising error rates, service downtime, excessive latency, network anomalies. Well-designed alerts let teams respond before users notice, which means faster detection, quicker recovery and higher reliability.

Reliability & Operations

Reliability & Operations