At what scale does a webhook-driven SaaS architecture collapse and how should it be redesigned before that happens? We have 8 backend engineers on AWS. Current volume is 50 events/sec growing 3x/year. Considering Kafka, AWS EventBridge, or SQS+SNS. Need to handle 10x current load within 6 months.

Council notes

Evidence boundary

Observed from your filing

• At what scale does a webhook-driven SaaS architecture collapse and how should it be redesigned before that happens? We have 8 backend engineers
• on AWS. Current volume is 50 events/sec growing 3x/year. Considering Kafka, AWS EventBridge, or SQS+SNS. Need to handle 10x current load within 6
• months.

Assumptions used for analysis

• Current webhook architecture is synchronous fan-out where the producer directly calls each subscriber endpoint, creating O(subscribers × events) HTTP calls
• The 8-engineer team can allocate 2-3 engineers to the Kafka migration while maintaining feature velocity with the remaining 5-6
• AWS infrastructure and networking costs are not the binding constraint — the ~$12k/year Confluent Cloud cost is within budget
• The 3x annual growth rate continues, meaning the system must handle ~1,500 events/sec within 2 years and ~4,500 events/sec within 3 years
• Downstream webhook consumers can tolerate 5-50ms additional latency introduced by Kafka buffering
• sla requirements defaulted: not specified (not_addressed)
• deployment model defaulted: not specified (not_addressed)
• observability state defaulted: not specified (not_addressed)

Inferred candidate specifics

These details were introduced by the Council during analysis. They were not supplied in your filing.

• Migrate to Apache Kafka via Confluent Cloud to handle the 500 events/sec target within 6 months. At 50 events/sec, direct webhook fan-out works. At 500 events/sec, webhook retry storms, endpoint timeouts, and cascading failures collapse the architecture — the dispatcher becomes the bottleneck. Kafka's partitioned commit log absorbs bursts durably, decouples producers from consumers, and scales horizontally via consumer groups. Use Confluent Cloud (not self-managed Kafka) to keep operational burden manageable for 8 engineers. Estimated cost: ~$12,000/year. Key failure mode: operational complexity still risks overwhelming the team — partition management, consumer group rebalancing, and offset tracking require dedicated learning investment. Mitigate by starting with a small number of partitions (8-16) and expanding only when throughput demands it. Second failure mode: under-provisioning the cluster leads to latency spikes under burst loads. Size for 2x the 500 events/sec target (1,000 events/sec peak capacity) to absorb growth and burst traffic. Retain webhooks as the delivery mechanism to downstream consumers but buffer through Kafka, converting synchronous fan-out into async consumer pulls.
• Deploy a Confluent Cloud Basic cluster in the existing AWS region with 8 partitions on a single topic, configure a proof-of-concept producer to mirror 10% of current webhook traffic (5 events/sec) into Kafka, and measure end-to-end latency from produce to consumer webhook delivery over 72 hours to validate the latency impact before committing to full migration
• b002 had the highest confidence at 0.85, survived 3 rounds of adversarial debate (split twice, then strengthened), named specific failure modes and cost thresholds, and provided concrete architectural guidance. b003 raised a valid latency concern but functioned as a critique (0.40 confidence) rather than an alternative architecture. b006's EventBridge proposal was interesting but lacked specificity and concrete failure mode analysis. b004 and b005 were empty.
• Deploy Confluent Cloud Basic cluster in existing AWS region, create primary event topic with 8 partitions and 7-day retention
• Build shadow-mode producer that mirrors 10% of live webhook traffic to Kafka topic; measure P50/P95/P99 end-to-end latency over 72 hours
• Audit all downstream webhook consumers for hard latency SLA requirements — document which endpoints require sub-100ms delivery vs. which tolerate seconds of delay
• Build consumer service that reads from Kafka and delivers webhooks with retry/DLQ logic, replacing direct fan-out
• Set up alerting on consumer lag, partition rebalance events, and producer error rates in Confluent Cloud dashboard

Unknowns blocking a firmer verdict

• Whether the 8-engineer team has sufficient Kafka expertise to execute migration in 6 months — Confluent Cloud reduces but does not eliminate the learning curve
• b003's latency concern is valid: Kafka buffering may add 5-50ms latency depending on configuration, and it's unclear whether downstream webhook consumers have hard sub-100ms SLA requirements that would be violated
• b004 and b005 were empty branches at 0.50 confidence — unclear what positions they would have represented, leaving potential alternatives unexplored
• The $12,000/year cost estimate is synthetic — actual Confluent Cloud pricing depends on throughput, retention, and connector usage that weren't specified
• Whether EventBridge might actually be sufficient at this scale (500 events/sec is well within its limits) with lower operational burden — this alternative was not thoroughly stress-tested in debate

Operational signals to watch

Branch battle map