Cloudflare Just Killed Kafka? (K2 Exposed)

0:00 I went into Cloudflare's K2 announcement expecting CFKA with an orange logo and the road map lists CFKA client support as still to come. Meanwhile, your workers app just took an order.

0:10 Analytics, fraud detection, and email all need that event, and email is offline. With a direct call, email never sees that order. Cloudflare's writeup says so plainly. If consumers become

0:22 unavailable, events are dropped. K2 in public beta since October 1st is their fix. It's an event log built from files on R2 Cloudflare's object storage. So my first question was simple. When email

0:34 comes back, does it still get the order before launch? James Ross summed it up on X a named a penstyle log of records with a retention window. Basically, Cloudflare CFKA that's fair about the

0:45 shape. CFKA's own docs say events aren't deleted after they're read and K2 keeps that idea. Several readers, one shared log. But Cloudflare's blog explains why they didn't just run CFKA. Small slices

0:58 of machines, machines that don't stick around, and traffic that often crosses the public internet. Brokers guarding data on their own discs fit that badly. So K2 hands the job to R2. And that one

1:09 choice shapes both the good part and the cost. The cost has a figure about 1 second of produce latency at the 99th percentile, Cloudflare's own number. Where that second goes is the question

1:20 I'll come back to. By the end, you'll have followed one order through K2, watched an offline service catchup, and you'll know whether K2 fits your app today. Follow the order. Then your

1:30 worker hands it to K2 through a binding or over plain HTTP. You'd expect it to hit storage right away. It doesn't. Writes collect in memory for a moment next to everything else arriving at the

1:42 same time. Every write and read on object storage has a cost. So, Cloudflare's blog says K2 has to write complete files big enough to be worth it. Think of a warehouse packing orders.

1:52 You don't ship one sock per box where the picture breaks. K2 doesn't wait for a full box. It waits a short period, then packs whatever arrived. That box is called a segment file. K2 seals every

2:04 collected event into one segment and writes it to R2. Picture R2 as a long shelf. Each sealed box goes on the end, and nothing gets slipped in between. Each record also gets a number that only

2:16 ever goes up. It's offset. You might expect a separate coordination service handing those numbers out. Cloudflare says K2 doesn't need one. R2's atomic operations give it ordering and strictly

2:27 increasing offsets. Now, the part the CFKA label hides. In CFKA, brokers copy your data to each other and that copying keeps it safe. In K2, the shelf does that job. R2's durability page says it's

2:39 designed for 11 9 of annual durability. The line under it matters more. R2 only reports a write as successful once the data is persisted to disk. So once a box is on the shelf, it's safe. The service

2:51 in front is more packing table than vault. Back to your three services. Each gets its own subscription and each subscription tracks its own progress through the log. On the shelf, that's

3:00 three bookmarks. Analytics, fraud detection, and email read the same boxes, and no reader moves another's place. Email goes offline upstream. Nothing changes. Orders still get boxed.

3:12 Boxes still land on the shelf and the other two bookmarks move past them. Email's bookmark just sits there facing a growing stack. Then email comes back. Its bookmark is where it left it. It

3:23 reads the order. Nothing was dropped. That answers my first question. Going by how the consume docs describe subscriptions rather than anything measured. There's a clock on it, though.

3:33 Records stay on the shelf for the retention period. 7 days by default, 30 at most, 1 hour at the least. If email stays down longer than that, the oldest boxes leave the shelf before its

3:43 bookmark gets there. The same shelf helps a service you haven't written yet. A new subscription can start from earliest. The oldest record still kept and read forward. Each stream allows up

3:53 to 100 subscriptions, and each record can be up to about a megabyte. So far, this sounds airtight, but reading a box doesn't finish the job. When a worker pulls a batch, K2 gives it a lease, a

4:04 five-minute hold on those records, and never leases the same records to two workers at once. The worker acknowledges when it's done. If it sends back a no, a knack, the batch gets redelivered. The

4:15 docs call this at least once delivery. A record can show up more than once, so your email service could send the same receipt twice. And because K2 works in batches, there are no retries for a

4:26 single message. If one order fails and you knack, the whole batch comes around again. Build consumers that expect repeats. Key each email on the order ID and skip any you've already sent and

4:37 don't count on order across workers. The doc say K2 doesn't guarantee processing order between them and message keys are still a roadmap item. The producing side has its own trap. A batch right is all

4:49 or nothing. Every record lands or none do. But the send call doesn't throw. When K2 rejects a batch, it returns a result object and you check result.success yourself. Skip that check

5:01 and a rejected batch looks exactly like a good one until somebody asks where Tuesday's orders went. Now, that 1 second from the start at the 99th percentile, 99 writes in 100 finish

5:13 faster than about a second. I guess network distance would explain it. Where does it actually go? Cloudflare names two causes, object storage speed and waiting for the box. Each order sits on

5:24 the packing table until the short wait ends. Then the whole box goes to R2. Then your producer hears back. The blog doesn't spell out that last step, but the 1second figure points to it. To

5:34 check that reading, I went through Warpstream's docs. It's a Kafka compatible log on object storage, and it never acknowledges a produce request until the data is in object storage. Its

5:45 documented defaults give 500 milliseconds at the 99th percentile for produce and 900 end to end. Different system, same trade-off and not a head-to-head with K2. The hacker news

5:56 thread added a wrinkle. A user called E1G wrote that their rights stayed under a second, but endtoend delivery hit 2 and 1/2 seconds at the 95th percentile and 7 and 1/2 at the 99th. That's one

6:08 person's testing. Cloudflare's published figure covers producing only. In the same thread, Nikubi described there as K2's tech lead wrote that produce and endto-end latency run higher than local

6:19 disk systems like Kafka around a second at the 99th percentile. Cloudflare's answer on the road map is a lower latency express tier. Kafka clients sit on that same road map. Drop-in support

6:30 is planned with no date attached. Today you reach K2 through a workers binding or its HTTP API. If your team already runs Kafka consumers, nothing plugs in yet. money. Briefly, usage is free

6:43 during the beta on the workers paid plan with up to 10 gigabytes of storage per account. After that, Cloudflare expects 4 cents per gigabyte produced, 4 cents per gigabyte consumed, and 2 cents per

6:54 gigabyte per month retained. Those are expected rates, and they can change. On Hacker News, NNX pointed out the part that's easy to miss. Consuming is build per gigabyte, too. So, every extra

7:05 subscription adds cost. send 1 gigabyte of orders to three services and by my count that's 4 cents in and 12 cents out. Fan out is free on the whiteboard. Back to that order and the email service

7:16 that was offline. K2 boxes the order. The shelf keeps it and emails bookmark waits. If email returns inside the retention window, it gets the order. The price is about a second on the way in.

7:28 Maybe more end to end plus duplicates you have to handle. The rule I'd follow, use K2 when a consumer can wait a few seconds but can't miss an event. Set retention longer than your worst outage.

7:39 Make handler safe to run twice. And check result.success on every send. My call. If you're on workers and fanning orders out to analytics, fraud checks or email. K2 is worth trying in the beta.

7:52 If you need fast delivery or have CFKA clients to keep, wait and watch the express tier and the CFKA support. What's the longest one of your services has been down? and Wood 7 days of

8:02 retention have covered