In the previous post ( here ) I tried to describe the common transactional problems people deal with when it comes to communication via messages. In short - we need to dispatch or consume a message (or both) and store data in DB and we would like all those operations to be in one transaction. In order to understand where event sourcing comes from we need to look at how most of the modern databases work. I`ll use MS SQL as an example but with some variations it can apply to almost any relational db. If you look at the DB files you`ll find two types of files: DB file itself (.mdf in ms SQL) and a transactional log file(,ldf) Why we need two? Because MS SQL uses Write-Ahead Transaction Log pattern. If we simplify the process it looks about this: add transactions to commit log tail commit-log and build DB (projection) based on that data All writes go via commit log, all reads are hitting our projection. (Sounds like CQRS? Yep, that's it) Why does it matter? Because m...
A common unit of work in a distributed system involves at least two things: 1) Storing the result of work to DB 2) Notifying consumers about the changes The problem is - they often can't be done in the same transaction: Kafka, Redis, RabitMQ, AlmostAnyCloud messaging system - they all don't support XA transactions. ( And not because they are lazy to implement one, but that deserves a separate post ) Let's say we have a transaction scope opened(Tb) and we want to store some changes to DB and dispatch an event: Tb -> DB -> Message -> Tc Looks valid, right? If DB changes fail we will rollback DB transaction. If dispatching an event failed, we still rollback. Seems quite transactional, where is the problem ¯\(°_o)/¯? Problem 1: Leaked notifications: on failure The problem is with the last part: Tc - Transaction commit (end) Why would it fail? It can be a network outage Our DB transaction can timeout while we are dispatching messages SQL Server might run ou...