Event sourcing, CQRS, and DataFrames

The event log

Event sourcing stores every change as an event: a small timestamped record added to the end of a log. Events are immutable; a later event can override an earlier one, but nothing is edited. The log is the source of truth, and current state is worked out from it. It can live in any database, or in a broker like Kafka.

Commands become events

A user's request, like "book seat 4", is a command. It is checked first; only if it is valid is it stored as an event, named in the past tense: "seat was booked". So the log holds only valid events, and no view may reject one. Unlike fact table rows, events vary in shape, and their order matters.

CQRS read models and benefits

CQRS keeps read-optimized materialized views apart from the log. Benefits:

• Events show intent, not just changed rows.
• A buggy view is fixed by fixing the code and rebuilding it.
• A wrong event is undone by a later one.
• The log is an audit trail.
• Appends are fast; views catch up.

Event sourcing pitfalls

A rebuild must give the same result every time, so never fetch live data, like today's exchange rate. A rebuild must not repeat side effects, like emails. Erasing personal data is hard in an immutable log; keep it outside events, or crypto-shred it by deleting its key. Every view must handle events in log order.

DataFrames and arrays

A DataFrame (Pandas, Spark, Flink) looks like a table and is changed step by step with commands; its join is called merge. It is used in analytics and finance, not OLTP. It helps turn rows into a numeric, often sparse matrix for ML. One-hot encoding turns a category into one 0/1 column per value. Array databases store big scientific grids.

Remember

• Events are immutable; fix mistakes with new events.
• Check the command, then write a past-tense event.
• Views can always be rebuilt from the log.
• Rebuilds must be repeatable and must not resend emails.
• DataFrames turn tables into numeric matrices for ML.