{"id":87,"date":"2018-11-03T17:47:03","date_gmt":"2018-11-03T17:47:03","guid":{"rendered":"http:\/\/www.webagam.com\/?p=87"},"modified":"2020-12-11T00:46:09","modified_gmt":"2020-12-11T00:46:09","slug":"87","status":"publish","type":"post","link":"https:\/\/webagam.com\/pages\/2018\/11\/03\/87\/","title":{"rendered":"Things to consider before adopting Event Sourcing"},"content":{"rendered":"
3<\/div> \r\n\t\t\t\t\t\t\t\t\t
\r\n\t\t\t\t\t\t\t\t\tClap\r\n\t\t\t\t\t\t\t\t\t<\/span>\r\n\t\t\t\t\t\t\t\t\t<\/div><\/div><\/div>

Event Sourcing is a pattern in which the application stores all change to the system as an event in an event store. These series of events are then run in an order to arrive at the current application state.
\nFor optimization, store the state periodically by running through the events. This is called a snapshot. Hence, the application can derive its state faster by applying events that have occurred after the snapshot to the state snapshot instead of deriving from the start.
\nSome examples,
\nRDBMS: All Relational Database uses transaction logs to store all the changes. The current state (event sourcing mentions this as a snapshot) is present in the table. It is also common to use \u201clog shipping\u201d, where the database ships transaction logs from one database to another, and data are restored in other end using these transaction logs.
\nThere is a lot to \u201cEvent Sourcing\u201d and did not plan to cover those in this post. This post assumes there is some basic understanding already known on event sourcing. This post focuses on challenges and some techniques to address those challenges.
\nEvent sourcing is one of the useful techniques especially when applied with CQRS and DDD.
\n[CQRS \u2013 Command Query Responsibility Segregation]
\n[DDD \u2013 Domain Driven Design]
\nHowever, it is hard to get it right or make everyone in your team understand to do it correctly.
\nBelow are some of the challenges I have come across when understanding & designing an application using \u201cEvent Sourcing\u201d. Please note there might be several ways and other challenges apart from these, but I only try to highlight the challenges that I faced and the measure that I took to tackle it.<\/p>\n

Things to ponder before adopting Event Sourcing,<\/p>\n

GDPR\/Security<\/strong><\/h4>\n

By theory in event sourcing \u201cevents\u201d are never deleted but only appended. Hence if the application handles customer data then it is possible that customers can ask to get these deleted, often called \u201cForget Me\u201d in GDPR.<\/p>\n

This can become a bit tricky with event sourcing. It is also said that in some cases the system needs to delete all traces of customer data (including backed up data in drives & tapes). The complexity to delete may differ from application to application, but in general, it is easier if the data is in a database.<\/p>\n

I am not a legal person and based on the domain the level of complexity may vary. Even if the application does not offer a direct way to delete the data, it is possible to run a SQL query to delete the data in the database. In the case of event sourcing deleting the data may be directly against the definition of event sourcing, remember event sourcing is only \u201cAPPENDING\u201d and the PAST events cannot be changed.<\/p>\n

Ways to handle it<\/strong><\/em><\/p>\n

    \n
  1. If we take regular snapshots and rely on events sourcing only for a few days then it may help to handle certain cases. Even if it is not immediate delete, it is possible to delete customer data on request. But this may be a constraint as we will be able to delete customer data only after certain days have passed. Also, snapshot state becomes a starting state of the system and the system state can be recovered by running events only from this snapshot \u201cSTARTING STATE”<\/li>\n
  2. REDACT is one way to handle if we use Apache Kafka. Apache Kafka supports a concept called \u201cLOG COMPACTION\u201c. With \u201cLog Compaction\u201d,\u00a0 events can be replaced by matching id. This way we can replace the customer details with a dummy value in the event store. Kafka provides a way to replace the message matching the primary key when the compaction runs.<\/li>\n<\/ol>\n

    You may read more on log compaction here, https:\/\/kafka.apache.org\/documentation\/#compaction<\/a><\/p>\n

    Schema Change<\/strong><\/h4>\n

    Data Model changes because of business changes and as our understanding of the domain mature. A traditional application where we store the last state of the application in the DB has a mature set of tools and well know practices to follow. Processes like configuration changes, database schema changes and many others have a good set of tools, guidelines and practices to follow.\u00a0 Similar to these events schema can change from version to version. Most Event Sourcing works on top of pub\/subtopic or on a log based streaming platform like Kafka. One main advantage of this model is that producers doesn\u2019t have to know consumers of the events.<\/p>\n

    This majorly helps as new consumers can come in or go out any time without affecting the producer.<\/p>\n

    Ways to handle it<\/strong><\/em><\/p>\n

      \n
    1. It depends on published the message type (JSON, XML, protocol buffer, or Objects). If it is JSON or XML, some technique used in API\/Service versioning can be applied here.<\/li>\n
    2. Avro is a most recommended tool by many but I am yet to use it. Hence I will not comment on it.<\/li>\n<\/ol>\n

      Development understanding & maturity<\/strong><\/h4>\n

      As for any design, event sourcing must have a commitment from the development team. It is difficult for the developer or any new joiners to understand and code correctly. Developers used to imperative coding find it more difficult as event sourcing is mostly functional.\u00a0 This one may take more time than you may think to make everyone understand.<\/p>\n

      Ways to handle it<\/strong><\/em><\/p>\n

        \n
      1. If event sourcing is for an existing application, then it is better to try out for one part of the subdomain before applying for all. This provides an opportunity to learn & to do it correctly for the next one.<\/li>\n
      2. If the plan is to use event sourcing for an existing application, then have a person who has already done it lead it from the front. More likely you may end up hiring a consultant than a full-time employee.<\/li>\n<\/ol>\n

        Operations<\/strong><\/h4>\n

        There are many tools and techniques for RDBMS. In some cases, even business people find it easier to query the database directly to pull a report to find out what is happening in the system. In addition to this, there are a rich set of tools available for activities such as monitoring, management, backups, reporting & transformation. Although the event sourcing concept been there for a while, it is gaining traction in recent years. Also, Event Sourcing is a concept, RDBMS or event store or log\/event streaming applications like \u201cApache Kafka\u201d still have to be used for achieving event sourcing.<\/p>\n

        Ways to handle it<\/strong><\/em><\/p>\n

        In the case of Kafka, then there are monitoring and management tools that can watch a topic and report details about it. This way it is possible to look at find slower consumers (projections and reactors). This may also be used to dynamically scale up or down projections and reactors for processing the events.<\/p>\n

        Beware of side effects of reactors & projections<\/strong><\/h4>\n

        The main test of event sourcing is events can be replayed from a point the application last processed to arrive at the current state. Hence different consumers (often referred to as projectors or reactors) can process the data at different times. It is also possible that these projectors or reactors raise more events.\u00a0 One of the advantages of an event management system (not just event sourcing) is consumers and producers don\u2019t have to know about each other.<\/p>\n

        When replaying events in a particular application it may raise more events and may need consumers of the events to handle it or to have a flag to stop raising events until a certain state.<\/p>\n

        For example, assume \u201cService A\u201d raises events for which service B is a consumer. Similarly \u201cService B\u201d processes the events and may raise more events for which \u201cService C\u201d is a consumer.<\/p>\n

        Say for any reason \u201cService B\u201d has lost its state completely and replays all the events raised by \u201cService A\u201d to arrive at its state. Then this may raise more events which can cause issues in Service C if there is no id column or if Service C is not idempotent.<\/p>\n

        Ways to handle it<\/strong><\/em><\/p>\n

        Projections are typically stateless and hence replaying the message may not have any side effects.<\/p>\n

        However, reactors maintain state and hence reactors will have to keep track of last processed events by primary key. This way reactors can ignore replayed events as these are already processed.<\/p>\n

        Is eventual consistency ok for the domain?<\/strong><\/h4>\n

        Often a common example used is “Bank Account Ledger” to describe as a real-world example of event sourcing. We all know that banks keep up all debits and credits.<\/p>\n

        We also know that if we apply for debit and credit on an available balance before that in the order we will arrive at the current available balance. However, in my experience bank maintains the current state i.e. Available Balance in a DB.\u00a0 If the customer reports an issue, then banks will run through the credit and debit to arrive at the available balance.<\/p>\n

        When two requests come at the same time for the same account to do a fund transfer, certainly the data have to be consistent and the request has to be processed in a transaction one after another.<\/p>\n

        At the back-end it may be \u201cEventually Consistent\u201d, but there are use cases where it has to be consistent. This can be tackled by making a \u201cWrite Command\u201d Synchronous and single-threaded. i.e. until the application persists the event and derives a state the response is not sent back.\u00a0 Using reactive programming, process manager, and keeping the write command \u201cSingle-threaded\u201d will help address this. \u00a0But in a distributed clustered environment where multiple application servers are running behind a load balancer, this becomes difficult to handle. Also, single-threaded means that a particular write command that handles an aggregate will have to be in a single server. In a clustered environment it is possible to make a write command single-threaded only within an instance of the application server. Multiple requests may still reach different servers at the same time and it is not possible to make the write command single-threaded.<\/p>\n

        Ways to handle it<\/strong><\/em><\/p>\n

          \n
        1. There are tools that specialized to handle such scenarios in a clustered environment.<\/li>\n
        2. Sharding is another approach i.e. although there are multiple application servers handling requests, request of the same action type is routed to the same application server.<\/li>\n
        3. Most of the applications do have databases and databases are good at handling these kinds of locking across transactions.<\/li>\n<\/ol>\n

          [Note: Alternatively we perform raising events and deriving state together before sending the response]<\/p>\n

          Below are the steps that we follow<\/p>\n

            \n
          1. On a write command after successful validation, create an event immediately.<\/li>\n
          2. Compute the state using that event in memory for the source application.<\/li>\n
          3. Persist the events to the event store and the state to the snapshot data store (DB).<\/li>\n
          4. This computed state is immediately used for the next \u201cWrite Command\u201d while the other reactors and projections use the persisted events.<\/li>\n<\/ol>\n

            In step 1) the state stored in the DB is used for successful validation.<\/p>\n

            I will keep this post updated as I learn more. Happy reading.<\/p>\n

            If you want to learn more about event sourcing there are very good videos on you tube.<\/p>\n

            Clap<\/h3>
            \"Clap\"Clap<\/a><\/div>
            \"Niranjan<\/span>Niranjan R.<\/a><\/span><\/span>\"Claps\"<\/span>Claps<\/a>(2)<\/span><\/span><\/span>
            <\/div><\/div><\/div>","protected":false},"excerpt":{"rendered":"
            3<\/div> \r\n\t\t\t\t\t\t\t\t\t
            \r\n\t\t\t\t\t\t\t\t\tClap\r\n\t\t\t\t\t\t\t\t\t<\/span>\r\n\t\t\t\t\t\t\t\t\t<\/div><\/div><\/div>

            3 Clap Event Sourcing is a pattern in which the application stores all change to the system as an event in an event store. These series of events are then run in an order to arrive at the current application state. For optimization, store the state periodically by running through the events. This is called a snapshot. Hence, the application… Continue reading “Things to consider before adopting Event Sourcing”<\/span><\/a><\/span><\/p>\n

            Clap<\/h3>
            \"Clap\"Clap<\/a><\/div>
            \"Niranjan<\/span>Niranjan R.<\/a><\/span><\/span>\"Claps\"<\/span>Claps<\/a>(2)<\/span><\/span><\/span>
            <\/div><\/div><\/div>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[13,14],"tags":[16,15,17],"class_list":["post-87","post","type-post","status-publish","format-standard","hentry","category-event","category-event-sourcing","tag-event-sourcing","tag-events","tag-kafka"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/sadP6e-87","_links":{"self":[{"href":"https:\/\/webagam.com\/pages\/wp-json\/wp\/v2\/posts\/87","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/webagam.com\/pages\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/webagam.com\/pages\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/webagam.com\/pages\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/webagam.com\/pages\/wp-json\/wp\/v2\/comments?post=87"}],"version-history":[{"count":6,"href":"https:\/\/webagam.com\/pages\/wp-json\/wp\/v2\/posts\/87\/revisions"}],"predecessor-version":[{"id":294,"href":"https:\/\/webagam.com\/pages\/wp-json\/wp\/v2\/posts\/87\/revisions\/294"}],"wp:attachment":[{"href":"https:\/\/webagam.com\/pages\/wp-json\/wp\/v2\/media?parent=87"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/webagam.com\/pages\/wp-json\/wp\/v2\/categories?post=87"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/webagam.com\/pages\/wp-json\/wp\/v2\/tags?post=87"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}