I left the discussion of distributed application design a while ago. We covered some general architectural concepts. The CAP Theorem and how the ACID Principal and the new BASE principal resolve that. However, I didn’t really focus on any of the technologies that would help. I want to take a moment and diverge from our basic tutorials to analyzing messaging.
In this series, we’ll go through how messaging works as a general technology. With that understanding, we’ll continue into designing messaging workflows abstractly, including popular design patterns for messaging. This will lead us to message design. Messages need to be architected like any data. With the abstract covered, we’ll move testing and demonstrating the message structure and workflows in code. This will lead us to a further analysis of specific messaging architectures. I won’t commit to any specific examples now as that could change. We’ll use RabbitMQ and ActiveMQ as two of the implementations. The abstract examples will be using Java with an embedded messaging broker to simplify the examples.
So, what is Messaging? It’s a surprisingly complex questions. Fundamentally, it’s simply the concept of grouping data and sending that data to another process. This can take any form really. We’ve used Messaging since the dawn of time. The ancient process of writing on tablets to send to another city is Messaging. More contemporary examples are TCP/IP and HTTP. Both work by structuring data into packets and requests and creating messages. With both of these technologies, our systems are waiting for a response. This is synchronous Messaging. A message is sent, and the sending, or producing, system awaits a response, then another message is sent back by the recipient of the message in response.
The real complexity comes into play when we want to make the message exchange asynchronous. Asynchronous messaging is only possible by adding a system, commonly called a broker, in the middle to coordinate the communication. This broker awaits the Producer to send it a message. The Broker then holds onto that message until a Consumer asks for that message. Some brokers have a way to alert a Consumer of a new message. This is more of a specific implementation. We’ll focus on an abstract analysis for now.
The broker is a role in the architecture. Exactly how the broker stores messages, or how applications get messages, can vary widely. We’ll examine each of these in following articles. I wanted to bring it up here because often people focus on queues as the de facto method to store and organize messages. Remember that Messaging is simply packaging data to send to another system. In asynchronous messaging, a broker is simply the intermediary system that stores messages until a Consumer retrieves them. This means we can look at systems like caches, key/value stores, and even databases as brokers. In a way, we can break every system down to Messaging workflows. A Messaging workflow is effectively a stream of data. With the broker acting as a buffer. In a weird way, this means we can reuse battle-hardened designs from other technologies.
Let’s take the old reliable Messaging technology TCP/IP. I’ll save a complete analysis of the OSI 7-layer burrito for another article. The important part is how packets function as messages. In sending packets back forth, TCP/IP has worked out a complex system to address the kind of problems we struggle with in software. For example, TCP/IP has worked out complex acknowledgement systems to ensure that packets aren’t corrupted. This is similar to concerns for malformed messages. Things like XML or JSON formatting errors. TCP/IP has to handle incoming packets of various sizes and complexities at different rates. This why we have buffers to contend with in network configurations. These buffers are similar to brokers.
We can actually make this sort of analogy to a variety of processes. HTTP, physical mail, accounting, and business practices. In this series, we’ll explore how to design Messaging workflows reusing these sorts of proven methods.