Advanced Message Queuing Protocol

The Advanced Message Queuing Protocol (AMQP) is an open standard application layer protocol for Message Oriented Middleware. The defining features of AMQP are message orientation, queuing, routing (including point-to-point and publish-and-subscribe), reliability and security^[1].

AMQP mandates the behaviour of the messaging provider and client to the extent that implementations from different vendors are truly interoperable, in the same way as SMTP, HTTP, FTP, etc. have created interoperable systems. Previous attempts to standardise middleware have happened at the API level (e.g. JMS) and this did not create interoperability^[2]. Unlike JMS, which merely defines an API, AMQP is a wire-level protocol. A wire-level protocol is a description of the format of the data that is sent across the network as a stream of octets. Consequently any tool that can create and interpret messages that conform to this data format can interoperate with any other compliant tool irrespective of implementation language.

Development

AMQP was developed from mid-2004 to mid-2006 by JPMorgan Chase & Co. and iMatix Corporation who also developed implementations in C/C++ and Java. JPMorgan Chase & Co. and iMatix documented the protocol as an interoperable specification and assigned to a working group that included Red Hat, Cisco Systems, TWIST, IONA, and iMatix. As of November 2009, the working group consists of Bank of America, Barclays, Cisco Systems, Credit Suisse, Deutsche Börse Systems, Envoy Technologies, Inc., Goldman Sachs, Progress Software, iMatix Corporation, JPMorgan Chase Bank Inc. N.A, Microsoft Corporation, Novell, Rabbit Technologies Ltd., Red Hat, Inc., Solace Systems, Tervela Inc., TWIST Process Innovations ltd, WS02 and 29West Inc.

A notable design goal of AMQP was to enable the creation of open standard protocol stacks for business messaging both within and between firms by combining AMQP with one of the many open standards describing business transactions, such as FpML or more generically as a reliable transport for SOAP.

Whilst AMQP originated in the financial services industry, it has general applicability to a broad range of middleware problems.

The AMQP model

AMQP defines a number of entities. From a connection perspective the relevant ones are:

Message broker: a server to which AMQ clients connect using the AMQ protocol. Message brokers can run in a cluster but these details are implementation specific and are not covered by the specification.
User: a user is an entity that, by providing credentials in form of a password, may or may not be authorized to connect to a broker.
Connection: a physical connection e.g. using TCP/IP or SCTP. A connection is bound to a user.
Channel: a logical connection that is tied to a connection. Hence communication over a channel is stateful. Clients that perform concurrent operations on a connection should maintain a distinct channel for each of those. Clients that use a threaded model of concurrency can for example encapsulate the channel declaration in a thread-local variable.

Entities in the AMQP model used for message transfer

The entities used for the actual sending and receiving of messages are all declared on a channel. A declaration assures the issuing client that the entity exists (or was previously declared by another client). Any attempt to declare a named entity with different properties than it was declared before will result in an error. In order to change the properties of such an entity it must be deleted prior to a re-declaration (with changed properties).

Some of these entities are named. The naming must be unique within the scope of the entity and its broker. Since clients usually (at least no such operations are defined in the AMQP specification) do not have the means to get a list of all available named entities, the knowledge of an entity name is what allows the client to perform operations on it.

Names are encoded in UTF-8, must be between 1 and 255 characters in length and must start with a digit, a letter or an underscore character.

Exchanges

Exchanges are the entities to which messages are sent. They are named and have a type as well as properties such as:

passive: the exchange will not get declared but an error will be thrown if it does not exist.
durable: the exchange will survive a broker restart.
auto-delete: the exchange will get deleted as soon as there are no more queues bound to it. Exchanges to which queues have never been bound will never get auto deleted.

Note that exchanges are scheduled for removal in AMQP/1.0.

Queues

Queues are the entities which receive messages. They are named and have properties but not a type. Clients can subscribe to queues to the effect that the message broker delivers (pushes) the contents of the queue to the client. Alternatively clients can pop (pull) messages from the queue as they see fit.

Message are guaranteed to be delivered in the order that they were first delivered to a queue. In some kinds of rerouting operations (e.g. due to failures) such an order is not guaranteed.

The properties of queues are:

alternate-exchange: when message are rejected by a subscriber or orphaned by queue deletion, its message get routed to this exchange and get removed from the queue.
passive: the queue will not get declared but an error will be thrown if it does not exist.
durable: the queue will survive a broker restart.
exclusive: there can only be one client for this specific queue.
auto-delete: the queue will get deleted as soon as no more subscriptions are active on it. This shares the same constraint as the auto-delete property for exchanges: if no subscription has been ever active on the queue it will not get auto-deleted. An exclusive queue however will always get auto-deleted when the client terminates its session.

Note that queues are scheduled to replace exchanges in AMQP/1.0.

Messages

Messages are unnamed and are published to an exchange. They consist of a header and a content body. While the body is opaque data the header contains a number of optional properties:

routing-key: this field is used in ways dependent on the type of the exchange.
immediate: the message will get handled as unroutable if at least one of the queues which would receive the message has no subscription on it.
delivery-mode: indicates that a message might need persistence. Only for such messages the broker makes a best-effort the prevent a loss of the message before consumption. If there is uncertainty on the broker's end about the successful delivery of a message (e.g. in case of errors) it might deliver a message more than once. Non persistent delivery modes do not show this kind of behavior.
priority: an indicator (a range between 0 and 9) that a message has higher precedence than others.
expiration: the duration in milliseconds before the broker may handle the message as unroutable.

Bindings

A binding is a relationship between one queue and one exchange that specifies how messages flow from the exchange to the queue. The binding properties match the routing algorithm used in exchanges. Bindings (and exchange algorithms) can be placed on a curve of increasing complexity:

Unconditional - the binding has no properties and requests "all" messages from the exchange.
Conditional on a fixed string - the binding has one property, the routing key and requests all messages that have an identical routing key.
Conditional on a pattern match - the binding has one property, the routing key and requests all messages that match the routing key using a pattern-matching algorithm. Arbitrary pattern syntaxes could be used. AMQP implements topic matching.
Conditional on multiple fixed strings - the binding has a table of properties, the arguments and requests all messages whose headers match these arguments, using logical ANDs or ORs to combine matches.
Conditional on multiple patterns - the binding has a table of properties, the arguments and requests all messages whose headers match these arguments, using a pattern matching algorithm and logical combinations.
Conditional on algorithmic comparison - the binding has an algorithmic expression (like an SQL SELECT WHERE clause) and requests all messages whose headers match that expression.
Conditional on content inspection - the binding specifies arbitrary criteria that are resolved by inspection of the actual message content.

Not all these are implemented as standard, or by all implementations.

Exchange types and the effect of bindings

These four entities form the basic model of the AMQP. The key to understand how a message is passed to a queue lies in the relationship between the type of an exchange and the resulting interpretation of the routing key.

An exchange will deliver up to one copy of a message to a queue if the routing key in the message matches a binding (subsequent semantically identical bindings will not lead to duplicate copies). What constitutes a match however is solely dependent on the type of an exchange:

a direct exchange matches when the routing key property of a message and the key of the binding are identical.
a fanout exchange always matches, even on bindings without a key.
a topic exchange matches the routing key property of a message on binding key words. Words are strings which are separated by dots. Two additional characters are also valid: the *, which matches 1 word and the #, which matches 0..N words. Example: *.stock.# matches the routing keys usd.stock and eur.stock.db but not stock.nasdaq.
a headers exchange matches on the presence of keys as well as key–value pairs which can be concatenated with logical and–or connections in a messages header. In this case the routing key is not a criterion for matching that is considered by the exchange. Neither does the binding carry a single routing key but special format which contains header keys and / or key-value-pairs which match on the header key being present or the header key being present and the value being the same respectively.

Other e.g. vendor-specific exchanges are explicitly permitted in the specification.

The concept of binding named queues to named exchanges has powerful properties (with binding making those two entities independent of each other). It is, for instance, possible to bind a single queue with multiple bindings to the same or to different exchanges. Or multiple consumers can share the name of a queue and bind to it with the same parameters and will therefore get only message that the other consumers did not consume. Or multiple consumers can declare independent queues but share the bindings and get all the message every other consumer would get on the bound exchange with these bindings.

Specification revisions and the future of AMQP

The following specifications of the AMQ protocol have been published, in chronological order:

0-8 in June 2006
0-9 in December 2006
0-10 (documents are undated)
0-9-1 in November 2008
1.0 draft (document is undated)

The draft 1.0 specification changes the AMQP model illustrated above by removing the concepts of exchanges and bindings, and replacing these with queues and links. This change aims to remedy two problems with the previous approach:

The publisher needs to know too much about the receivers topology (what exchanges and exchange types are available).
Producer flow control is challenging - if an Exchange is routing a message to 2 different queues, one empty and the other nearly full, what flow control information should be relayed to the producer and how would that be determined?

Other changes include the introduction of a queue addressing schema similar to E-mail and XMPP. This raises addresses to first-class entities, and allows for the publication of service location records using the DNS.

Implementations

These are the known publicly available AMQP implementations:

OpenAMQ, original open-source implementation of AMQP, written in C by iMatix. Runs on Linux, AIX, Solaris, Windows, OpenVMS. Includes broker, APIs in C/C++ and Java JMS, remote admin shell, scripting, federation, failover, and AMQP-over-HTTP via the RestMS protocol.
Apache Qpid, a project in the Apache Foundation. Bindings to many languages without the use of DLL's.
Red Hat Enterprise MRG implements the latest version of AMQP 0-10 providing rich set of features like full management, federation, Active-Active clustering using Qpid as upstream, adds a web console and many enterprise features.
RabbitMQ, an independent open-source implementation. The server is written in Erlang
AMQP Infrastructure yum installable AMQP 0-10 (maintained in the latest 3 versions of Fedora) which includes the broker, management tools, agents and clients.
ØMQ, a high-performance messaging platform that is able to treat AMQP-compliant brokers as nodes in distributed messaging network. It has bindings to its underlying C implementation from a lot of languages such as Python, C++, Lisp, Ruby etc.
Zyre, a broker that implements RestMS and AMQP to provide RESTful HTTP access to AMQP networks.
DE.SETF.AMQP, a Common Lisp client library for AMQP.

Comparative specifications

These are the known open specifications that cover the same or similar space as AMQP:

Stomp, a text-based pub-sub protocol developed by Codehaus, uses the JMS-like semantics of 'destination'.
RestMS, an HTTP-based message routing and queuing protocol that provides AMQP interoperability through an optional profile.
XMPP, the Extensible Messaging and Presence Protocol.
OpenWire - Protocol used by ActiveMQ

There has not as yet been a formal comparison of these and other protocols in the same space, although an informal comparison of XMPP and AMQP may be found here. While JMS, the Java Messaging service, is often compared to AMQP, JMS is an API specification (which does not guarantee interoperability between implementations, and thus requires deployment of software from one provider on both client and server) while AMQP is a wire-level protocol specification (which in theory does provide interoperability). Note that, like HTTP and XMPP, AMQP does not have a standard API.

References

^ O'Hara, J. (2007). "Toward a commodity enterprise middleware" (PDF). Acm Queue. 5: 48–55. doi:10.1145/1255421.1255424.
^ Vinoski, S. (2006). "Advanced Message Queuing Protocol". Ieee Internet Computing. 10: 87–89. doi:10.1109/MIC.2006.116.

External links

[1] O'Hara, J. (2007). "Toward a commodity enterprise middleware" (PDF). Acm Queue. 5: 48–55. doi:10.1145/1255421.1255424.

[2] Vinoski, S. (2006). "Advanced Message Queuing Protocol". Ieee Internet Computing. 10: 87–89. doi:10.1109/MIC.2006.116.

[1]

[2]