8

I try to understand some aspects of AMQP protocol. Currently I have project with RabbitMQ and use python pika library. So question is about acknowledgement and message prefetching.

  1. Consider we have a queue with only consumer (for sure this queue was declared as exclusive). So do I understand correctly: no matter if I consume with or with no ack flag? Anyway I should not be able to process several messages simultaneously and there are no another consumers that could take some of other still queued messages. Even better not to turn acknowledgement on, because possibly this may reduce AMQP server load.

  2. Prefetch count doesn't mean anything if there is no acknowledgement. Correct?

  3. I am not sure how prefetching works. I have a callback on new message and in its finally statement I acknowledge or reject the message. This is the only function and no matter how big prefetch count would be - anyway another message would not be processed until the current one is completed. So why should I need change prefetch_count value?

thanks in advance.

1 Answer 1

38

With autoack flag unset, if your application failed during message processing all received messages will be lost. If such situation is quite rare and message lose is appropriate option in your application (for example, but no limited to, logs processing) you may turn autoack off.

And yes, having autoack unset requires simpler broker logic, so it utilize less resources.

As to prefetch count (and prefetch size), these options tell broker how large maybe the payload that sent to client in advance to single message. Usually it is used to save time on network operations to wait for a new messages. When prefetch size used client will receive one or more messages that has total size equal or less to preset pretch size (and/or count, which is less).

Both prefetch count and prefetch size rules is they set are applied. When one of them is set to zero (unset) it will not be applied.

The most important thing, prefetching defines behavior to send messages in advance of more message that client has unacked.

In conjunction these two settings produce something like this:

Prefetch with messages count limit and sending messages in advance:

Conditions:

  • Queue: N messages x 1kb
  • Prefetch: prefetch-size=5kb, prefetch-count=4
  • Autoack: off

The workflow:

  • Broker sends 4 messages (limited by prefetch-count=4) to client. 4 messages will be marked as un-acked and moved out from queue (so they wont be delivered to other clients).
  • Client ack 1 message.
  • Broker has -1 message unacked (remove that message) and send 1 more message to client (+1 un-aked, -1 from queue, while client already has 3 messages un-acked).
  • Client ack the rest 3 messages + new delivered one.
  • Broker has -4 messages un-acked and send 4 message again, +4 un-aked, -4 from queue.
  • Client ack 1 message and fails.
  • Broker will -1 un-acked and then move the rest un-acked to queue back, so -3 un-aked and +3 queue, so they may be delivered to this or other client again.

Prefetch with large messages:

Conditions:

  • Queue: 1 message x 5Kb, N messages x 1kb
  • Prefetch: prefetch-size=5kb, prefetch-count=2
  • Autoack: off

The workflow:

  • Broker sends 1 message (limited by prefetch-size=5kb) to client and that message marked as un-aked.
  • Client ack 1 message.
  • Broker has -1 messages un-acked, send 2 messages again (limited by prefetch-count=2, note, that only first message was 5kb, the rest are 1kb) and these messages marked as un-acked.
  • Client ack 1 message and fails.
  • Broker will move acked message out of messages queue and the rest of un-acked messages will be moved again to queue they belongs to, so they may be delivered to this or other client again.

With auto-ack:

Conditions:

  • Queue: N messages x 1kb
  • Prefetch: prefetch-size=5kb, prefetch-count=10
  • Autoack: on

The workflow:

  • While both prefetch-size and prefetch-count ignored when no-ack set to true (that is how auto-ack functionality called in RabbitMQ and in AMQP docs), messages will be sent to client one-by-one and removed from queue after successful sending.

Note, that AMQP has asynchronous architecture, so under certain conditions two clients MAY receive single message at the same time. Also un-acked message MAY be delivered to the same client back (especially if you have single client).

Also, look at prefetch-size and prefetch-count official documentation and experiment a bit with these options.

P.S.: autoack is basically no-ack AMQP flag set to true.

4
  • 1
    Great answer, thanks for the examples! So have I understood correctly: though client cannot process more than 1 message from queue simultaneously it can prefetch some messages (limited by number and / or by size) in some kind of cache to have quick access for them (not to wait them from the server)? Seems it is implemented internally by AMQP library and is transparent for user so I work with prefetching in the same way like without it.
    – Serge
    Feb 11, 2014 at 7:59
  • In case of "No autoack", how does broker come to know how much consumer has processed? In your example, you mentioned 3 messages are processed and then broker sends two more. Can you please explain this again?
    – Akash Jain
    Jun 13, 2015 at 12:24
  • @AkashJain, I had multiple errors in my answer, stackoverflow.com/posts/21662129/revisions, should be fixed now. Sorry for any inconvenience.
    – pinepain
    Jun 14, 2015 at 20:32
  • @Serge A client can process many message 'simultaneously' - ie. with threads. This requires more coordination and individual message tracking but is entirely possible. The only restriction is that it client read the messages from the front of the queue.. from there until the client ACKs/NACKs/DCs, is the client's responsibility. Sep 14, 2017 at 18:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.