Repository navigation
Connection stays open randomly after closing/crashing consumer #635
Description
Hi, I have no idea how this or why this happends. But on our enterprise application (with 2 consumer connections always open) it sometimes happends that the consumer restarts or loses connection. After this RabbitMQ Manager shows that there are now 4 connections (instead of the intended 2). This keeps duplicating over time.
Only solution we have found. Is to manually log into the management page and kill the connections manually.
Do you recognise this issue or have a fix for it available?
Activity
Matts commentedon Apr 19, 2021
Another one of our applications, that uses the Symfony Messenger, does not have this issue. So I don't think this is an architectural IT issue on our side.
[-]Connection stays open randomly after closing consumer[/-][+]Connection stays open randomly after closing/crashing consumer[/+]Matts commentedon Apr 19, 2021
Also using SimpleBus/rabbitmq-bundle-bridge but thinking this would be an issue on this level seeing as they don't do anything with connection. And it also happends when just using a normal consumer
I am thinking it may have something to do with the sigkill/sigterm
MKCG commentedon Apr 21, 2021
One workaround is to defined a heartbeat in order for the rabbitmq server to proactively close any dead tcp connection : https://www.rabbitmq.com/heartbeats.html
mihaileu commentedon Apr 21, 2021
@Matts is this solving your issue ?
Matts commentedon May 3, 2021
I will check this later today. I remember however something saying in the documentation that the heartbeat should be 2x the time of your running consumer. This can possibly cause an issue as our consumers can sometimes be processing for multiple hours. Will reply with results.
Matts commentedon Jun 16, 2021
What I found, was that adding a graceful termination period to my k8s helm config solved the issue partially
terminationGracePeriodSeconds https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#hook-handler-execution
Now is just still the issue that the consumer does not detect a closed TCP connection after running for a while. I expect that with heartbeats/keepalive I may be able to solve it.