Docker nginx-proxy round robin…

nginx + docker
docker + nginx

So we use docker for container services at work. One container that is part of our ‘dev’ tools is an nginx reverse proxy tied to port 80. IT allows us to run many projects at once using hostnames and port 80; just like in production.

So we are chugging along and I start an applications http service. Hit the service in the browser, all good, reload: broken, reload: good, reload: broken. I am all like wait a minute, thats not cool. After about 5 minutes of debugged we make a realization: I have two instances of the http service running.

Turns out the nginx reverse proxy round robins requests when more than one service is running.

*Note: This was all done w/o docker swarm enabled. Just plain docker-compose and docker run where used.

When removing an orphans is bad…

So Docker creates containers, when you stop these containers the container still exists it is simply stopped.

And then we create more containers…and more containers…and more containers…until we look at docker ps -a and see dozens. As such I recently began using the --remove-orphan flag of docker-compose. 

As the project progressed it got to the point where we were ready to integration out micro-services. So I started up the API server, then the user frontend…and the server exited with a code 137. Hua? Checked ports: ok, checked logs: nothing, check Google / GitHub / Stack Overflow; not much stated explaining why one service would cause another to exit with a SIGTERM command.

Come to find out when you start a stack based on a docker-compose.yml docker uses the directory name as the project name. Since we try not to clutter the root of projects both/all projects have a structure of ./{app root}/docker/{service name}. Soooo --remove-orphan was removing the services not attached to the YML file of the application. Essentially termination everything not itself.

Well, I thought, we can name containers we should be able to name the project right? Nope. As of 1.13, networks, containers, and swarm can be name (iirc) in the docker-compose.yml configuration. But to name a  projects it can only be done on the CLI during container run.

:S there went 3 hours of my life under the assumption of consistency from a tool.

TL;DR: Never assume your toolset operates consistently, always verify.