Docker Swarm; the simple and quick way to build a docker cluster. That is until you need to re-address the cluster especially the master.
According to the Docker website, it is not really possible (warning: my vague interpretation of their documentation) and that you should be running your masters with static IP Addresses. Although workers are perfectly fine with dynamic addressing. This is great but what happens when your “test” environment is suddenly your “production” environment?
Create a second master on the new address. This is what I ended up doing – Learn from my mistakes.
If for some reason you can’t create a second master, then you can really hack up docker and force it to do what you want; technically once the swarm is initialised, you can’t change any of these via the docker command line but this will force some changes. This was highly unstable for me.
Within the directory /var/lib/docker/swarm there are two json files that you need to edit, state and docker-state.
On the master node you need to edit the “LocalAddr”, the “ListenAddr” and “AdvertiseAddr” in the docker-state.json, you also need to edit the “addr” in state.json.
On any worker nodes, you need to edit the “RemoteAddr” in the docker-state.json and the “addr” in stat.json.
Then simply restart docker; you may have to remove nodes and rejoin them.
If you’re using puppet and exported resources and get an error while running the puppet agent showing
root@server:$ Error: Could not retrieve catalog from remote server: Error 400 on SERVER: A duplicate resource was found while collecting exported resources, with the type and title CMD[Title] on node HOSTNAME Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run
Then you’ve got a duplicate resource in the puppetdb which typically happens when you export resources to be imported else which has already happened, usually because you’ve rebuilt the host and not deactivated it prior
ZFS in linux doesn’t support quotas, this is a pain in the arse.
After deploying ZFS on our various NFS servers, we were hitting issues where users can’t check their ZFS quotas and specifically on their home space directories.
After a frustrating amount of time researching, I found that there is a Github PR/Merge awaiting for the support and not wanting to compile my own version of ZFS for production gear, I wrote a little set of scripts to hack around the issue for the time being.
Just pop the server script on your ZFS server and have cron run it every x minutes along with piping the stdout to a shared location (we’re using NFS mounted on /mnt/home and then the client script needs to be on all client machines (we used puppet to distribute it). The you should have ZFS Quotas for your end users.
ZFS Quota Output
Its a very quick and dirty script hacked together to solve the problem quickly, so your mileage my vary. If you have any improvements or suggestions, please don’t hesitate to contact me.
As previously mentioned, I have written a nasty bash script to get the user certificate DN for a given username in ARGUS.
This is a step further, it is a WebUI that is seeded by data updated via a script in a cronjob on ARGUS, it requires a shared filesystem (in this case, /mt/admin or it requires ARGUS to have PHP and a WebServer.