Posted on: 2024-08-26
So, I wanted to set up a container registry in our GitLab instance. This is very useful for automatic CI/CD, Docker images can be built directly by a GitLab runner and pushed into the projects registry for later pulling into the Docker Swarm. When it works, that is.
When the pipeline first ran, it all seemed to work. It would login using the temporary token, build the container, but on pushing, this "helpful" message appeared:
error parsing HTTP 404 response body: invalid character '<' looking for beginning of valuefollowed by a bit of HTML. Now that doesn't look right... It seems there was a misconfiguration, but what was it? What got me very confused is that the docker login worked. I tried it with a User Access Token on my own PC, with the same result. I could login with a correct password, then pushing failed. Incorrect passwords resulted in an error, as they should.
Digging throug the forums I found that there are two likely problems which cause this error:
a) The Container registry needs to be on a different domain or port. I chose to use a different domain because I didn't want to mess with port numbers. In hindsight, switching port would have helped me to diagnose the problem quicker.
b) A load balancer in front of GitLab forwards requests to the wrong service. We don't have a load balancer.
Opening the registry URL in the browser showed me the Gitlab Login page, which I found weird but again didn't think too much about it, maybe it was a feature. So it took me way too many hours to debug this frustrating issue.
On a whim I tried messing with the curl options a bit when I suddenly had a lightbulb moment:
$ curl https://gitlab-registry.example.com/
<html><body>You are being <a href="https://gitlab-registry.example.com/users/sign_in">redirected</a>.</html>
$ curl -4 https://gitlab-registry.example.com/
$ Huh? That's weird. I get a different response on IPv6 (default) than on IPv4. Long story short, GitLab is one of those things which should support IPv6 just out of the box. Afterall everything goes through nginx which has excellent IPv6 support, but for some reason, GitLab is being difficult here. Just to get GitLab to even listen on IPv6 you have to add extra configuration:
nginx['listen_addresses'] = ['*', '[::]']But the thing is that the registry server needs it's own listener configuration. Yes, it's mentioned in the overly long and complex configuration file. No, it's not intuitive. So when a request comes in via IPv6, nginx only knows the default listener and forwards everything to it. When the request comes in via IPv4, it differentiates between registry and main listener.
This line fixes it:
registry_nginx['listen_addresses'] = ['*', '[::]']