cPanel server configuration as code: GitLab and Ansible in practice

The configuration of a cPanel/WHM server lives in many places: Apache and PHP settings, the CloudLinux PHP Selector, the CSF firewall, Exim and SpamAssassin, MySQL, DNS, Imunify360 and the files of the monitoring tools. When you operate a smaller fleet, these files easily drift apart: on one server someone changed a value by hand, on another cPanel rewrote its own part, and on a third there is a leftover fix from some troubleshooting. In this article we describe how we handled this with a Git repo, an Ansible playbook and GitLab CI, and what we found along the way.

Why aren’t WHM and copying by hand enough?

WHM takes care of a lot, but it doesn’t track the state of the whole system. It has no history of who changed what and why, and comparing the settings of two servers by hand is slow and error-prone work. Our earlier solution was a file tree that we copied up to the servers with shell scripts. It worked, as long as nobody needed to know what was on which server.

Managing configuration as code means that:

  • every change is a commit that can be traced and reverted,
  • the desired state is described in one place, not scattered across the servers,
  • the rollout is repeatable and gives the same result on every server.

The structure of the repo

The most important decision was that the repo’s structure mirrors the server’s file system. The file files/etc/apache2/... is rolled out exactly to /etc/apache2/.... This way there is no separate mapping table, and a file’s path makes it clear where it goes.

files/      the configuration file tree, with the real absolute paths
ansible/    playbook and inventory that roll out the contents of files/
scripts/    helper scripts, for example the comparison with the live servers

The Ansible playbook contains one task per component, and each has a handler: a service is only restarted if the file belonging to it has actually changed. Apache, for example, is only reloaded if files in the apache2, PHP Selector or EasyApache directories have changed.

There is one important limitation worth knowing in advance: Ansible’s copy module doesn’t delete. If you remove a file from the repo, it stays on the server. This has to be documented, otherwise the state of the repo and of the servers gradually diverges.

GitLab CI: checks in the merge request, rollout on the main branch

The pipeline has two levels:

  • In a merge request: syntax check and a --check --diff run. The reviewer sees exactly what would change on the servers before the code gets into the main branch.
  • On a commit to the main branch: syntax check, then automatic deployment to every server.

The latter is the biggest risk, and it was no accident that we decided it should work this way. A change pushed to the main branch affects production servers, without a dry run. So the team adopted a clear rule: code stays as a local commit until the change is ready to be rolled out. A push already means a deployment.

Comparing the live servers with the repo

The repo is what should be there. To see what is actually on the servers, we wrote a script that pulls every file in files/ from every server into a git-ignored live/ directory, and then diffs them. The script is read-only on the servers and produces two outputs: a summary list of which files differ or are missing, and a detailed diff per server.

The live/ directory is deliberately not under version control. It can contain firewall rules and API keys too, so it must not get into the repo.

What the comparison showed

The first comparison didn’t give the picture we expected. The servers resembled each other more than they resembled the repo. On several dozen files the servers were identical to each other, but differed from the repo’s version. This meant that the servers followed a shared “reality” that hadn’t made it into the repo, and the repo’s contents reflected an older state.

Two lessons came out of this:

  1. Bring in reality first. We didn’t rewrite the repo’s contents because that was theoretically more correct, but because we started from the actual state of the servers. After the full reconciliation, the repo became the source, and we stick to that rule.
  2. Per-server differences need a separate decision. A custom firewall rule or memory setting on one server must not get mixed in at fleet level. If it goes into the shared file, it takes effect on every server, and that has to be thought through in advance.

What we ran into

While introducing it, we hit several bugs that only showed on real servers:

The required_packages variable was overridden. In Ansible, a variable given in the playbook’s vars: block overrides the value in the inventory. If you provide a default in the playbook, the per-server list doesn’t take effect. Solution: no default in the playbook; the | default([]) goes where the variable is used.

The yaml output callback didn’t work. This callback belongs to the community.general collection, and the ansible-core package doesn’t include it. Only ansible-core is installed in CI, so we use the default callback.

GitLab’s file-type variables arrive with permissions that are too open. The key file was created with mode 0666, and ssh-add doesn’t accept that. The pipeline fixes it with chmod 600, then checks the key’s validity with ssh-keygen -y. If the key is corrupted, the error message is “error in libcrypto”; in that case the variable’s contents have to be pasted in again.

The copy module loses the execute permission. Newly created files don’t get the exec permission without mode. Files that need to be executable need mode: preserve.

cPanel rewrites its own files. The keys in cpanel.config, ea4.conf and exim.conf.localopts appear in a different order on every run. So the --diff output shows false changes. The comparison has to be done on sorted content. Lines starting with # Autoconfigured by cPanel are overwritten by cPanel, so there is no point maintaining them by hand.

Several keys ended up on one line. In the root authorized_keys file, two keys accidentally ended up on one line, so on one of the servers the second key didn’t work. Files like this, where the number of lines matters for how they work, have to be checked separately.

What we took away

  • The repo starts from the servers’ reality, not from expectations. As a first step, you have to pull down the production state and compare it, otherwise the rollout causes surprises.
  • A push is a production deployment, even if that isn’t why you wrote the code. If deployment to the main branch is automatic, the review of merge requests and the dry run are the safety net. It’s worth making this an explicit team rule.
  • The checks have to run in the real environment. Most bugs only came to light when the pipeline or the server actually ran.
  • cPanel’s own files are noisy. In the comparison, insignificant differences (ordering, automatically generated lines) can drown out the real changes.

If you operate a similar cPanel fleet and would like to adopt this kind of management, or need help introducing it, get in touch.

Let’s talk about your project

Tell us briefly how we can help, and we will get back to you as soon as we can.

Get in touch