Many sysadmins first come across Git when a developer hands them a repo. Yet version control is just as much an operations tool: if your server configurations, maintenance scripts, Ansible playbooks and Terraform code are in Git, you can always see who changed what, when and why, and you can undo a bad change in minutes. In this article we go through the most important git commands – the fifteen or so that we actually use in daily work – and show how a small team can get started.
The latest version of Git is 2.46, released on 29 July. Distributions ship older versions – Ubuntu 24.04 LTS has 2.43, Debian 12 has 2.39 – but everything below works with all of them.
What should go into Git, and what shouldn’t?
Anything that is text and that you write by hand belongs in Git:
- Ansible playbooks, roles and the inventory,
- Terraform code and modules,
- shell and Python scripts, systemd units,
- configuration templates (nginx, Postfix, exports of your Zabbix templates),
- runbooks and documentation in Markdown.
What doesn’t belong in it: passwords, private keys, API tokens, the Terraform state file, backups, large binary files. There is a separate section on this below.
The first steps
Once per machine, set whose name commits are made under, and make main the default branch for new repos:
git config --global user.name "John Smith"
git config --global user.email "ops@example.com"
git config --global init.defaultBranch main
From 2.46, git config also has subcommands (git config get, git config set), but the old form above still works, and on older Git versions it’s the only one that does.
git init creates a new repo, and git clone downloads an existing one:
git init ansible
git clone git@gitlab.example.com:ops/ansible.git
The git commands you use every day
| Command | What is it for? |
|---|---|
git status |
What has changed, and what is staged for the commit |
git diff / git diff --staged |
The unstaged and the staged changes, line by line |
git add <file> / git add -p |
Stage a change; -p asks about each hunk |
git commit -m "…" |
Record the staged changes |
git log --oneline --graph |
A compact history, with the branches |
git branch |
List branches, delete them (-d) |
git switch <branch> / git switch -c <branch> |
Switch to an existing branch, or create a new one |
git merge <branch> |
Merge a branch into the current one |
git pull / git push |
Sync with the remote repo (GitLab) |
git stash / git stash pop |
Put unfinished work aside, then bring it back |
git revert <commit> |
Undo the effect of a commit as a new commit |
git reset |
Reset the branch to an earlier state (carefully!) |
git tag |
Name a state, for example a deployment |
git switch has existed since version 2.23; older guides use git checkout for the same thing.
A typical change from start to finish looks like this:
git switch main
git pull
git switch -c nginx-body-limit
# ... editing ...
git status
git diff
git add roles/nginx/templates/site.conf.j2
git commit -m "nginx: raise client_max_body_size to 50 MB"
git push -u origin nginx-body-limit
revert or reset?
These two commands cause the most trouble, so it’s worth being clear about the difference:
git revertcreates a new commit that exactly undoes the changes of an earlier commit. The history stays, and everyone can see what happened. If a change you have already pushed caused a problem in production, use this.git resetmoves the branch back to an earlier commit and “drops” the later ones from the branch.--softkeeps the changes staged, while--hardalso resets the working directory, along with any uncommitted changes. Only use it on commits you haven’t pushed yet.
git revert a1b2c3d # undo the faulty commit as a new commit
git reset --soft HEAD~1 # undo the last, still local commit; the changes are kept
If you do lose something, git reflog shows where the branch was before, and you can usually recover it from there.
Tags for deployments
A tag tells you exactly which state went to production. When you are troubleshooting, it’s worth its weight in gold:
git tag -a v2024.08.1 -m "Postfix relay change, deployed 22 Aug"
git push origin v2024.08.1
.gitignore: keep the secrets out
Patterns listed in the .gitignore file never get into the repo, not even by accident. A good starting point for an operations repo:
# Terraform
.terraform/
*.tfstate
*.tfstate.*
crash.log
# Ansible
*.retry
.vault_pass*
# General
.env
*.pem
*.key
A few rules we always follow:
- Secrets go into the repo encrypted, for example with Ansible Vault. The vault password never does.
- The Terraform state goes to remote storage, for example GitLab’s built-in Terraform state backend, not into the repo; the state can contain passwords and keys too.
.gitignoreonly affects new files. Anything already committed is removed from tracking withgit rm --cached <file>.- If a secret has already been pushed, replace it. Rewriting the history (for example with
git filter-repo) also helps, but by then the repo may have been downloaded to several machines, so changing the password or key comes first.
Commit messages that still make sense six months later
Commits with the message “fix” or “update” tell you nothing six months later. A few simple habits:
- The first line is short (around 50 characters) and says what changed, with the component name at the front.
- After a blank line comes why: this is the part you can’t tell from the code.
- One commit is one logical change. Changing the nginx settings and changing the user list are two commits.
- If you have a ticketing system, refer to the ticket.
postfix: switch relayhost to the new smarthost
The old provider is shutting down the relay at the end of September,
and the new server is already accepting our mail in production.
Ticket: OPS-142
A simple workflow with GitLab merge requests
A small team doesn’t need a complicated branching model. This setup has worked for us for years:
- The
mainbranch is protected: nobody pushes to it directly, and it always contains what runs in production. - Every change is made on its own branch, with a descriptive name (
zabbix-new-servers,nginx-body-limit). - After the push, a merge request is opened. GitLab can do this with a push option too:
git push -o merge_request.create -o merge_request.target=main. - CI checks the change:
ansible-lint,terraform fmt -check,terraform validate, and the output ofterraform plangoes into the merge request. - A colleague reviews the diff, and the change only gets into
mainafter approval. - Deployments only go out from
main, and then you add the tag.
A second pair of eyes and automatic checks aren’t bureaucracy: July’s CrowdStrike outage also showed how much it’s worth when a change doesn’t reach everywhere at once, unchecked.
How to get started
You don’t have to put everything into Git at once. Start with the scripts and the most frequently changed configurations, in one GitLab project, with a protected main branch. After a few weeks the team will start using merge requests on its own, because it’s more convenient than agreeing by email on who changed what on the server.
If you need help setting up GitLab, tidying up Ansible or Terraform code, or building a CI pipeline, get in touch; you can read more about this on the Cloud and DevOps page.