<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://www.csb-it.hu/en/feed.xml" rel="self" type="application/atom+xml" /><link href="https://www.csb-it.hu/en/" rel="alternate" type="text/html" /><updated>2026-11-22T17:50:33+00:00</updated><id>https://www.csb-it.hu/en/feed.xml</id><title type="html">CSB IT</title><subtitle>Design, build and operation of cloud, Linux server, virtualisation and hosting platforms: AWS, Google Cloud, OVHcloud, Proxmox, cPanel, WHMCS.</subtitle><author><name>CSB IT Bt.</name><uri>https://www.csb-it.hu</uri></author><entry xml:lang="en"><title type="html">Moving a PHP application into containers with a GitOps approach</title><link href="https://www.csb-it.hu/en/blog/php-app-containers-gitops" rel="alternate" type="text/html" title="Moving a PHP application into containers with a GitOps approach" /><published>2026-11-21T00:00:00+00:00</published><updated>2026-11-22T16:47:55+00:00</updated><id>https://www.csb-it.hu/en/blog/php-app-containers-gitops</id><content type="html" xml:base="https://www.csb-it.hu/blog/php-app-containers-gitops"><![CDATA[<p>Over the past few weeks we moved a PHP application that had been running on cPanel for years into containers. The goal was for the development, test and production environments to be built from the same definition, and for a new version to go out after a <code class="language-plaintext highlighter-rouge">git push</code> without any human intervention. In this article we describe how the solution is put together and the pitfalls we ran into, because those taught us the most.</p>

<h2 id="what-the-system-is-made-of">What the system is made of</h2>

<ul>
  <li><strong>Application:</strong> PHP-FPM and Nginx in separate containers, with Redis for sessions and Solr for search alongside them.</li>
  <li><strong>Infrastructure repo:</strong> the Dockerfiles, the compose files and the pipeline live in a separate Git repository, not next to the application code.</li>
  <li><strong>Build:</strong> GitLab CI builds the images and pushes them to the GitLab Container Registry.</li>
  <li><strong>Runtime:</strong> Docker Compose stacks managed by Portainer.</li>
  <li><strong>Traffic:</strong> Traefik receives the requests and obtains a Let’s Encrypt certificate for every environment.</li>
  <li><strong>Database:</strong> runs on a separate server; the stack only connects to it.</li>
</ul>

<h2 id="a-separate-repo-for-the-infrastructure">A separate repo for the infrastructure</h2>

<p>We deliberately did not put the Docker and CI definitions into the application’s repository. The main reason is separation of responsibilities: the development team owns the application repo, and we, the operations team, own the infrastructure repo. Everyone makes and approves changes in their own repo. The developers don’t have to deal with operational settings, and we don’t touch the application code.</p>

<p>The separation has practical benefits too. The application repo contains nothing but code, and operational changes get their own history: every change is a commit that can be reviewed and reverted.</p>

<p>This works because the Dockerfile and the build context don’t have to be in the same place. The pipeline fetches the application code and uses it as the context, and takes the Dockerfile from the infrastructure repo. The <code class="language-plaintext highlighter-rouge">php.ini</code> and the Nginx configuration come from a second, named context:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker build \
  --build-context infra=. \
  -f docker/php-fpm/Dockerfile \
  -t registry.example.com/app/php-fpm:dev-1a2b3c4 \
  app
</code></pre></div></div>

<p>In the Dockerfile, <code class="language-plaintext highlighter-rouge">COPY --from=infra</code> refers to these files.</p>

<h2 id="pipeline-from-push-to-running-container">Pipeline: from push to running container</h2>

<p>The application is on GitHub, and the build runs on GitLab. A plain GitHub webhook isn’t enough for this, because the two systems expect different formats. So a single small workflow in the application repo calls the GitLab trigger API and passes on the exact commit ID and the branch name. That way the pipeline is guaranteed to build the commit that was pushed.</p>

<p>Every image gets two tags:</p>

<ul>
  <li>the branch name (<code class="language-plaintext highlighter-rouge">dev</code>, <code class="language-plaintext highlighter-rouge">test</code>, <code class="language-plaintext highlighter-rouge">prod</code>), which always points to the latest build;</li>
  <li>the branch name plus the short commit ID (<code class="language-plaintext highlighter-rouge">dev-1a2b3c4</code>), which never changes.</li>
</ul>

<p>After the build, the deploy step calls the environment’s Portainer webhook and passes the commit ID as a parameter:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl --request POST "$WEBHOOK?APP_SHA=$APP_SHA"
</code></pre></div></div>

<p>The compose file builds the image tag from this:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>image: registry.example.com/app/php-fpm:${ENVIRONMENT}${APP_SHA:+-${APP_SHA}}
</code></pre></div></div>

<p>At first we used only the branch name as the tag. It’s convenient, but it doesn’t tell you exactly which version is running, and rolling back is awkward. With the commit ID, the running version is unambiguous, and a rollback is just a redeploy with an earlier ID.</p>

<p>The webhook URL is different for each environment. GitLab’s environment-scoped variables handle this: the deploy job states which environment it deploys to, and GitLab passes the matching <code class="language-plaintext highlighter-rouge">WEBHOOK</code> value. So the pipeline contains a single variable name.</p>

<h2 id="several-environments-on-one-server">Several environments on one server</h2>

<p>The environment is part of every name in the stack: the project, the containers, the network and the Traefik routes. That’s why <code class="language-plaintext highlighter-rouge">dev</code> and <code class="language-plaintext highlighter-rouge">test</code> run side by side on the same machine, behind the same Traefik, without any trouble. Only a few values differ per environment, such as the domain and the externally exposed ports.</p>

<p>Solr and the database are not reachable from the internet. Access to them is provided by a WireGuard VPN per environment: the client reaches only the necessary addresses through the tunnel, and all its other traffic takes the usual route.</p>

<h2 id="what-we-ran-into">What we ran into</h2>

<p><strong>Nginx didn’t even start.</strong> The regular expression in a <code class="language-plaintext highlighter-rouge">rewrite</code> rule contained <code class="language-plaintext highlighter-rouge">{2}</code>. Nginx interprets the curly brace as the start of a block, so such an expression has to be put in quotes.</p>

<p><strong>The health check always reported an error.</strong> In an Alpine-based image, <code class="language-plaintext highlighter-rouge">localhost</code> resolved to an IPv6 address, but Nginx only listened on IPv4. The fix: <code class="language-plaintext highlighter-rouge">127.0.0.1</code> instead of <code class="language-plaintext highlighter-rouge">localhost</code>.</p>

<p><strong>The scheduled tasks wouldn’t have run.</strong> The cron jobs are run by a separate container from the same image, with its own CPU and memory limits, so that it doesn’t take resources away from visitors. The scheduler, however, starts the tasks with a <code class="language-plaintext highlighter-rouge">proc_open</code> call, which <code class="language-plaintext highlighter-rouge">php.ini</code> disables for security reasons. So the cron container gets an additional setting that enables this function only there; on the web side the restriction stays.</p>

<p><strong>The health endpoint returned 200 even when the database was unavailable.</strong> Only the text of the response indicated the error. Now it returns 503, so both the container status and external monitoring notice the problem.</p>

<p><strong>The container logs grew without limit.</strong> Docker doesn’t rotate logs on its own. Every service got a size limit before the disk filled up.</p>

<p><strong>The upload limit lives in two places.</strong> Raising the value in PHP was no use as long as Nginx’s lower limit rejected the request first. The two values have to be set together.</p>

<p><strong>A directory ended up with the wrong owner.</strong> We store the data in the server’s <code class="language-plaintext highlighter-rouge">/data/&lt;environment&gt;/</code> directories, so that backups are simple and a mistyped <code class="language-plaintext highlighter-rouge">docker compose down -v</code> can’t delete them. Along the way it turned out that every image that mounts an empty volume changes its owner, even if it mounts it read-only. So Nginx, which started later, took root ownership of one of the upload directories, and PHP could no longer write to it. The fix is the <code class="language-plaintext highlighter-rouge">nocopy</code> option on the read-only mounts.</p>

<p><strong>Traefik wrote a security warning to the log.</strong> Because of a recent Traefik vulnerability (CVE-2026-88011), headers in the form <code class="language-plaintext highlighter-rouge">X.Forwarded.For</code> sent by the client reach the application untouched: Traefik treats them as separate headers, but PHP reads them as the same as the <code class="language-plaintext highlighter-rouge">X-Forwarded-For</code> header, so the client could forge the values set by the proxy. In our case, Nginx’s default configuration already dropped these, but we turned on filtering in Traefik as well (<code class="language-plaintext highlighter-rouge">aliasHeadersStrategy</code>), so that the protection doesn’t depend on a single default.</p>

<h2 id="what-we-took-away-from-this">What we took away from this</h2>

<ul>
  <li><strong>Test everything before it goes into production.</strong> Most of the problems above came to light not from the documentation, but from running the change on real containers.</li>
  <li><strong>Implicit defaults are the most dangerous.</strong> Missing log rotation, volume ownership and a moving image tag all work fine until one day they don’t.</li>
  <li><strong>A migration is also an inventory.</strong> The old environment quietly took care of a lot of things, such as sending mail and backups. In containers these need to be handled separately, so these gaps have to be mapped out in good time.</li>
</ul>

<p>Today the system works like this: a pushed commit is running in the right environment a few minutes later, and at any time you can tell exactly which version is live. If you’re facing a similar migration and need help, <a href="https://portal.awh.hu/submitticket.php?step=2&amp;deptid=1">get in touch</a>.</p>]]></content><author><name>CSB IT Bt.</name><uri>https://www.csb-it.hu</uri></author><category term="Sysadmin" /><category term="Development" /><category term="docker" /><category term="gitlab" /><category term="linux" /><summary type="html"><![CDATA[We moved a PHP application from cPanel to Docker: infrastructure repo, GitLab pipeline, Portainer, Traefik – what we built and what we ran into.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://www.csb-it.hu/assets/images/og/devops-cicd.png" /><media:content medium="image" url="https://www.csb-it.hu/assets/images/og/devops-cicd.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry xml:lang="en"><title type="html">What is DevOps? Its meaning and practice in small teams</title><link href="https://www.csb-it.hu/en/blog/what-is-devops" rel="alternate" type="text/html" title="What is DevOps? Its meaning and practice in small teams" /><published>2026-11-16T00:00:00+00:00</published><updated>2026-11-22T16:47:55+00:00</updated><id>https://www.csb-it.hu/en/blog/what-is-devops</id><content type="html" xml:base="https://www.csb-it.hu/blog/what-is-devops"><![CDATA[<p><strong>DevOps</strong> is a blend of the words <em>development</em> and <em>operations</em>. It means a way of working in which developers and operations staff don’t hand the software over to each other, but are jointly responsible for it throughout its whole life: from writing it, through deployment, to running in production. The goal is frequent, small and reliable releases.</p>

<p>So DevOps isn’t a program you can install, and it isn’t a job title either – although “DevOps engineer” is a common role today. It is first and foremost a mindset, made tangible by a few well-defined practices and tools.</p>

<h2 id="where-does-devops-come-from">Where does DevOps come from?</h2>

<p>In traditional organisations, development and operations were two separate teams with different goals. Developers were valued for delivering new features – in other words, change. Operations staff were valued for keeping the system running stably – in other words, for the absence of change. Between the two stood a “wall”: developers threw the finished code over it, and operations tried to make it work in production.</p>

<p>The result was infrequent, large, risky releases, and the two teams pointing fingers at each other when something went wrong. The term DevOps caught on in 2009, around the first DevOpsDays conference (Ghent, Belgium), and it was a response to this problem: if the two teams work towards a shared goal with shared tools, change and stability don’t exclude each other.</p>

<h2 id="what-does-it-mean-in-practice">What does it mean in practice?</h2>

<p>DevOps is often summarised with the acronym <strong>CALMS</strong>:</p>

<table>
  <thead>
    <tr>
      <th> </th>
      <th>In practice</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>C</strong>ulture</td>
      <td>Shared responsibility, blameless incident analysis</td>
    </tr>
    <tr>
      <td><strong>A</strong>utomation</td>
      <td>CI/CD, infrastructure as code, automated tests</td>
    </tr>
    <tr>
      <td><strong>L</strong>ean</td>
      <td>Small steps, fast feedback, no unnecessary waiting</td>
    </tr>
    <tr>
      <td><strong>M</strong>easurement</td>
      <td>Monitoring, logging, metrics of the release process</td>
    </tr>
    <tr>
      <td><strong>S</strong>haring</td>
      <td>Shared documentation, knowledge sharing, shared on-call</td>
    </tr>
  </tbody>
</table>

<p>These turn into the following concrete practices:</p>

<ul>
  <li><strong>Everything in version control.</strong> Not just the application code, but the descriptions of the servers, containers and pipelines also live in Git.</li>
  <li><strong>CI/CD.</strong> Every change is automatically built, tested and made release-ready. We have a separate article on this: <a href="/en/blog/mi-az-a-ci-cd-pipeline">What is CI/CD?</a></li>
  <li><strong>Infrastructure as code.</strong> Servers and cloud resources can be built from files that describe them (Ansible, Terraform and the like), without clicking around by hand, and rebuilt at any time.</li>
  <li><strong>Identical environments.</strong> The development, test and production environments are created from the same description, for example from the same Docker image.</li>
  <li><strong>Monitoring and alerting.</strong> The team knows about a fault before the client does, and sees what effect each release had.</li>
  <li><strong>Small, frequent releases.</strong> With a small change it is easy to find what broke the system, and easy to roll back.</li>
</ul>

<h2 id="how-can-devops-be-measured">How can DevOps be measured?</h2>

<p>For a long time, the DORA (DevOps Research and Assessment) research group, part of Google, described how well a team’s software delivery works with four metrics:</p>

<ul>
  <li><strong>Deployment frequency:</strong> how often a new version goes to production.</li>
  <li><strong>Change lead time:</strong> how much time passes between a commit and its production release.</li>
  <li><strong>Change fail rate:</strong> what percentage of releases cause a fault that has to be fixed or reverted immediately.</li>
  <li><strong>Failed deployment recovery time:</strong> how long it takes for the service to recover after a faulty release.</li>
</ul>

<p>In 2024, DORA added a fifth: the <strong>rework rate</strong>, that is, the share of unplanned releases made necessary by a production fault. Together, the metrics measure speed and stability. The most important lesson of the research is that the two aren’t opposites: the best-performing teams are fast and stable at the same time – precisely because they release often, in small steps.</p>

<h2 id="what-isnt-devops">What isn’t DevOps?</h2>

<ul>
  <li><strong>It isn’t a tool.</strong> A Jenkins server or a Kubernetes cluster isn’t DevOps just because the team uses it.</li>
  <li><strong>It isn’t developers running all the operations.</strong> Operations knowledge doesn’t disappear, it just becomes part of the team, and much of the work gets automated.</li>
  <li><strong>It isn’t only for big companies.</strong> In a small team it can be introduced faster, because there are fewer organisational boundaries.</li>
  <li><strong>You don’t need Kubernetes for it.</strong> A well-automated Docker Compose setup or a managed cloud service is simpler and cheaper in many cases.</li>
</ul>

<h2 id="what-does-a-devops-engineer-do">What does a DevOps engineer do?</h2>

<p>A DevOps engineer is typically the specialist who builds and maintains the automation between development and operations: writes CI/CD pipelines, manages infrastructure as code, containerises, designs cloud environments and sets up monitoring. Ideally they don’t “do DevOps” alone, but give developers tools they can use to release safely themselves.</p>

<h2 id="related-concepts">Related concepts</h2>

<table>
  <thead>
    <tr>
      <th>Concept</th>
      <th>What does it mean?</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>DevSecOps</td>
      <td>Building security checks (dependency scanning, secret detection, vulnerability scanning) into the pipeline, rather than leaving them until after the release</td>
    </tr>
    <tr>
      <td>GitOps</td>
      <td>The desired state (what should run, with what settings) is described in a Git repo, and the system adapts to it; a change is a commit. A practical example: <a href="/en/blog/php-alkalmazas-kontenerbe-koltoztetese-gitops-szemlelettel">Moving a PHP application into containers with a GitOps approach</a></td>
    </tr>
    <tr>
      <td>SRE</td>
      <td>Site Reliability Engineering: an approach developed at Google that treats operations as a software engineering task, with measurable reliability targets</td>
    </tr>
    <tr>
      <td>Platform engineering</td>
      <td>Building an internal developer platform on which teams can spin up environments and release in a self-service way</td>
    </tr>
  </tbody>
</table>

<h2 id="where-should-a-team-of-310-people-start">Where should a team of 3–10 people start?</h2>

<p>A small development team doesn’t need a reorganisation or a new position to work with a DevOps mindset. A proven order:</p>

<ol>
  <li><strong>Put everything under version control</strong>, including server settings and deployment steps.</li>
  <li><strong>Automate the tests</strong>, so that they run on every push.</li>
  <li><strong>Containerise the application</strong>, so that the developer machine and the server run the same thing.</li>
  <li><strong>Automate releases</strong>: first to the test environment, then with a one-click production deployment.</li>
  <li><strong>Set up monitoring and alerting</strong>, so that you don’t find out about a fault from a client.</li>
  <li><strong>Write down and test the rollback</strong>, including restoring from backup.</li>
</ol>

<p>Many small teams have no operations person of their own, and it wouldn’t be a full-time job anyway. In that case it works well if an external partner builds the pipeline, the infrastructure and the monitoring, and the team then uses them day to day on its own. We help with this through our <a href="/en/services/cloud-devops">Cloud and DevOps</a> service.</p>]]></content><author><name>CSB IT Bt.</name><uri>https://www.csb-it.hu</uri></author><category term="Development" /><category term="devops" /><category term="gitlab" /><summary type="html"><![CDATA[What does DevOps mean, where does it come from, which practices belong to it, how can it be measured, and where should a development team of 3–10 people start?]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://www.csb-it.hu/assets/images/og/en/mi-az-a-devops.jpg" /><media:content medium="image" url="https://www.csb-it.hu/assets/images/og/en/mi-az-a-devops.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry xml:lang="en"><title type="html">What is CI/CD? Pipeline stages with a GitLab example</title><link href="https://www.csb-it.hu/en/blog/what-is-a-ci-cd-pipeline" rel="alternate" type="text/html" title="What is CI/CD? Pipeline stages with a GitLab example" /><published>2026-11-09T00:00:00+00:00</published><updated>2026-11-22T16:47:55+00:00</updated><id>https://www.csb-it.hu/en/blog/what-is-a-ci-cd-pipeline</id><content type="html" xml:base="https://www.csb-it.hu/blog/what-is-a-ci-cd-pipeline"><![CDATA[<p><strong>CI/CD</strong> stands for <em>continuous integration</em> and <em>continuous delivery</em> or <em>continuous deployment</em>. It is the practice of having an automated process – the <strong>CI/CD pipeline</strong> – build and test every code change and bring it to a release-ready state, or even deploy it, without manual work.</p>

<p>The goal is simple: a release shouldn’t be a dreaded event clicked through by hand, but a boring, repeatable step that can run several times a day.</p>

<h2 id="what-does-ci-continuous-integration-mean">What does CI (continuous integration) mean?</h2>

<p>Continuous integration means that developers merge their work into the shared branch often, even several times a day, and the build and tests run automatically for every such change. If something breaks, it turns up within minutes – while the developer still remembers what they changed – and not weeks later, in a big merge.</p>

<p>CI needs three things:</p>

<ul>
  <li><strong>version control</strong> (Git), in which every change is a commit,</li>
  <li><strong>automated tests</strong>, which make it machine-verifiable that the code works,</li>
  <li>a <strong>CI server</strong>, which runs them on every push.</li>
</ul>

<h2 id="what-does-cd-mean-delivery-or-deployment">What does CD mean: delivery or deployment?</h2>

<p>“CD” can refer to two closely related things. The difference is in the last step: whether a person starts the production deployment, or it happens by itself.</p>

<table>
  <thead>
    <tr>
      <th> </th>
      <th>Continuous delivery</th>
      <th>Continuous deployment</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>What is automated?</td>
      <td>Build, tests, releasable package, deployment to a test environment</td>
      <td>Everything, including the production deployment</td>
    </tr>
    <tr>
      <td>Production release</td>
      <td>A person starts it, with one click</td>
      <td>Happens by itself if all tests are green</td>
    </tr>
    <tr>
      <td>Who is it for?</td>
      <td>Most teams should start with this</td>
      <td>Mature teams with strong test coverage</td>
    </tr>
  </tbody>
</table>

<p>Continuous delivery alone is a huge step compared with manual releases: the production deployment uses the same automated process as the test one; only the trigger remains a human decision.</p>

<h2 id="what-is-a-cicd-pipeline">What is a CI/CD pipeline?</h2>

<p>The pipeline is the sequence of steps that turns a commit into a deployed application. The steps are organised into stages; a stage only starts if the previous one was successful. A typical pipeline:</p>

<ol>
  <li><strong>Build:</strong> installing dependencies, compiling the code or building a container image.</li>
  <li><strong>Test:</strong> unit and integration tests, code style checks, static analysis.</li>
  <li><strong>Security checks:</strong> looking for vulnerable dependencies and committed passwords.</li>
  <li><strong>Packaging:</strong> uploading the releasable artefact (for example a Docker image) to a registry, tagged with the commit ID.</li>
  <li><strong>Deployment to a test environment (staging):</strong> automatically, for every change.</li>
  <li><strong>Deployment to production:</strong> with one click (delivery) or automatically (deployment).</li>
</ol>

<p>An important principle is that the same package goes to the test environment and to production. If you rebuild for the production deployment, you are no longer releasing exactly what you tested.</p>

<h2 id="a-simple-pipeline-with-gitlab-ci">A simple pipeline with GitLab CI</h2>

<p>In GitLab CI, the pipeline is described by the <code class="language-plaintext highlighter-rouge">.gitlab-ci.yml</code> file in the root of the repo. The example below tests a PHP application, builds a Docker image from it, then deploys it automatically to the test environment, and to production at the press of a button:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">stages</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="s">test</span>
  <span class="pi">-</span> <span class="s">build</span>
  <span class="pi">-</span> <span class="s">deploy</span>

<span class="na">test</span><span class="pi">:</span>
  <span class="na">stage</span><span class="pi">:</span> <span class="s">test</span>
  <span class="na">image</span><span class="pi">:</span> <span class="s">composer:2</span>
  <span class="na">script</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="s">composer install --no-interaction --prefer-dist</span>
    <span class="pi">-</span> <span class="s">vendor/bin/phpunit</span>

<span class="na">build</span><span class="pi">:</span>
  <span class="na">stage</span><span class="pi">:</span> <span class="s">build</span>
  <span class="na">image</span><span class="pi">:</span> <span class="s">docker:27</span>
  <span class="na">services</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="s">docker:27-dind</span>
  <span class="na">variables</span><span class="pi">:</span>
    <span class="na">DOCKER_TLS_CERTDIR</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/certs"</span>
  <span class="na">script</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="s">echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"</span>
    <span class="pi">-</span> <span class="s">docker build -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA" .</span>
    <span class="pi">-</span> <span class="s">docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"</span>
  <span class="na">rules</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="na">if</span><span class="pi">:</span> <span class="s">$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH</span>

<span class="na">deploy_staging</span><span class="pi">:</span>
  <span class="na">stage</span><span class="pi">:</span> <span class="s">deploy</span>
  <span class="na">script</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="s">./deploy.sh staging "$CI_COMMIT_SHORT_SHA"</span>
  <span class="na">environment</span><span class="pi">:</span> <span class="s">staging</span>
  <span class="na">rules</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="na">if</span><span class="pi">:</span> <span class="s">$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH</span>

<span class="na">deploy_production</span><span class="pi">:</span>
  <span class="na">stage</span><span class="pi">:</span> <span class="s">deploy</span>
  <span class="na">script</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="s">./deploy.sh production "$CI_COMMIT_SHORT_SHA"</span>
  <span class="na">environment</span><span class="pi">:</span> <span class="s">production</span>
  <span class="na">rules</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="na">if</span><span class="pi">:</span> <span class="s">$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH</span>
      <span class="na">when</span><span class="pi">:</span> <span class="s">manual</span>
</code></pre></div></div>

<p>A few things worth noticing:</p>

<ul>
  <li>The tests run on every branch, on every push (their result is also visible in the merge request), but the build and the deployment only on the main branch.</li>
  <li>The image tag is the commit’s short ID (<code class="language-plaintext highlighter-rouge">$CI_COMMIT_SHORT_SHA</code>), so you always know exactly which code is running, and a rollback is the deployment of an earlier tag.</li>
  <li>The <code class="language-plaintext highlighter-rouge">CI_REGISTRY_*</code> variables are provided by GitLab, so you don’t need to write a password into the repo.</li>
  <li><code class="language-plaintext highlighter-rouge">deploy.sh</code> contains your own deployment steps: a <code class="language-plaintext highlighter-rouge">docker compose pull &amp;&amp; docker compose up -d</code> over SSH, a Kubernetes update or a call to a cloud service’s API.</li>
  <li>Because of <code class="language-plaintext highlighter-rouge">when: manual</code>, this is continuous delivery: the production deployment is a button in the GitLab interface. If you remove it, it becomes continuous deployment.</li>
</ul>

<h2 id="gitlab-ci-github-actions-or-jenkins">GitLab CI, GitHub Actions or Jenkins?</h2>

<table>
  <thead>
    <tr>
      <th>Tool</th>
      <th>Where does the definition live?</th>
      <th>Typically suits</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>GitLab CI</td>
      <td><code class="language-plaintext highlighter-rouge">.gitlab-ci.yml</code> in the repo</td>
      <td>Those whose code is on GitLab (in the cloud or on their own server); built-in registry and environment management</td>
    </tr>
    <tr>
      <td>GitHub Actions</td>
      <td><code class="language-plaintext highlighter-rouge">.github/workflows/*.yml</code></td>
      <td>For projects hosted on GitHub; a large library of ready-made steps (actions)</td>
    </tr>
    <tr>
      <td>Jenkins</td>
      <td><code class="language-plaintext highlighter-rouge">Jenkinsfile</code> in the repo, plus the Jenkins server’s settings</td>
      <td>Where Jenkins is already in place, or a very custom process connecting many systems is needed</td>
    </tr>
    <tr>
      <td>AWS CodePipeline</td>
      <td>AWS console or infrastructure code</td>
      <td>For applications running on AWS, deploying into AWS services</td>
    </tr>
  </tbody>
</table>

<p>In most cases the right choice is wherever your code already is: GitLab CI for a GitLab repo, GitHub Actions for a GitHub repo. Running a separate Jenkins server is a job in itself.</p>

<h2 id="when-is-it-worth-it-for-a-small-team">When is it worth it for a small team?</h2>

<p>Almost always, if any of the following sounds familiar:</p>

<ul>
  <li>a release is an FTP upload or a series of commands run by hand over SSH,</li>
  <li>only one person knows how to do a release,</li>
  <li>nobody dares to deploy on a Friday,</li>
  <li>“it works on my machine” – just not on the server.</li>
</ul>

<p>You don’t have to introduce everything at once. A good order:</p>

<ol>
  <li>Have the tests run automatically on every push (if you don’t have any yet, start with a few critical flows).</li>
  <li>Have the releasable package (image or archive) built in the pipeline, not on a developer’s machine.</li>
  <li>Have every change deployed automatically to the test environment.</li>
  <li>Make the production deployment a single button, with a tested rollback step.</li>
</ol>

<h2 id="common-mistakes">Common mistakes</h2>

<ul>
  <li><strong>Passwords in the repo.</strong> Secrets have to be stored in the CI tool’s protected variables (on GitLab as <em>masked</em> and <em>protected</em> variables), never in the code.</li>
  <li><strong>Slow pipeline.</strong> If a run takes half an hour, developers work around it. Caching dependencies and parallel jobs help a lot.</li>
  <li><strong>No rollback.</strong> If every release gets a unique tag, a rollback is just redeploying an earlier tag.</li>
  <li><strong>Flaky tests.</strong> Tests that occasionally fail for no reason teach the team that a red pipeline “doesn’t matter”. These have to be fixed or removed.</li>
  <li><strong>Servers changed by hand.</strong> If server settings are changed by hand, after a while the pipeline doesn’t get the environment it expects. The solution is to manage the <a href="/en/blog/cpanel-szerverek-konfiguracioja-kodkent-gitlab-es-ansible-segitsegevel">infrastructure as code too</a>.</li>
</ul>

<h2 id="frequently-asked-questions">Frequently asked questions</h2>

<p><strong>Is CI/CD only for big companies?</strong>
No. It pays off for a two-person team too; in fact that is where the gain is biggest: there is no separate person to deal with releases, so automation directly frees up development time.</p>

<p><strong>Do you need Kubernetes or Docker for it?</strong>
It isn’t mandatory. A pipeline can also deploy to a traditional server. Containers do simplify a lot, though, because the same image runs in test and in production.</p>

<p><strong>What is the difference between CI/CD and DevOps?</strong>
DevOps is a broader way of working in which development and operations are jointly responsible for the software; CI/CD is one of its most important tools. More on this: <a href="/en/blog/mi-az-a-devops">What is DevOps?</a></p>

<h2 id="how-do-we-do-it">How do we do it?</h2>

<p>In a recently completed project we moved a PHP application running on cPanel into containers, with a GitLab pipeline, Portainer and Traefik. In <a href="/en/blog/php-alkalmazas-kontenerbe-koltoztetese-gitops-szemlelettel">Moving a PHP application into containers with a GitOps approach</a> we describe what we built and what we ran into.</p>

<p>If your releases are still manual and you would like to automate them, you can read about how we can help on the <a href="/en/services/cloud-devops">Cloud and DevOps</a> page.</p>]]></content><author><name>CSB IT Bt.</name><uri>https://www.csb-it.hu</uri></author><category term="Development" /><category term="gitlab" /><category term="docker" /><category term="devops" /><summary type="html"><![CDATA[What does CI/CD mean, what is the difference between continuous delivery and deployment, how is a pipeline built, and how do you get started with GitLab CI?]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://www.csb-it.hu/assets/images/og/en/mi-az-a-ci-cd-pipeline.jpg" /><media:content medium="image" url="https://www.csb-it.hu/assets/images/og/en/mi-az-a-ci-cd-pipeline.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry xml:lang="en"><title type="html">cPanel server configuration as code: GitLab and Ansible in practice</title><link href="https://www.csb-it.hu/en/blog/cpanel-configuration-as-code-gitlab-ansible" rel="alternate" type="text/html" title="cPanel server configuration as code: GitLab and Ansible in practice" /><published>2026-11-02T00:00:00+00:00</published><updated>2026-11-22T16:47:55+00:00</updated><id>https://www.csb-it.hu/en/blog/cpanel-configuration-as-code-gitlab-ansible</id><content type="html" xml:base="https://www.csb-it.hu/blog/cpanel-configuration-as-code-gitlab-ansible"><![CDATA[<p>The configuration of a cPanel/WHM server lives in many places: Apache and PHP settings, the CloudLinux PHP Selector, the CSF firewall, Exim and SpamAssassin, MySQL, DNS, Imunify360 and the files of the monitoring tools. When you operate a smaller fleet, these files easily drift apart: on one server someone changed a value by hand, on another cPanel rewrote its own part, and on a third there is a leftover fix from some troubleshooting. In this article we describe how we handled this with a Git repo, an Ansible playbook and GitLab CI, and what we found along the way.</p>

<h2 id="why-arent-whm-and-copying-by-hand-enough">Why aren’t WHM and copying by hand enough?</h2>

<p>WHM takes care of a lot, but it doesn’t track the state of the whole system. It has no history of who changed what and why, and comparing the settings of two servers by hand is slow and error-prone work. Our earlier solution was a file tree that we copied up to the servers with shell scripts. It worked, as long as nobody needed to know what was on which server.</p>

<p>Managing configuration as code means that:</p>

<ul>
  <li>every change is a commit that can be traced and reverted,</li>
  <li>the desired state is described in one place, not scattered across the servers,</li>
  <li>the rollout is repeatable and gives the same result on every server.</li>
</ul>

<h2 id="the-structure-of-the-repo">The structure of the repo</h2>

<p>The most important decision was that the repo’s structure mirrors the server’s file system. The file <code class="language-plaintext highlighter-rouge">files/etc/apache2/...</code> is rolled out exactly to <code class="language-plaintext highlighter-rouge">/etc/apache2/...</code>. This way there is no separate mapping table, and a file’s path makes it clear where it goes.</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>files/      the configuration file tree, with the real absolute paths
ansible/    playbook and inventory that roll out the contents of files/
scripts/    helper scripts, for example the comparison with the live servers
</code></pre></div></div>

<p>The Ansible playbook contains one task per component, and each has a handler: a service is only restarted if the file belonging to it has actually changed. Apache, for example, is only reloaded if files in the <code class="language-plaintext highlighter-rouge">apache2</code>, PHP Selector or EasyApache directories have changed.</p>

<p>There is one important limitation worth knowing in advance: Ansible’s <code class="language-plaintext highlighter-rouge">copy</code> module doesn’t delete. If you remove a file from the repo, it stays on the server. This has to be documented, otherwise the state of the repo and of the servers gradually diverges.</p>

<h2 id="gitlab-ci-checks-in-the-merge-request-rollout-on-the-main-branch">GitLab CI: checks in the merge request, rollout on the main branch</h2>

<p>The pipeline has two levels:</p>

<ul>
  <li><strong>In a merge request:</strong> syntax check and a <code class="language-plaintext highlighter-rouge">--check --diff</code> run. The reviewer sees exactly what would change on the servers before the code gets into the main branch.</li>
  <li><strong>On a commit to the main branch:</strong> syntax check, then automatic deployment to every server.</li>
</ul>

<p>The latter is the biggest risk, and it was no accident that we decided it should work this way. A change pushed to the main branch affects production servers, without a dry run. So the team adopted a clear rule: code stays as a local commit until the change is ready to be rolled out. A push already means a deployment.</p>

<h2 id="comparing-the-live-servers-with-the-repo">Comparing the live servers with the repo</h2>

<p>The repo is what should be there. To see what is actually on the servers, we wrote a script that pulls every file in <code class="language-plaintext highlighter-rouge">files/</code> from every server into a git-ignored <code class="language-plaintext highlighter-rouge">live/</code> directory, and then diffs them. The script is read-only on the servers and produces two outputs: a summary list of which files differ or are missing, and a detailed diff per server.</p>

<p>The <code class="language-plaintext highlighter-rouge">live/</code> directory is deliberately not under version control. It can contain firewall rules and API keys too, so it must not get into the repo.</p>

<h2 id="what-the-comparison-showed">What the comparison showed</h2>

<p>The first comparison didn’t give the picture we expected. The servers resembled each other more than they resembled the repo. On several dozen files the servers were identical to each other, but differed from the repo’s version. This meant that the servers followed a shared “reality” that hadn’t made it into the repo, and the repo’s contents reflected an older state.</p>

<p>Two lessons came out of this:</p>

<ol>
  <li><strong>Bring in reality first.</strong> We didn’t rewrite the repo’s contents because that was theoretically more correct, but because we started from the actual state of the servers. After the full reconciliation, the repo became the source, and we stick to that rule.</li>
  <li><strong>Per-server differences need a separate decision.</strong> A custom firewall rule or memory setting on one server must not get mixed in at fleet level. If it goes into the shared file, it takes effect on every server, and that has to be thought through in advance.</li>
</ol>

<h2 id="what-we-ran-into">What we ran into</h2>

<p>While introducing it, we hit several bugs that only showed on real servers:</p>

<p><strong>The <code class="language-plaintext highlighter-rouge">required_packages</code> variable was overridden.</strong> In Ansible, a variable given in the playbook’s <code class="language-plaintext highlighter-rouge">vars:</code> block overrides the value in the inventory. If you provide a default in the playbook, the per-server list doesn’t take effect. Solution: no default in the playbook; the <code class="language-plaintext highlighter-rouge">| default([])</code> goes where the variable is used.</p>

<p><strong>The <code class="language-plaintext highlighter-rouge">yaml</code> output callback didn’t work.</strong> This callback belongs to the <code class="language-plaintext highlighter-rouge">community.general</code> collection, and the <code class="language-plaintext highlighter-rouge">ansible-core</code> package doesn’t include it. Only <code class="language-plaintext highlighter-rouge">ansible-core</code> is installed in CI, so we use the <code class="language-plaintext highlighter-rouge">default</code> callback.</p>

<p><strong>GitLab’s file-type variables arrive with permissions that are too open.</strong> The key file was created with mode 0666, and <code class="language-plaintext highlighter-rouge">ssh-add</code> doesn’t accept that. The pipeline fixes it with <code class="language-plaintext highlighter-rouge">chmod 600</code>, then checks the key’s validity with <code class="language-plaintext highlighter-rouge">ssh-keygen -y</code>. If the key is corrupted, the error message is “error in libcrypto”; in that case the variable’s contents have to be pasted in again.</p>

<p><strong>The <code class="language-plaintext highlighter-rouge">copy</code> module loses the execute permission.</strong> Newly created files don’t get the exec permission without <code class="language-plaintext highlighter-rouge">mode</code>. Files that need to be executable need <code class="language-plaintext highlighter-rouge">mode: preserve</code>.</p>

<p><strong>cPanel rewrites its own files.</strong> The keys in <code class="language-plaintext highlighter-rouge">cpanel.config</code>, <code class="language-plaintext highlighter-rouge">ea4.conf</code> and <code class="language-plaintext highlighter-rouge">exim.conf.localopts</code> appear in a different order on every run. So the <code class="language-plaintext highlighter-rouge">--diff</code> output shows false changes. The comparison has to be done on sorted content. Lines starting with <code class="language-plaintext highlighter-rouge"># Autoconfigured by cPanel</code> are overwritten by cPanel, so there is no point maintaining them by hand.</p>

<p><strong>Several keys ended up on one line.</strong> In the root <code class="language-plaintext highlighter-rouge">authorized_keys</code> file, two keys accidentally ended up on one line, so on one of the servers the second key didn’t work. Files like this, where the number of lines matters for how they work, have to be checked separately.</p>

<h2 id="what-we-took-away">What we took away</h2>

<ul>
  <li><strong>The repo starts from the servers’ reality, not from expectations.</strong> As a first step, you have to pull down the production state and compare it, otherwise the rollout causes surprises.</li>
  <li><strong>A push is a production deployment, even if that isn’t why you wrote the code.</strong> If deployment to the main branch is automatic, the review of merge requests and the dry run are the safety net. It’s worth making this an explicit team rule.</li>
  <li><strong>The checks have to run in the real environment.</strong> Most bugs only came to light when the pipeline or the server actually ran.</li>
  <li><strong>cPanel’s own files are noisy.</strong> In the comparison, insignificant differences (ordering, automatically generated lines) can drown out the real changes.</li>
</ul>

<p>If you operate a similar cPanel fleet and would like to adopt this kind of management, or need help introducing it, <a href="https://portal.awh.hu/submitticket.php?step=2&amp;deptid=1">get in touch</a>.</p>]]></content><author><name>CSB IT Bt.</name><uri>https://www.csb-it.hu</uri></author><category term="Sysadmin" /><category term="cpanel" /><category term="gitlab" /><category term="linux" /><summary type="html"><![CDATA[How can you manage cPanel/WHM server configuration in a Git repo, roll it out from GitLab CI with Ansible, and what did we learn while introducing it?]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://www.csb-it.hu/assets/images/og/cpanel.png" /><media:content medium="image" url="https://www.csb-it.hu/assets/images/og/cpanel.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry xml:lang="en"><title type="html">Prometheus and Grafana or Zabbix? Monitoring for smaller companies</title><link href="https://www.csb-it.hu/en/blog/prometheus-grafana-vs-zabbix" rel="alternate" type="text/html" title="Prometheus and Grafana or Zabbix? Monitoring for smaller companies" /><published>2026-10-16T05:00:00+00:00</published><updated>2026-11-22T16:47:55+00:00</updated><id>https://www.csb-it.hu/en/blog/prometheus-grafana-vs-zabbix</id><content type="html" xml:base="https://www.csb-it.hu/blog/prometheus-grafana-vs-zabbix"><![CDATA[<p>For server monitoring, two open source solutions come up most often: the <strong>Prometheus</strong> and <strong>Grafana</strong> pair, and <strong>Zabbix</strong>. With either of them you can watch server load and service availability, and get an alert if something breaks. Their underlying approach differs, however, and that matters more when choosing than the feature lists do. In this article we compare them and describe when we recommend which.</p>

<h2 id="two-different-approaches">Two different approaches</h2>

<p><strong>Zabbix</strong> is a unified monitoring system: a single server, a database and a web interface in which you configure everything, from adding hosts to alerts. The Zabbix agent installed on the machines collects the data, but Zabbix can also take measurements over SNMP, IPMI, HTTP and in many other ways. It works with templates: when you add a Linux server or a switch, the template brings the measured values, the graphs and the alert thresholds.</p>

<p><strong>Prometheus</strong> is a time series database and data collector that queries (scrapes) the metrics from the monitored targets over HTTP. The metrics are made available by so-called exporters: <code class="language-plaintext highlighter-rouge">node_exporter</code> for the Linux machine’s data, <code class="language-plaintext highlighter-rouge">mysqld_exporter</code> for MySQL’s, and <code class="language-plaintext highlighter-rouge">blackbox_exporter</code> for the availability of websites and ports. Prometheus itself only collects, stores and evaluates rules; sending out alerts is done by <strong>Alertmanager</strong>, and visualisation by <strong>Grafana</strong>. That is three or four separate components, which you connect with configuration files.</p>

<h2 id="the-current-versions">The current versions</h2>

<ul>
  <li><strong>Zabbix:</strong> the current LTS is 7.0, with full support until 30 June 2027, and the latest standard release is 7.4. According to the vendor’s announcement, the next LTS, Zabbix 8.0, is expected later in 2026. Since 7.0, Zabbix has been <a href="https://blog.zabbix.com/striking-the-right-balance-zabbix-7-0-to-be-released-under-agplv3-license/">licensed under AGPLv3</a>.</li>
  <li><strong>Prometheus:</strong> a new minor version comes out every six weeks, with one LTS release a year. The current LTS is <a href="https://prometheus.io/docs/introduction/release-cycle/">3.13, released in July, which gets fixes until 31 July 2027</a>. Its licence is Apache 2.0.</li>
  <li><strong>Grafana:</strong> major version 13 was released in April 2026, at GrafanaCON. The free Grafana OSS is licensed under AGPLv3.</li>
</ul>

<h2 id="comparison">Comparison</h2>

<table>
  <thead>
    <tr>
      <th> </th>
      <th>Prometheus + Grafana</th>
      <th>Zabbix</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Data collection</td>
      <td>the server queries the exporters (pull)</td>
      <td>agent (active or passive), SNMP, IPMI, HTTP, SSH</td>
    </tr>
    <tr>
      <td>Data storage</td>
      <td>its own time series database (TSDB), 15 days by default</td>
      <td>MySQL/MariaDB or PostgreSQL (with TimescaleDB)</td>
    </tr>
    <tr>
      <td>Configuration</td>
      <td>YAML files, easy to keep under version control</td>
      <td>web interface and templates, can be automated via the API</td>
    </tr>
    <tr>
      <td>Query language</td>
      <td>PromQL, very flexible</td>
      <td>predefined items and trigger expressions</td>
    </tr>
    <tr>
      <td>Alerting</td>
      <td>Alertmanager: grouping, silencing, routing</td>
      <td>built-in triggers, actions, escalation</td>
    </tr>
    <tr>
      <td>Visualisation</td>
      <td>Grafana, very rich</td>
      <td>built-in dashboards; there is also a Grafana plugin</td>
    </tr>
    <tr>
      <td>Discovery</td>
      <td>Kubernetes, cloud and DNS-based service discovery</td>
      <td>network discovery, low-level discovery (LLD)</td>
    </tr>
    <tr>
      <td>Network devices</td>
      <td><code class="language-plaintext highlighter-rouge">snmp_exporter</code>, with more manual work</td>
      <td>strong, with many ready-made templates</td>
    </tr>
    <tr>
      <td>Containers, Kubernetes</td>
      <td>its strength; this is the de facto standard</td>
      <td>supported, but not what it was designed for</td>
    </tr>
    <tr>
      <td>Long-term storage</td>
      <td>needs a separate tool (Thanos, Mimir, VictoriaMetrics)</td>
      <td>built in, with trend data</td>
    </tr>
    <tr>
      <td>Learning curve</td>
      <td>steeper, more components</td>
      <td>slower start in the interface, simple after that</td>
    </tr>
  </tbody>
</table>

<h2 id="when-should-you-choose-zabbix">When should you choose Zabbix?</h2>

<ul>
  <li><strong>Mixed environment:</strong> Linux and Windows servers, switches, routers, printers and UPSs, where SNMP devices make up a significant share.</li>
  <li><strong>Classic infrastructure:</strong> virtual machines, physical servers, a few dozen or a few hundred machines, in a setup that rarely changes.</li>
  <li><strong>Few operations staff:</strong> one web interface, one database, one package source; with the ready-made templates, adding a new machine takes minutes.</li>
  <li><strong>Multiple sites:</strong> a Zabbix proxy collects the data at a remote site and forwards it in one go to the central server.</li>
</ul>

<p>We wrote <a href="/en/blog/zabbix-7-szervermonitorozas">about the new features of Zabbix 7.0 in detail when it was released</a>.</p>

<h2 id="when-should-you-choose-prometheus-and-grafana">When should you choose Prometheus and Grafana?</h2>

<ul>
  <li><strong>Containers and Kubernetes:</strong> targets come and go, and Prometheus’s service discovery keeps track of this automatically.</li>
  <li><strong>Application metrics:</strong> your own application exposes its own metrics (number of requests, response time, error rate) directly on a <code class="language-plaintext highlighter-rouge">/metrics</code> endpoint. Most modern software – from Proxmox through GitLab to Traefik – provides Prometheus-format metrics out of the box, or has an exporter for it.</li>
  <li><strong>Infrastructure as code:</strong> every setting is a text file that can be stored in Git and deployed from CI/CD.</li>
  <li><strong>Flexible analysis:</strong> with PromQL you can express almost any question, for example CPU utilisation per machine, averaged over the last five minutes:</li>
</ul>

<pre><code class="language-promql">100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
</code></pre>

<p>An alerting rule that fires if free space on a file system drops below 10 per cent:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">groups</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">disk</span>
    <span class="na">rules</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="na">alert</span><span class="pi">:</span> <span class="s">LowDiskSpace</span>
        <span class="na">expr</span><span class="pi">:</span> <span class="s">node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes &lt; </span><span class="m">0.10</span>
        <span class="na">for</span><span class="pi">:</span> <span class="s">15m</span>
        <span class="na">labels</span><span class="pi">:</span>
          <span class="na">severity</span><span class="pi">:</span> <span class="s">warning</span>
        <span class="na">annotations</span><span class="pi">:</span>
          <span class="na">summary</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Low</span><span class="nv"> </span><span class="s">disk</span><span class="nv"> </span><span class="s">space:</span><span class="nv"> </span><span class="s">{{</span><span class="nv"> </span><span class="s">$labels.instance</span><span class="nv"> </span><span class="s">}}</span><span class="nv"> </span><span class="s">{{</span><span class="nv"> </span><span class="s">$labels.mountpoint</span><span class="nv"> </span><span class="s">}}"</span>
</code></pre></div></div>

<h2 id="trying-them-out-in-a-few-minutes">Trying them out in a few minutes</h2>

<p><strong>Prometheus and node_exporter</strong> on Debian 13, from the distribution’s packages:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt <span class="nb">install </span>prometheus prometheus-node-exporter prometheus-alertmanager
<span class="c"># web interface: http://server:9090, node_exporter: http://server:9100/metrics</span>
</code></pre></div></div>

<p>It’s best to install Grafana from Grafana Labs’ own APT repository so that you get the current version; then you add Prometheus as a data source, and you can import a ready-made Node Exporter dashboard.</p>

<p>For <strong>Zabbix</strong>, once you select the operating system, the database and the web server, the <a href="https://www.zabbix.com/download">zabbix.com download page</a> gives you the exact commands for installing the package repository and setting up the server. The <code class="language-plaintext highlighter-rouge">zabbix-agent2</code> package goes on the machines; the server uses TCP port 10051 and the agent port 10050.</p>

<h2 id="they-can-be-used-together">They can be used together</h2>

<p>The two don’t exclude each other. A common setup is for Zabbix to monitor the network devices and classic servers, Prometheus the containerised applications, and Grafana to show the data from both in one place (there is an official Grafana plugin for Zabbix). In a small team, however, maintaining two systems is twice the work, so we recommend starting with one and only adding the other if you really need it.</p>

<p>Whichever you choose, don’t forget two things. The monitoring system shouldn’t run on the same machine or in the same data centre as what it monitors, because in a bigger outage it becomes unreachable too. We also wrote about this <a href="/en/blog/cloudflare-leallas">in connection with the Cloudflare outage</a>. The other is to install the agents, exporters and templates from code, for example with Ansible, as we showed in our <a href="/en/blog/ansible-kezdoknek">Ansible for beginners</a> article, so that a new machine is automatically added to monitoring as well.</p>

<h2 id="our-recommendation-in-brief">Our recommendation in brief</h2>

<ul>
  <li><strong>A few dozen servers, network devices, few operations staff:</strong> Zabbix.</li>
  <li><strong>Containers, Kubernetes, in-house applications:</strong> Prometheus and Grafana.</li>
  <li><strong>A mix of both:</strong> Zabbix for the infrastructure, Prometheus for the applications, Grafana as a shared interface.</li>
</ul>

<p>We also help with building monitoring systems, setting up alerts and running them; you can read more on our <a href="/en/services/infrastructure">Infrastructure and operations</a> page.</p>]]></content><author><name>CSB IT Bt.</name><uri>https://www.csb-it.hu</uri></author><category term="Sysadmin" /><category term="linux" /><category term="servers" /><summary type="html"><![CDATA[Prometheus and Grafana or Zabbix? We compare the two monitoring approaches: how they work, data storage, alerting, dashboards, and when each is the better fit.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://www.csb-it.hu/assets/images/og/en/prometheus-grafana-vagy-zabbix.jpg" /><media:content medium="image" url="https://www.csb-it.hu/assets/images/og/en/prometheus-grafana-vagy-zabbix.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry xml:lang="en"><title type="html">GitLab Runner on your own server: installation and setup</title><link href="https://www.csb-it.hu/en/blog/gitlab-runner-installation" rel="alternate" type="text/html" title="GitLab Runner on your own server: installation and setup" /><published>2026-09-18T05:00:00+00:00</published><updated>2026-11-22T16:47:55+00:00</updated><id>https://www.csb-it.hu/en/blog/gitlab-runner-installation</id><content type="html" xml:base="https://www.csb-it.hu/blog/gitlab-runner-installation"><![CDATA[<p>GitLab CI/CD jobs aren’t run by GitLab itself, but by a separate program, <strong>GitLab Runner</strong>. GitLab.com provides shared runners, which come with a limited number of compute minutes per month on the free plan, and with a self-managed GitLab you have to set up a runner yourself anyway. With your own runner, the jobs run on your machine: with as many resources as you give it, with access to your internal network, and without per-minute charges. In this article we install <strong>GitLab Runner</strong> with the Docker executor on a Debian or Ubuntu server, and register it for a project.</p>

<h2 id="what-machine-should-you-install-it-on">What machine should you install it on?</h2>

<p>The runner itself doesn’t need a powerful machine, but the jobs (builds, tests, building Docker images) can need a lot more. For a small team, a good starting point is:</p>

<ul>
  <li>2–4 vCPUs, 4–8 GB of memory, 50–100 GB of SSD (Docker images and the cache take up space quickly),</li>
  <li>Debian 12/13 or Ubuntu 24.04/26.04 LTS,</li>
  <li>a <strong>separate virtual machine</strong>, not the production application server, and not an LXC container either: Docker works predictably in a virtual machine. We wrote about the difference between LXC- and KVM-based virtualisation on the <a href="https://www.awh.hu/en/blog/should-we-choose-an-lxc-or-kvm-based-vps-server">blog of our hosting brand, aWh</a>.</li>
</ul>

<p>The runner connects outwards to GitLab (it asks over HTTPS whether there is a job to run), so you don’t need to open any inbound port for it. It is enough if it can reach the GitLab instance.</p>

<h2 id="choosing-the-executor">Choosing the executor</h2>

<p>The executor determines where and how a job runs. The most common ones:</p>

<table>
  <thead>
    <tr>
      <th>Executor</th>
      <th>Where does the job run?</th>
      <th>When is it a good fit?</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">shell</code></td>
      <td>directly on the runner machine, as the <code class="language-plaintext highlighter-rouge">gitlab-runner</code> user</td>
      <td>simple deployment scripts on a dedicated machine</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">docker</code></td>
      <td>each job in a fresh Docker container</td>
      <td>in most cases; this is our default choice</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">kubernetes</code></td>
      <td>in pods in a Kubernetes cluster</td>
      <td>if you already have a cluster and need to scale</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">docker-autoscaler</code></td>
      <td>on cloud machines started on demand</td>
      <td>for variable, heavy workloads</td>
    </tr>
  </tbody>
</table>

<p>The advantage of the Docker executor is that every job starts in a clean environment, the tools it needs come with the image the job specifies, and the jobs don’t clutter up the runner machine. This is what we set up here.</p>

<h2 id="1-installing-docker">1. Installing Docker</h2>

<p>The Docker executor needs Docker on the machine. We described the installation from the official repository step by step in the <a href="https://www.awh.hu/en/kb/linux/install-docker-ubuntu-debian-almalinux">aWh knowledge base</a>. To check:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>docker run <span class="nt">--rm</span> hello-world
</code></pre></div></div>

<h2 id="2-installing-gitlab-runner">2. Installing GitLab Runner</h2>

<p>GitLab provides its own package repository, and the install script adds it to the system. The <a href="https://docs.gitlab.com/runner/install/linux-repository/">documentation</a> recommends that you review the script before running it:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-L</span> <span class="s2">"https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh"</span> <span class="nt">-o</span> script.deb.sh
less script.deb.sh
<span class="nb">sudo </span>bash script.deb.sh
<span class="nb">sudo </span>apt <span class="nb">install </span>gitlab-runner

gitlab-runner <span class="nt">--version</span>
systemctl status gitlab-runner
</code></pre></div></div>

<p>On RHEL-based systems (AlmaLinux, Rocky Linux), use the <code class="language-plaintext highlighter-rouge">script.rpm.sh</code> script and <code class="language-plaintext highlighter-rouge">dnf install gitlab-runner</code> instead.</p>

<p>The runner’s version should ideally match GitLab’s major and minor version. GitLab.com always runs the latest release (19.4 came out yesterday, 17 September), and the package repository always installs the latest runner to match. With a self-managed GitLab, update the runner together with GitLab.</p>

<h2 id="3-creating-the-runner-in-gitlab">3. Creating the runner in GitLab</h2>

<p>These days you create runners in the GitLab interface, and you get a <strong>runner authentication token</strong> for each one, starting with the <code class="language-plaintext highlighter-rouge">glrt-</code> prefix. The old project- or group-level <strong>registration token</strong> is deprecated, and GitLab plans to remove it in version 20.0; don’t use it for new runners.</p>

<p>For a project runner:</p>

<ol>
  <li>Open the project, then <strong>Settings › CI/CD</strong>, and expand the <strong>Runners</strong> section.</li>
  <li>Select <strong>Create project runner</strong>.</li>
  <li>Enter the <strong>tags</strong>, for example <code class="language-plaintext highlighter-rouge">docker, linux</code>. With these you can specify in <code class="language-plaintext highlighter-rouge">.gitlab-ci.yml</code> which jobs should run on this runner. If it should also run untagged jobs, tick the <strong>Run untagged jobs</strong> option.</li>
  <li>Enter a description, then <strong>Create runner</strong>.</li>
  <li>On the next page you get the registration command with the token. You only see the token for a short time, so use it straight away.</li>
</ol>

<p>You can create a group runner in the group under <strong>Build › Runners</strong>, and one for the whole instance under <strong>Admin › CI/CD › Runners</strong>.</p>

<p>In the new workflow, the tags, the range of jobs it can run and the maximum job timeout are set in the GitLab interface, not in the registration command.</p>

<h2 id="4-registering-on-the-server">4. Registering on the server</h2>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>gitlab-runner register <span class="se">\</span>
  <span class="nt">--non-interactive</span> <span class="se">\</span>
  <span class="nt">--url</span> <span class="s2">"https://gitlab.com"</span> <span class="se">\</span>
  <span class="nt">--token</span> <span class="s2">"glrt-..."</span> <span class="se">\</span>
  <span class="nt">--executor</span> <span class="s2">"docker"</span> <span class="se">\</span>
  <span class="nt">--docker-image</span> <span class="s2">"alpine:latest"</span> <span class="se">\</span>
  <span class="nt">--description</span> <span class="s2">"docker-runner-01"</span>
</code></pre></div></div>

<p>With a self-managed GitLab, <code class="language-plaintext highlighter-rouge">--url</code> is the address of your own instance. To check:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>gitlab-runner list
<span class="nb">sudo </span>gitlab-runner verify
</code></pre></div></div>

<p>On the project’s <strong>Settings › CI/CD › Runners</strong> page, a green indicator should appear next to the runner.</p>

<h2 id="5-fine-tuning-configtoml">5. Fine-tuning config.toml</h2>

<p>The settings are in the <code class="language-plaintext highlighter-rouge">/etc/gitlab-runner/config.toml</code> file. A useful starting point:</p>

<div class="language-toml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">concurrent</span> <span class="o">=</span><span class="w"> </span><span class="mi">4</span>          <span class="c"># this many jobs can run on the machine at once</span>
<span class="n">check_interval</span> <span class="o">=</span><span class="w"> </span><span class="mi">3</span>

<span class="k">[[</span><span class="n">runners</span><span class="k">]]</span>
  <span class="n">name</span> <span class="o">=</span><span class="w"> </span><span class="s">"docker-runner-01"</span>
  <span class="n">url</span> <span class="o">=</span><span class="w"> </span><span class="s">"https://gitlab.com"</span>
  <span class="n">token</span> <span class="o">=</span><span class="w"> </span><span class="s">"glrt-..."</span>
  <span class="n">executor</span> <span class="o">=</span><span class="w"> </span><span class="s">"docker"</span>
  <span class="k">[</span><span class="n">runners</span><span class="k">.</span><span class="n">docker</span><span class="k">]</span>
    <span class="n">image</span> <span class="o">=</span><span class="w"> </span><span class="s">"alpine:latest"</span>
    <span class="n">privileged</span> <span class="o">=</span><span class="w"> </span><span class="kc">false</span>
    <span class="n">pull_policy</span> <span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="s">"if-not-present"</span><span class="p">]</span>
    <span class="n">volumes</span> <span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="s">"/cache"</span><span class="p">]</span>
    <span class="n">shm_size</span> <span class="o">=</span><span class="w"> </span><span class="mi">268435456</span>
</code></pre></div></div>

<p>Adjust the <code class="language-plaintext highlighter-rouge">concurrent</code> value to the machine’s resources: on a 4 vCPU machine, 2–4 parallel jobs is realistic. The runner detects changes by itself, but a <code class="language-plaintext highlighter-rouge">sudo gitlab-runner restart</code> doesn’t hurt either.</p>

<h3 id="building-docker-images-in-the-pipeline">Building Docker images in the pipeline</h3>

<p>If you build Docker images in your jobs, there are two ways. Docker-in-Docker (the <code class="language-plaintext highlighter-rouge">docker:dind</code> service) requires the <code class="language-plaintext highlighter-rouge">privileged = true</code> setting, which effectively gives the job root access on the runner machine. Mounting the host’s Docker socket (<code class="language-plaintext highlighter-rouge">/var/run/docker.sock</code>) is just as risky. A safer alternative is rootless image building, for example with Buildah or BuildKit’s rootless mode. Only assign a privileged runner to projects you trust.</p>

<h2 id="6-the-first-job">6. The first job</h2>

<p>The <code class="language-plaintext highlighter-rouge">.gitlab-ci.yml</code> in the root of the project:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">test</span><span class="pi">:</span>
  <span class="na">image</span><span class="pi">:</span> <span class="s">php:8.4-cli</span>
  <span class="na">tags</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="s">docker</span>
  <span class="na">script</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="s">php -v</span>
    <span class="pi">-</span> <span class="s">php -l index.php</span>
</code></pre></div></div>

<p>After a push, you can see on the <strong>Build › Pipelines</strong> page whether the job runs on your own runner.</p>

<h2 id="security-and-maintenance">Security and maintenance</h2>

<ul>
  <li><strong>Don’t run untrusted code with the shell executor.</strong> With the shell executor, anyone who can start a pipeline can run commands on the runner machine.</li>
  <li><strong>Protected runner, protected variables.</strong> Mark the runners that do deployments as <em>protected</em>, so only pipelines on protected branches (for example <code class="language-plaintext highlighter-rouge">main</code>) can use them, and secret variables can’t leak out from an arbitrary branch.</li>
  <li><strong>One runner, one purpose.</strong> The runner with access to production deployments should be separate from the one running development jobs.</li>
  <li><strong>Clean-up.</strong> Docker images and the cache fill up the disk within weeks. A weekly <code class="language-plaintext highlighter-rouge">docker system prune</code> solves this. You can schedule it with a systemd timer or a crontab entry; we described the crontab syntax in the <a href="https://www.awh.hu/en/kb/linux/crontab-syntax-and-scheduled-tasks-on-linux">aWh knowledge base</a>.</li>
  <li><strong>Updates.</strong> Update the runner together with the system updates (<code class="language-plaintext highlighter-rouge">sudo apt update &amp;&amp; sudo apt install gitlab-runner</code>).</li>
</ul>

<p>If you manage several runners or several environments, it’s worth building the runner machine from code too: the virtual machine with Terraform or OpenTofu, for example, and the installation and <code class="language-plaintext highlighter-rouge">config.toml</code> with Ansible.</p>

<p>We also help with building and operating CI/CD environments, GitLab runners and deployment processes; you can read more on our <a href="/en/services/cloud-devops">Cloud and DevOps</a> page.</p>]]></content><author><name>CSB IT Bt.</name><uri>https://www.csb-it.hu</uri></author><category term="Development" /><category term="gitlab" /><category term="docker" /><category term="devops" /><summary type="html"><![CDATA[Installing and registering GitLab Runner on your Linux server with the Docker executor: creating the runner, glrt- token, config.toml, security, maintenance.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://www.csb-it.hu/assets/images/og/en/gitlab-runner-telepitese.jpg" /><media:content medium="image" url="https://www.csb-it.hu/assets/images/og/en/gitlab-runner-telepitese.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry xml:lang="en"><title type="html">Debian 11 EOL: LTS ends on 31 August</title><link href="https://www.csb-it.hu/en/blog/debian-11-eol" rel="alternate" type="text/html" title="Debian 11 EOL: LTS ends on 31 August" /><published>2026-08-07T05:00:00+00:00</published><updated>2026-11-22T16:47:55+00:00</updated><id>https://www.csb-it.hu/en/blog/debian-11-eol</id><content type="html" xml:base="https://www.csb-it.hu/blog/debian-11-eol"><![CDATA[<p>Long-term support (LTS) for <strong>Debian 11</strong> (“Bullseye”) <a href="https://wiki.debian.org/LTS">ends on 31 August 2026</a>. After that, the Debian project will release no more security updates for it. Your servers won’t stop because of this, but every vulnerability found from then on will stay unpatched on them. If you still run Debian 11, you have three weeks to decide before <strong>Debian 11 EOL</strong>.</p>

<h2 id="how-did-we-get-here">How did we get here?</h2>

<p>Debian supports each release for about three years through the security team; after that, the LTS team – made up of volunteers and sponsoring companies – provides security fixes for the most important packages and architectures for another two years.</p>

<table>
  <thead>
    <tr>
      <th>Release</th>
      <th>Released</th>
      <th>End of regular support</th>
      <th>End of LTS</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Debian 10 “Buster”</td>
      <td>July 2019</td>
      <td>June 2022</td>
      <td>30 June 2024</td>
    </tr>
    <tr>
      <td><strong>Debian 11 “Bullseye”</strong></td>
      <td>14 August 2021</td>
      <td>14 August 2024</td>
      <td><strong>31 August 2026</strong></td>
    </tr>
    <tr>
      <td>Debian 12 “Bookworm”</td>
      <td>10 June 2023</td>
      <td>July 2026</td>
      <td>30 June 2028</td>
    </tr>
    <tr>
      <td>Debian 13 “Trixie”</td>
      <td>9 August 2025</td>
      <td>expected 2028</td>
      <td>expected 30 June 2030</td>
    </tr>
  </tbody>
</table>

<p>Regular support for Debian 12 ended in July, and <a href="https://www.debian.org/News/2026/20260712">since July the security updates come from the LTS team</a>, until 30 June 2028. The current stable release is Debian 13, and its latest point release is 13.6 from July.</p>

<h2 id="what-does-this-mean-in-practice">What does this mean in practice?</h2>

<ul>
  <li><strong>No more security fixes.</strong> For bugs found in OpenSSH, OpenSSL, the kernel or PHP after 31 August, no updates will come for Debian 11. On a server that is reachable from the internet, this is the biggest risk.</li>
  <li><strong>Ageing software.</strong> Debian 11 ships PHP 7.4, whose upstream support already ended in 2022; until now only the Debian LTS team’s fixes kept it alive. Fewer and fewer newer web applications, plugins and frameworks run on it.</li>
  <li><strong>External repositories.</strong> Third-party package sources (databases, monitoring agents, Docker) are dropping Bullseye one after another, so these won’t be updated any more either.</li>
  <li><strong>Compliance.</strong> If the machine holds personal or payment data, most auditors won’t accept an unsupported operating system.</li>
</ul>

<h2 id="how-do-you-find-the-affected-machines">How do you find the affected machines?</h2>

<p>On one machine:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span> /etc/debian_version     <span class="c"># for example 11.11</span>
lsb_release <span class="nt">-a</span>              <span class="c"># if installed</span>
</code></pre></div></div>

<p>If you manage several machines with Ansible, you can query the major version of all of them with a single command (look for the ones with the value <code class="language-plaintext highlighter-rouge">"11"</code>):</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ansible all <span class="nt">-m</span> ansible.builtin.setup <span class="nt">-a</span> <span class="s2">"filter=ansible_distribution_major_version"</span>
</code></pre></div></div>

<p>Don’t forget the less visible places either: old virtual machines, LXC containers on a Proxmox server, Docker images built on a <code class="language-plaintext highlighter-rouge">debian:bullseye</code> or <code class="language-plaintext highlighter-rouge">php:7.4-*</code> base, and appliances on which the vendor shipped Debian 11.</p>

<h2 id="what-are-your-options">What are your options?</h2>

<h3 id="upgrading-to-debian-12-then-to-13">Upgrading to Debian 12, then to 13</h3>

<p>For most servers this is the right direction. You can’t skip a release: you have to go from 11 to 12, and then from 12 to 13; the two can’t be combined. Since Debian 12 is already in LTS too, it’s worth going through both steps in one go and leaving the machine on Debian 13.</p>

<p>Over the two release upgrades, the most important components change like this:</p>

<table>
  <thead>
    <tr>
      <th>Component</th>
      <th>Debian 11</th>
      <th>Debian 12</th>
      <th>Debian 13</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Linux kernel</td>
      <td>5.10</td>
      <td>6.1</td>
      <td>6.12</td>
    </tr>
    <tr>
      <td>PHP</td>
      <td>7.4</td>
      <td>8.2</td>
      <td>8.4</td>
    </tr>
    <tr>
      <td>Python</td>
      <td>3.9</td>
      <td>3.11</td>
      <td>3.13</td>
    </tr>
    <tr>
      <td>MariaDB</td>
      <td>10.5</td>
      <td>10.11</td>
      <td>11.8</td>
    </tr>
    <tr>
      <td>PostgreSQL</td>
      <td>13</td>
      <td>15</td>
      <td>17</td>
    </tr>
    <tr>
      <td>OpenSSH</td>
      <td>8.4</td>
      <td>9.2</td>
      <td>10.0</td>
    </tr>
  </tbody>
</table>

<p>The jump from PHP 7.4 to 8.2 and then 8.4 is the riskiest part: an old web application often doesn’t run on the newer PHP. It’s worth trying this on a test machine before the upgrade.</p>

<h3 id="extended-support-elts">Extended support (ELTS)</h3>

<p>With the paid <a href="https://www.freexian.com/lts/extended/">Extended LTS</a> service from a company called Freexian, you can get security fixes for some Debian 11 packages until 30 June 2031, on a limited set of architectures and packages. It is meant for buying time if a machine can’t be upgraded right now for some reason, not for postponing the upgrade.</p>

<h3 id="replacement-or-isolation">Replacement or isolation</h3>

<p>If only one or two services run on a machine, it is often simpler to install a fresh Debian 13 and move the data over to it. Where none of this is possible (for example because of an old application that is no longer developed), at least isolate the machine: a separate network segment, no internet access, only the necessary connections, and regular backups.</p>

<h2 id="upgrading-from-debian-11-to-12-step-by-step">Upgrading from Debian 11 to 12, step by step</h2>

<p>The steps below follow the Debian 12 <a href="https://www.debian.org/releases/bookworm/amd64/release-notes/ch-upgrading.en.html">release notes</a>. Before you start, make a full backup or snapshot, and if you can, work with console access or in a <code class="language-plaintext highlighter-rouge">screen</code>/<code class="language-plaintext highlighter-rouge">tmux</code> session.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># 1. full update of the current system</span>
apt update <span class="o">&amp;&amp;</span> apt full-upgrade
apt autoremove

<span class="c"># 2. switch the repositories to bookworm</span>
<span class="nb">sed</span> <span class="nt">-i</span> <span class="s1">'s/bullseye/bookworm/g'</span> /etc/apt/sources.list /etc/apt/sources.list.d/<span class="k">*</span>.list

<span class="c"># 3. minimal upgrade, then the full one</span>
apt update
apt upgrade <span class="nt">--without-new-pkgs</span>
apt full-upgrade

<span class="c"># 4. reboot and check</span>
reboot
<span class="nb">cat</span> /etc/debian_version
</code></pre></div></div>

<p>If the machine has <code class="language-plaintext highlighter-rouge">non-free</code> firmware packages, since Debian 12 these are in the separate <code class="language-plaintext highlighter-rouge">non-free-firmware</code> component; add it to the lines in <code class="language-plaintext highlighter-rouge">sources.list</code>. In Debian 12, the line for the security repository looks like this:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>deb http://security.debian.org/debian-security bookworm-security main contrib non-free-firmware
</code></pre></div></div>

<p>During the upgrade the package manager may ask whether you want to keep modified configuration files. If you aren’t sure, keep your own, and compare it with the <code class="language-plaintext highlighter-rouge">.dpkg-dist</code> file afterwards.</p>

<h2 id="upgrading-from-debian-12-to-13">Upgrading from Debian 12 to 13</h2>

<p>The second step works the same way, with <code class="language-plaintext highlighter-rouge">trixie</code> instead of <code class="language-plaintext highlighter-rouge">bookworm</code>. Also note the new deb822 format for repository definitions; the <code class="language-plaintext highlighter-rouge">apt modernize-sources</code> command converts the old files to it. We also described the steps of a major version upgrade in the <a href="https://www.awh.hu/en/kb/linux/updating-ubuntu-and-debian-to-a-new-major-version">knowledge base</a> of our hosting brand, aWh.</p>

<h2 id="checklist-before-31-august">Checklist before 31 August</h2>

<ol>
  <li>Make an inventory of every Debian 11 machine, container and image.</li>
  <li>Prioritise them: which one is reachable from the internet, which one handles sensitive data.</li>
  <li>Decide for each machine: upgrade, reinstall, ELTS or isolation.</li>
  <li>Test the applications with the new PHP, Python and database versions.</li>
  <li>Make a backup, and test the restore too.</li>
  <li>Upgrade a test machine first, then the production ones.</li>
  <li>After the upgrade, go through the logs (<code class="language-plaintext highlighter-rouge">journalctl -p err -b</code>) and the services (<code class="language-plaintext highlighter-rouge">systemctl --failed</code>).</li>
</ol>

<p>We have written about similar situations before: the same steps helped at the time of <a href="/en/blog/centos-7-eol">CentOS 7 EOL</a> and <a href="/en/blog/ubuntu-20-04-eol">Ubuntu 20.04 EOL</a>. We also help with assessing, upgrading and migrating end-of-life Linux servers; you can read more on our <a href="/en/services/infrastructure">Infrastructure and operations</a> page.</p>]]></content><author><name>CSB IT Bt.</name><uri>https://www.csb-it.hu</uri></author><category term="Sysadmin" /><category term="linux" /><category term="eol" /><summary type="html"><![CDATA[Debian 11 “Bullseye” LTS support ends on 31 August 2026. What does it mean, what are your options, and how do you upgrade to Debian 12 and then to 13?]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://www.csb-it.hu/assets/images/og/en/debian-11-eol.jpg" /><media:content medium="image" url="https://www.csb-it.hu/assets/images/og/en/debian-11-eol.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry xml:lang="en"><title type="html">What is a postmortem? Blameless incident analysis in IT operations</title><link href="https://www.csb-it.hu/en/blog/what-is-a-postmortem" rel="alternate" type="text/html" title="What is a postmortem? Blameless incident analysis in IT operations" /><published>2026-07-17T05:00:00+00:00</published><updated>2026-11-22T16:47:55+00:00</updated><id>https://www.csb-it.hu/en/blog/what-is-a-postmortem</id><content type="html" xml:base="https://www.csb-it.hu/blog/what-is-a-postmortem"><![CDATA[<p>The term comes from medicine, but in IT it has nothing to do with autopsies: in IT operations, a <strong>postmortem</strong> is a written analysis after an incident that describes what happened, what impact it had, why it occurred, and what we are changing so that it doesn’t happen again. You may also come across it as an incident review or post-incident review (PIR). It isn’t written during troubleshooting but afterwards, in calm conditions – and it doesn’t look for a scapegoat, but for things that can be fixed.</p>

<p>The big providers regularly publish their postmortems, and we have already written about several of them. In this article we look at how to write a postmortem for your own systems in a smaller team.</p>

<h2 id="what-does-a-postmortem-mean-in-practice">What does a postmortem mean in practice?</h2>

<p>A good postmortem answers three questions:</p>

<ol>
  <li><strong>What happened, and who was affected?</strong> Facts, timeline, impact.</li>
  <li><strong>Why did it happen?</strong> Root cause and contributing factors.</li>
  <li><strong>What are we doing about it?</strong> Specific tasks with an owner and a deadline.</li>
</ol>

<p>If the third part is missing, the document is just a story. If the first two are incomplete, the tasks have no solid basis.</p>

<h2 id="a-blameless-culture">A blameless culture</h2>

<p>According to the <a href="https://sre.google/sre-book/postmortem-culture/">chapter on postmortems</a> in Google’s SRE book, a blameless postmortem identifies the contributing causes of the incident without indicting any individual or team, and assumes that everyone involved had good intentions and acted on the information they had. There is a practical reason for this: if colleagues fear that the document will be used to hold someone accountable, they keep quiet about the details, and exactly the ones you could learn from are lost.</p>

<p>The difference shows in the wording. Instead of “The colleague restarted the wrong server”: “The names of the two servers differed by one character, and there was no confirmation step before the restart.” The first blames a person, the second points to a process that can be fixed. If someone was able to make a mistake, the system allowed it – that is what needs to change.</p>

<h2 id="when-should-you-write-a-postmortem">When should you write a postmortem?</h2>

<p>You don’t need one after every alert. It’s worth setting the thresholds in advance, so you don’t have to argue about it after the incident. Google’s SRE book lists these typical cases:</p>

<ul>
  <li>a user-visible outage or slowdown above a defined threshold;</li>
  <li>any data loss;</li>
  <li>on-call intervention (rolling back to an earlier version, rerouting traffic);</li>
  <li>a recovery time longer than a threshold;</li>
  <li>a monitoring failure, that is, when the incident was noticed by a person, not by an alert.</li>
</ul>

<p>At a small company, a good starting point is: after every outage affecting clients that lasts longer than 30 minutes, every data loss and every security incident. “Near misses” – for example a backup that turns out not to have run for weeks – are also worth writing up; they are the cheapest lessons.</p>

<h2 id="the-structure-of-a-postmortem">The structure of a postmortem</h2>

<table>
  <thead>
    <tr>
      <th>Section</th>
      <th>What goes in it?</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Summary</td>
      <td>3–5 sentences: what happened, when, for how long, what the cause was, what the main action is</td>
    </tr>
    <tr>
      <td>Impact</td>
      <td>Affected services and clients, duration, data loss, SLA impact</td>
    </tr>
    <tr>
      <td>Timeline</td>
      <td>Times in one time zone: first sign, detection, interventions, recovery</td>
    </tr>
    <tr>
      <td>Root cause and contributing factors</td>
      <td>The direct technical cause, and every circumstance that made it possible or made it worse</td>
    </tr>
    <tr>
      <td>What went well?</td>
      <td>Anything that sped up detection or recovery</td>
    </tr>
    <tr>
      <td>What didn’t work?</td>
      <td>Where we lost time, what was missing</td>
    </tr>
    <tr>
      <td>Action items</td>
      <td>Specific tasks with an owner, deadline and priority</td>
    </tr>
  </tbody>
</table>

<p>A few practical tips:</p>

<ul>
  <li><strong>Timeline:</strong> use a single, stated time zone. Put it together from the timestamps of alerts, chat messages and logs, not from memory. The time between the first sign and detection often says more than the fault itself.</li>
  <li><strong>Root cause:</strong> there is rarely only one. The “5 whys” method is a good start, but in real incidents several factors typically come together: a bug in the code, a missing test, an alert that didn’t fire.</li>
  <li><strong>Action items:</strong> “we’ll pay more attention” is not an action item. An action item is good if it can be verified: “Alert if the daily backup hasn’t run for 26 hours – owner: X, deadline: 14 August.” Track the action items in your existing ticketing system, not just in the document, otherwise they get forgotten.</li>
</ul>

<h2 id="a-short-template">A short template</h2>

<div class="language-markdown highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gh"># Postmortem: [short title] – [date]</span>

Status: draft / reviewed / closed
Author: ...        Participants: ...

<span class="gu">## Summary</span>

<span class="gu">## Impact</span>
<span class="p">-</span> Affected service(s):
<span class="p">-</span> Duration: [start] – [end] (CEST)
<span class="p">-</span> Affected clients, users:
<span class="p">-</span> Data loss:

<span class="gu">## Timeline</span>
| Time | Event |
|------|-------|

<span class="gu">## Root cause and contributing factors</span>

<span class="gu">## What went well?</span>

<span class="gu">## What didn’t work?</span>

<span class="gu">## Action items</span>
| Action item | Owner | Deadline | Ticket |
|-------------|-------|----------|--------|
</code></pre></div></div>

<p>Make the review a mandatory step. According to Google’s SRE book, “an unreviewed postmortem might as well never have existed.” In a small team this can be a half-hour meeting in the week after the incident, with the owners of the action items present.</p>

<h2 id="what-can-we-learn-from-public-postmortems">What can we learn from public postmortems?</h2>

<p>The big providers’ public analyses are a good model of what a thorough postmortem looks like.</p>

<p><strong>CrowdStrike, July 2024.</strong> After the <a href="/en/blog/crowdstrike-leallas-tanulsagai">faulty Falcon update</a>, the company published a preliminary review five days later, and the <a href="https://www.crowdstrike.com/en-us/blog/channel-file-291-rca-available/">detailed root cause analysis</a> on 6 August. This is a good example of how the first version can be incomplete, as long as it says what is still to come. According to the root cause analysis, the new template type defined 21 input fields, but the sensor only passed 20; the content interpreter didn’t check the bounds, and the pre-release validator didn’t catch the error either. Several contributing factors together, rather than a single fault.</p>

<p><strong>AWS, October 2025.</strong> The summary of the <a href="/en/blog/aws-us-east-1-leallas">us-east-1 outage</a> gives a timeline accurate to the minute, and it doesn’t stop at the direct cause, a race condition in DynamoDB’s DNS automation: it describes how the fault rippled on into EC2 and the Network Load Balancer. The action items are specific: the DNS automation was disabled worldwide until the fix, and for the NLB they are limiting how much capacity health check failures can remove. It doesn’t name a person anywhere.</p>

<p><strong>Cloudflare, November 2025.</strong> The analysis of the <a href="/en/blog/cloudflare-leallas">Cloudflare outage</a> came out the same day, with a timeline table, a code snippet and an apology. The most important part is the list of action items: they will validate their own generated configuration files the same way as user input, and they are building in more global kill switches. This rules out not just that particular fault, but a whole class of faults.</p>

<p>What they have in common is that they are factual, they don’t make excuses, and the action items are systemic.</p>

<h2 id="common-mistakes">Common mistakes</h2>

<ul>
  <li><strong>It is finished too late.</strong> After two weeks the details are already lost; the first draft should ideally be ready within a few days.</li>
  <li><strong>The action items have no owner</strong>, so they never get done.</li>
  <li><strong>It only looks for the technical cause</strong>, not the process (notification, escalation, documentation, access).</li>
  <li><strong>Nobody reads it.</strong> Put the postmortems in a shared, searchable place (a wiki or a Git repository), and when a similar incident happens, check what became of the earlier action items.</li>
</ul>

<p>A postmortem is worth the time spent on it if the next incident is shorter, or doesn’t happen at all. If you need help setting up the incident management process, monitoring or the on-call rota, take a look at our <a href="/en/services/cloud-devops">Cloud and DevOps</a> services.</p>]]></content><author><name>CSB IT Bt.</name><uri>https://www.csb-it.hu</uri></author><category term="Sysadmin" /><category term="devops" /><summary type="html"><![CDATA[What does a postmortem mean in IT operations, when should you write one, what goes into it, and why blameless? With a ready-to-use template and public examples.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://www.csb-it.hu/assets/images/og/en/postmortem-jelentese.jpg" /><media:content medium="image" url="https://www.csb-it.hu/assets/images/og/en/postmortem-jelentese.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry xml:lang="en"><title type="html">Proxmox storage: ZFS, LVM-thin, Ceph or NFS – which one should you choose?</title><link href="https://www.csb-it.hu/en/blog/proxmox-storage-zfs-lvm-ceph" rel="alternate" type="text/html" title="Proxmox storage: ZFS, LVM-thin, Ceph or NFS – which one should you choose?" /><published>2026-06-12T05:00:00+00:00</published><updated>2026-11-22T16:47:55+00:00</updated><id>https://www.csb-it.hu/en/blog/proxmox-storage-zfs-lvm-ceph</id><content type="html" xml:base="https://www.csb-it.hu/blog/proxmox-storage-zfs-lvm-ceph"><![CDATA[<p>When you install a new Proxmox server, one of the first decisions – and later one of the hardest to change – is the choice of <strong>Proxmox storage</strong>. The installer offers ext4 with LVM-thin, ZFS and Btrfs, and in a cluster there are also Ceph and external NFS or iSCSI storage. They all work, but they differ in what they can do with snapshots, live migration and high availability (HA), and they need very different hardware.</p>

<p>In this article, based on Proxmox VE 9.2 released on 21 May (Debian 13.5, ZFS 2.4, and by default Ceph 20.2 “Tentacle” on new installations), we compare the most common storage types and show what we recommend for one-, two- and three-server environments.</p>

<h2 id="lvm-thin-the-simple-default">LVM-thin: the simple default</h2>

<p>If you choose ext4 or XFS in the installer, Proxmox creates an LVM-thin storage called <code class="language-plaintext highlighter-rouge">local-lvm</code> on the rest of the system disk. The virtual machines’ disks are logical volumes on it, with thin provisioning: a 100 GB virtual disk only takes up as much space as the machine has actually written.</p>

<ul>
  <li><strong>Advantage:</strong> fast, needs little memory and CPU, and supports snapshots.</li>
  <li><strong>Disadvantage:</strong> it has no redundancy of its own, so it needs hardware RAID underneath to protect against disk failures. It isn’t shared, and Proxmox’s replication feature doesn’t work with it either.</li>
</ul>

<p>It is a good choice if you are dealing with a single server that already has a reliable RAID controller with a battery-backed cache.</p>

<h2 id="zfs-local-storage-with-lots-of-extras">ZFS: local storage with lots of extras</h2>

<p>ZFS is a volume manager and a file system at the same time: it manages the disks itself, protects the data with checksums, compresses, and takes snapshots in an instant. The Proxmox installer can also install directly onto ZFS, as a mirror (RAID 1/10) or in RAID-Z.</p>

<p>The RAID-Z levels are the equivalents of RAID 5 and 6 (we wrote about the basics in <a href="/en/blog/raid-szintek">our article on RAID levels</a>), but we rarely choose them for virtual machines. According to the <a href="https://pve.proxmox.com/wiki/ZFS_on_Linux">Proxmox documentation</a>, in terms of IOPS a RAID-Z vdev behaves roughly like a single disk, whereas mirrored vdevs are better suited to VM workloads. So from four disks we usually build two mirrored pairs (RAID 10).</p>

<p>A few things to watch out for:</p>

<ul>
  <li><strong>Memory.</strong> The ZFS cache, the ARC, lives in RAM. Since Proxmox VE 8.1 the installer limits the ARC to 10% of memory, at most 16 GiB; as a rule of thumb it needs at least 2 GiB plus 1 GiB per terabyte stored. You have to account for this when planning the virtual machines’ memory.</li>
  <li><strong>Hardware.</strong> A RAID controller with its own cache doesn’t belong under ZFS; you need an HBA or a controller set to “IT mode”, and the documentation recommends ECC memory.</li>
  <li><strong>SSDs.</strong> In our experience, ZFS’s write pattern wears out cheap consumer SSDs quickly, and they are slow with synchronous writes; in servers we use enterprise SSDs with power loss protection (PLP).</li>
</ul>

<p>ZFS’s biggest advantage in a cluster is <strong>storage replication</strong>: on a schedule (every 15 minutes by default, but as often as every minute), Proxmox copies the changes to the machines’ disks to a ZFS pool with the same name on another node. This way the machine can be migrated quickly, and HA can also restart it on the other node – losing the data written since the last replication.</p>

<h2 id="ceph-distributed-storage-for-three-or-more-nodes">Ceph: distributed storage for three or more nodes</h2>

<p>Ceph builds a single, shared and replicated storage from the nodes’ local disks (OSDs). By default every piece of data exists in three copies, on three different nodes, so even the loss of a whole server causes no data loss, and HA restarts the machines immediately, from the most recent state.</p>

<p>This comes at a price. According to <a href="https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster">Proxmox’s Ceph guide</a>:</p>

<ul>
  <li>you need at least <strong>three</strong> servers, preferably identical ones, and at least 12 OSDs in total are recommended, evenly distributed,</li>
  <li>Ceph traffic needs a <strong>dedicated network of at least 10 Gbit/s</strong>, and 25 Gbit/s for high-performance systems,</li>
  <li>at least 8 GiB of memory per OSD is recommended for good performance, and at least one CPU core for each Ceph service,</li>
  <li>here too you need an HBA instead of a RAID controller, and in small systems SSDs for faster recovery.</li>
</ul>

<p>Because of the threefold replication, only about a third of the raw capacity is usable. With three to five nodes, the machines can also be connected without a switch, in a full-mesh network.</p>

<h2 id="shared-nfs-and-iscsi">Shared NFS and iSCSI</h2>

<p>If you already have a separate storage device (NAS or SAN), Proxmox can also access it over NFS, CIFS or iSCSI. The storage is visible from every node, so live migration and HA are simple.</p>

<ul>
  <li><strong>NFS:</strong> the disks are qcow2 files, and snapshots are possible, but according to the documentation internal qcow2 snapshots are slow on network storage, and on a large disk they can take minutes or even hours.</li>
  <li><strong>iSCSI or Fibre Channel with LVM:</strong> block-level, shared storage. Snapshots were missing for a long time; since Proxmox VE 9.0 they are possible in “snapshot as volume chain” mode, but still as a technology preview.</li>
  <li><strong>ZFS over iSCSI:</strong> shared and snapshot-capable, but it only works with certain storage systems that run ZFS.</li>
</ul>

<p>The drawback of external storage is that if it isn’t redundant, it becomes the single point of failure itself: if it goes down, every machine goes down.</p>

<h2 id="what-can-each-proxmox-storage-type-do">What can each Proxmox storage type do?</h2>

<table>
  <thead>
    <tr>
      <th> </th>
      <th>LVM-thin</th>
      <th>ZFS (local)</th>
      <th>Ceph RBD</th>
      <th>NFS</th>
      <th>iSCSI + LVM</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Shared</td>
      <td>no</td>
      <td>no</td>
      <td>yes</td>
      <td>yes</td>
      <td>yes</td>
    </tr>
    <tr>
      <td>Snapshot</td>
      <td>yes</td>
      <td>yes</td>
      <td>yes</td>
      <td>yes (qcow2)</td>
      <td>technology preview</td>
    </tr>
    <tr>
      <td>Live migration</td>
      <td>with disk copy, slow</td>
      <td>fast with replication</td>
      <td>immediate</td>
      <td>immediate</td>
      <td>immediate</td>
    </tr>
    <tr>
      <td>HA</td>
      <td>not in practice</td>
      <td>with replication, with data loss</td>
      <td>yes</td>
      <td>yes</td>
      <td>yes</td>
    </tr>
    <tr>
      <td>Replication</td>
      <td>no</td>
      <td>yes</td>
      <td>built in (3 copies)</td>
      <td>depends on the storage</td>
      <td>depends on the storage</td>
    </tr>
    <tr>
      <td>RAM/CPU requirements</td>
      <td>low</td>
      <td>medium (ARC)</td>
      <td>high</td>
      <td>low</td>
      <td>low</td>
    </tr>
    <tr>
      <td>Minimum</td>
      <td>1 node</td>
      <td>1 node</td>
      <td>3 nodes</td>
      <td>1 node + storage</td>
      <td>1 node + storage</td>
    </tr>
    <tr>
      <td>Typical use</td>
      <td>one server with hardware RAID</td>
      <td>1–2 nodes, small cluster</td>
      <td>3+ node HA</td>
      <td>existing NAS</td>
      <td>existing SAN</td>
    </tr>
  </tbody>
</table>

<h2 id="what-do-we-recommend">What do we recommend?</h2>

<p><strong>One server:</strong> a ZFS mirror with two enterprise SSDs, or RAID 10 from four disks if you need more space. If the machine already has good hardware RAID, LVM-thin is fine too. Whichever you choose, keep the backups on a separate machine: RAID and snapshots are not backups.</p>

<p><strong>Two servers:</strong> a ZFS pool with the same name on both nodes, storage replication for the important machines, and a QDevice running on a third machine for quorum. This is cheap, but asynchronous: it has to be acceptable that a few minutes of data may be lost in an outage. We described the details in <a href="/en/blog/proxmox-ha-klaszter">our article on the Proxmox HA cluster</a>. We don’t build Ceph on two nodes.</p>

<p><strong>Three or more servers:</strong> if you have the 10/25 Gbit/s network, several enterprise SSDs per node and enough memory, Ceph gives the best availability. If the hardware falls short of that, ZFS replication or redundant external storage is more realistic.</p>

<p>Storage is hard to change later: the machines have to be moved, which means downtime and a lot of data transfer. So it’s worth planning for the needs of the next three to five years from the start. If you are still on Proxmox VE 8, remember that its support is scheduled to end in August 2026; we wrote about the new features of major version 9 <a href="/en/blog/megjelent-a-proxmox-ve-9">when Proxmox VE 9.0 was released</a>.</p>

<p>We help with planning Proxmox servers and clusters, and with choosing and building their storage; you can read more on our <a href="/en/services/infrastructure">Infrastructure and operations</a> page.</p>]]></content><author><name>CSB IT Bt.</name><uri>https://www.csb-it.hu</uri></author><category term="Sysadmin" /><category term="proxmox" /><category term="virtualisation" /><summary type="html"><![CDATA[Which Proxmox storage suits one, two or three servers? Comparing LVM-thin, ZFS, Ceph and NFS/iSCSI by snapshots, live migration and high availability.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://www.csb-it.hu/assets/images/og/en/proxmox-tarolo-zfs-lvm-ceph.jpg" /><media:content medium="image" url="https://www.csb-it.hu/assets/images/og/en/proxmox-tarolo-zfs-lvm-ceph.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry xml:lang="en"><title type="html">Terraform or OpenTofu? Infrastructure as code in 2026</title><link href="https://www.csb-it.hu/en/blog/terraform-vs-opentofu" rel="alternate" type="text/html" title="Terraform or OpenTofu? Infrastructure as code in 2026" /><published>2026-05-15T05:00:00+00:00</published><updated>2026-11-22T16:47:55+00:00</updated><id>https://www.csb-it.hu/en/blog/terraform-vs-opentofu</id><content type="html" xml:base="https://www.csb-it.hu/blog/terraform-vs-opentofu"><![CDATA[<p>For years, <strong>Terraform</strong> has been the standard tool for managing infrastructure as code (IaC): you describe virtual machines, networks, DNS records and cloud services in text files, and the tool creates, changes or deletes them. Since 2023, however, it has had an open source fork, <strong>OpenTofu</strong>, and if you are starting out now or maintaining existing code, it is a fair question: which one should you use? In this article we compare the current state of the two tools.</p>

<h2 id="what-does-infrastructure-as-code-mean">What does infrastructure as code mean?</h2>

<p>Instead of clicking around by hand, the description of the infrastructure lives in a Git repository. This has three big advantages:</p>

<ul>
  <li><strong>Repeatable:</strong> the same code produces the same environment in test and in production.</li>
  <li><strong>Transparent:</strong> every change is a commit, you can see who changed what and why, and it can be reverted.</li>
  <li><strong>Verifiable:</strong> the <code class="language-plaintext highlighter-rouge">plan</code> step shows in advance what will change, before anything happens.</li>
</ul>

<p>A simple example that creates an S3 bucket in AWS. This code runs the same way with Terraform and OpenTofu:</p>

<div class="language-hcl highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">terraform</span> <span class="p">{</span>
  <span class="nx">required_providers</span> <span class="p">{</span>
    <span class="nx">aws</span> <span class="o">=</span> <span class="p">{</span>
      <span class="nx">source</span>  <span class="o">=</span> <span class="s2">"hashicorp/aws"</span>
      <span class="nx">version</span> <span class="o">=</span> <span class="s2">"~&gt; 6.0"</span>
    <span class="p">}</span>
  <span class="p">}</span>
<span class="p">}</span>

<span class="nx">provider</span> <span class="s2">"aws"</span> <span class="p">{</span>
  <span class="nx">region</span> <span class="o">=</span> <span class="s2">"eu-central-1"</span>
<span class="p">}</span>

<span class="nx">resource</span> <span class="s2">"aws_s3_bucket"</span> <span class="s2">"backup"</span> <span class="p">{</span>
  <span class="nx">bucket</span> <span class="o">=</span> <span class="s2">"example-company-backups"</span>
<span class="p">}</span>
</code></pre></div></div>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>terraform init <span class="o">&amp;&amp;</span> terraform plan    <span class="c"># or: tofu init &amp;&amp; tofu plan</span>
terraform apply                      <span class="c"># or: tofu apply</span>
</code></pre></div></div>

<h2 id="where-does-opentofu-come-from">Where does OpenTofu come from?</h2>

<p>In August 2023, HashiCorp changed Terraform’s licence from the open source MPL 2.0 to the Business Source License (BSL 1.1). Under the BSL, Terraform can be used freely to manage your own infrastructure, but it can’t be built into a product that competes with HashiCorp’s services. In response, several companies and developers forked the last MPL version; the project moved under the Linux Foundation as OpenTofu, and in January 2024 <a href="https://www.linuxfoundation.org/press/opentofu-announces-general-availability">the first stable release came out</a>, 1.6.0. In April 2025, OpenTofu became a sandbox project of the CNCF (Cloud Native Computing Foundation).</p>

<p>Meanwhile HashiCorp changed hands: IBM completed its acquisition of the company in February 2025. Terraform development continues under IBM.</p>

<h2 id="where-do-the-two-tools-stand-now">Where do the two tools stand now?</h2>

<p>Both projects are active, and they release new versions at a roughly similar pace:</p>

<ul>
  <li><strong>Terraform 1.15</strong> <a href="https://github.com/hashicorp/terraform/releases/tag/v1.15.0">was released on 29 April 2026</a>. Among other things, variables and local values can now be used in the <code class="language-plaintext highlighter-rouge">source</code> and <code class="language-plaintext highlighter-rouge">version</code> values of modules, variables and outputs can be marked <code class="language-plaintext highlighter-rouge">deprecated</code>, and the S3 backend supports <code class="language-plaintext highlighter-rouge">aws login</code> authentication.</li>
  <li><strong>OpenTofu 1.12</strong> <a href="https://opentofu.org/blog/opentofu-1-12-0/">was released yesterday, 14 May 2026</a>. <code class="language-plaintext highlighter-rouge">prevent_destroy</code> can now also be controlled from a variable, the new <code class="language-plaintext highlighter-rouge">destroy = false</code> lifecycle setting lets you remove a resource from the state without deleting it, and <code class="language-plaintext highlighter-rouge">tofu init</code> records the providers’ checksums for every platform.</li>
</ul>

<h2 id="the-differences-in-one-table">The differences in one table</h2>

<table>
  <thead>
    <tr>
      <th> </th>
      <th>Terraform</th>
      <th>OpenTofu</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Licence</td>
      <td>BSL 1.1 (source available, with restrictions)</td>
      <td>MPL 2.0 (open source)</td>
    </tr>
    <tr>
      <td>Developer</td>
      <td>HashiCorp (IBM)</td>
      <td>community, Linux Foundation / CNCF</td>
    </tr>
    <tr>
      <td>Command</td>
      <td><code class="language-plaintext highlighter-rouge">terraform</code></td>
      <td><code class="language-plaintext highlighter-rouge">tofu</code></td>
    </tr>
    <tr>
      <td>Configuration language</td>
      <td>HCL</td>
      <td>HCL, same syntax</td>
    </tr>
    <tr>
      <td>Providers</td>
      <td>Terraform Registry</td>
      <td>OpenTofu Registry, with the same providers</td>
    </tr>
    <tr>
      <td>State file encryption</td>
      <td>on the backend side (for example S3 encryption)</td>
      <td>built-in client-side encryption as well (since 1.7)</td>
    </tr>
    <tr>
      <td>Features unique to each</td>
      <td><code class="language-plaintext highlighter-rouge">terraform test</code>, queries (<code class="language-plaintext highlighter-rouge">query</code>), actions</td>
      <td><code class="language-plaintext highlighter-rouge">enabled</code> meta-argument, provider-level <code class="language-plaintext highlighter-rouge">for_each</code>, variables in the backend block</td>
    </tr>
    <tr>
      <td>Cloud platform</td>
      <td>HCP Terraform (paid plans and a free tier)</td>
      <td>none of its own; third-party or your own CI/CD</td>
    </tr>
  </tbody>
</table>

<p>The two tools split around versions 1.5–1.6. Since the split, both have added features that the other doesn’t have, or that work differently. Simpler, “traditional” configurations, however – resources, modules, variables, outputs – run the same way in both.</p>

<h2 id="what-matters-when-choosing">What matters when choosing?</h2>

<h3 id="licence-and-business-model">Licence and business model</h3>

<p>If you manage your own infrastructure or your clients’, the BSL doesn’t restrict you in practice. But if you are building a product or service that itself offers infrastructure automation, it’s worth having a lawyer review the BSL terms, and here OpenTofu is the safer choice.</p>

<h3 id="hcp-terraform-and-teamwork">HCP Terraform and teamwork</h3>

<p>HashiCorp’s cloud platform, HCP Terraform, provides state storage, runs, access management and policy checks. The old free plan ended on 31 March 2026, and accounts were moved to the <a href="https://www.hashicorp.com/blog/continuing-hcp-terraform-s-enhanced-free-tier-experience">new, limited free tier</a>. If your team uses it and is happy with it, Terraform remains the obvious choice. If runs happen in your own CI/CD system anyway (for example in GitLab CI, where GitLab can also store the state), OpenTofu fits in just as well.</p>

<h3 id="features">Features</h3>

<p>If you need a specific feature – with OpenTofu, for example, the built-in state encryption; with Terraform, the HCP integration or the latest provider capabilities – that may settle the question. It’s worth trying both on your own code before you commit.</p>

<h3 id="stability-and-future">Stability and future</h3>

<p>Both projects have serious backing: IBM behind Terraform, and a foundation and several companies building on it behind OpenTofu. Neither is going to disappear in the next few years.</p>

<h2 id="migrating-from-terraform-to-opentofu">Migrating from Terraform to OpenTofu</h2>

<p>If you use Terraform 1.5 or earlier, the migration is simple in most cases:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># 1. fresh state backup</span>
terraform state pull <span class="o">&gt;</span> state-backup.tfstate

<span class="c"># 2. check: according to Terraform there are no pending changes</span>
terraform plan

<span class="c"># 3. in the same directory, with OpenTofu</span>
tofu init
tofu plan      <span class="c"># ideally: "No changes."</span>
</code></pre></div></div>

<p>From a newer Terraform version, follow OpenTofu’s <a href="https://opentofu.org/docs/intro/migration/">migration guide</a>, because a feature that exists only in Terraform (for example certain <code class="language-plaintext highlighter-rouge">removed</code> or <code class="language-plaintext highlighter-rouge">import</code> blocks, or some parts of <code class="language-plaintext highlighter-rouge">terraform test</code>) may need changes. Always back up the state file beforehand, and try it on a test environment first.</p>

<h2 id="what-do-we-recommend">What do we recommend?</h2>

<ul>
  <li><strong>Existing Terraform code that works well, with HCP Terraform:</strong> you have no reason to switch.</li>
  <li><strong>New project, with your own CI/CD, without licence concerns:</strong> both are good; if there is no strong argument for Terraform, OpenTofu’s open licence raises fewer questions in the long run.</li>
  <li><strong>You are building a product or service on IaC:</strong> OpenTofu.</li>
</ul>

<p>Whichever you choose, the point is the same: the infrastructure should live in Git, changes should go to production through <code class="language-plaintext highlighter-rouge">plan</code> and code review, and nobody should make changes by hand in the cloud console. For the internal configuration of servers (packages, configuration files, services), a configuration management tool complements this well; we introduced the basics in our <a href="/en/blog/ansible-kezdoknek">Ansible for beginners</a> article. If you are starting out in the cloud, you can read about cloud service models (IaaS, PaaS, SaaS) on the <a href="https://www.awh.hu/en/blog/iaas-paas-and-saas-what-are-they">blog of our hosting brand, aWh</a>.</p>

<p>We also help with building and operating AWS, Google Cloud and OVHcloud infrastructure as code; you can read more on our <a href="/en/services/cloud-devops">Cloud and DevOps</a> page.</p>]]></content><author><name>CSB IT Bt.</name><uri>https://www.csb-it.hu</uri></author><category term="Cloud" /><category term="devops" /><category term="aws" /><category term="google" /><summary type="html"><![CDATA[Terraform or OpenTofu? Licence, features, compatibility and migration: we compare the two IaC tools and show when to choose which one.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://www.csb-it.hu/assets/images/og/en/terraform-vagy-opentofu.jpg" /><media:content medium="image" url="https://www.csb-it.hu/assets/images/og/en/terraform-vagy-opentofu.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry></feed>