Your Logs Die With the Container. Loki Fixes That on One VPS

What Loki actually is

Think of Loki as Prometheus for log lines. Prometheus stores numbers over time, indexed by labels. Loki stores text over time, indexed by labels. That last part is the design decision that makes it light: **Loki does not build a full-text index**. It indexes only the labels — which container, which project, which level — and compresses the raw lines into chunks. When you search for a string, it narrows by label first and then greps the matching chunks.

The practical result: a full-text engine like Elasticsearch would want more RAM than your whole box has, while Loki runs happily next to your app. The trade-off is that queries which can't narrow by label are slow. In exchange, you get to keep weeks of logs on a $10 server.

Two containers, one file

You need Loki (stores and queries) and a collector that ships lines to it. The collector to use in 2026 is Grafana Alloy — Promtail, which most older tutorials still reference, has been deprecated.

services:
  loki:
    image: grafana/loki:3.6.13
    container_name: loki
    restart: unless-stopped
    command: ["-config.file=/etc/loki/config.yaml"]
    volumes:
      - ./loki-config.yaml:/etc/loki/config.yaml:ro
      - loki-data:/loki

  alloy:
    image: grafana/alloy:v1.18.0
    container_name: alloy
    restart: unless-stopped
    command:
      - run
      - --storage.path=/var/lib/alloy/data
      - /etc/alloy/config.alloy
    volumes:
      - ./alloy-config.alloy:/etc/alloy/config.alloy:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - alloy-data:/var/lib/alloy/data

volumes:
  loki-data:
  alloy-data:

Note that neither service publishes a port. Loki has no authentication of its own in this mode, so it should never be reachable from the internet — you talk to it through Grafana, which sits on the same Docker network.

The retention setting that silently does nothing

Loki's config is mostly boilerplate, with one trap worth knowing about. This is the part that matters:

limits_config:
  retention_period: 336h
  reject_old_samples: true
  reject_old_samples_max_age: 168h

compactor:
  working_directory: /loki/compactor
  delete_request_store: filesystem
  retention_enabled: true
  compaction_interval: 10m
  retention_delete_delay: 2h

retention_period looks like it's doing the work, but on its own it deletes nothing. The component that actually enforces it is the compactor, and **retention in the compactor is off by default**. Set retention_period alone and you get a disk that fills forever while the config file claims a 14-day window. It's the single most common way people end up with a full disk months later.

Fourteen days is a sensible default for a small box. Old logs are rarely useful, and if you need "what did this look like last quarter", that question belongs in your database, not in your logging stack.

Collecting logs without touching your app

Docker has a Loki logging driver, and it's the first thing most tutorials suggest. I'd avoid it on a machine you care about. That driver makes your application's logging path depend on Loki being up: if Loki is unreachable and its buffer fills, writes can block — and now a monitoring outage has become an application outage. Monitoring should never be able to take down the thing it monitors.

The safer route is to let the collector *read* logs through the Docker API, changing nothing about your app containers:

discovery.docker "containers" {
  host             = "unix:///var/run/docker.sock"
  refresh_interval = "15s"
}

discovery.relabel "containers" {
  targets = discovery.docker.containers.targets
  rule {
    source_labels = ["__meta_docker_container_name"]
    regex         = "/(.*)"
    target_label  = "container"
  }
}

loki.source.docker "containers" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.relabel.containers.output
  labels     = { job = "docker" }
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint { url = "http://loki:3100/loki/api/v1/push" }
}

That's the whole collector. New containers are discovered automatically, which matters: redeploy your app, and the fresh container is picked up within fifteen seconds without you touching anything.

One security note, since the config mounts a socket: read-only or not, access to the Docker socket is effectively root on the host. It's the standard way to collect container logs, but keep the collector on an internal network and don't expose its port.

Labels are where people wreck their install

The one rule that decides whether your Loki stays fast: **labels are for things with few possible values**. Container name, project, log level, environment — good. Request ID, user ID, IP address, trace ID — catastrophic. Every unique combination of label values creates a separate stream, and thousands of streams will bring the whole thing to its knees.

The instinct that gets people in trouble is "I want to search by request ID, so I'll make it a label." You don't need to. Put it in the line and search the text:

{container="api"} |= "req-1f0"

Loki narrows by container first, then greps. That's exactly the workflow it's built for.

The one label worth investing in is severity, because "show me only errors" is the query you'll run most. This is fiddlier than it sounds, because every piece of software announces its level differently. A Node app using pino writes JSON with a number ("level":50). Caddy writes JSON with a string ("level":"error"). Prometheus and Grafana use logfmt (level=info). PostgreSQL writes plain text with ERROR: after the process ID. Redis marks severity with a single character. Go tools built on klog start the line with a letter: E0725 14:14:14.

So the level extraction is a stack of parsers tried in order, each writing into its own field, with the first hit winning and getting normalised to one vocabulary. It's worth the twenty minutes. Without it, half your logs land in a bucket called unknown — and the errors hiding in that bucket won't show up in your error panel or fire the alert you eventually build on top of it.

Two things to know before you spend a day tuning: parsing happens at ingest, and **labels are not applied retroactively**. Fix your parser today and only new lines get the better labels. Also, some containers genuinely have no level to extract — plenty of apps just print text — and for those, searching by substring is the honest answer.

Actually using it

Point Grafana at http://loki:3100 as a Loki data source and you can query in LogQL, which reads like PromQL's cousin:

{container="api", level="error"}
sum by (container) (count_over_time({job="docker"}[5m]))
{container=~"api|worker"} |= "timeout" != "healthcheck"

One syntax rule bites everybody once: a query needs at least one matcher that can't match an empty value. {container=~".*"} is rejected, which is annoying when a Grafana dashboard variable set to "All" expands to exactly that. The fix is to keep a constant anchor label in every query — that's what job="docker" is doing in the examples above.

The payoff is the workflow, not the storage. Put a logs panel on the same dashboard as your metrics and both obey the same time range. You see the spike at 03:12, and the lines from 03:12 are already on screen. No SSH, no guessing at --since timestamps, no worrying about which container was running at the time.

With Loki catching everything, you can finally cap the local logs without losing anything:

logging:
  driver: json-file
  options: { max-size: "10m", max-file: "3" }

Set that per-service in your compose file rather than in /etc/docker/daemon.json. Daemon-level log options only take effect after restarting the Docker daemon, which bounces every container on the box; the compose setting applies at the next ordinary deploy, with no downtime.

The whole stack costs about 80 MB for Loki, 95 MB for Alloy, and a few hundred megabytes of disk for two weeks of a normal app's output. In return, the next time something breaks at 3 AM, the explanation is still there in the morning — even if you've redeployed twice since.