Home › Uncategorized › Keeping a single-node Wazuh server healthy: heap, retention and cron

Keeping a single-node Wazuh server healthy: heap, retention and cron

Posted on Posted in Uncategorized

A single-node Wazuh server (manager, indexer and dashboard on one box) runs fine for months, until the indexer runs out of memory or the disk fills up. These are the checks and fixes I now apply to every Wazuh server.

1. Give the indexer enough heap

Symptom: dashboard and Alerting queries fail with circuit_breaking_exception [parent] Data too large. Check heap use with _cat/nodes?h=heap.percent,heap.max. If it sits above about 90%, raise -Xms/-Xmx (always the same value) in /etc/wazuh-indexer/jvm.options. Keep it at or below about 50% of RAM, and leave room for the manager and dashboard. On a 9 GB VM, 1 GB → 3 GB solved it.

cp -a /etc/wazuh-indexer/jvm.options /etc/wazuh-indexer/jvm.options.bak.$(date +%Y%m%d)
sed -i 's/^-Xms1024m/-Xms3g/; s/^-Xmx1024m/-Xmx3g/' /etc/wazuh-indexer/jvm.options
systemctl restart wazuh-indexer

2. Stop the JVM from swapping

echo 'vm.swappiness=1' > /etc/sysctl.d/99-wazuh-indexer.conf
sysctl -p /etc/sysctl.d/99-wazuh-indexer.conf

3. Set index retention with ISM

Wazuh creates a daily wazuh-alerts-4.x-* index with 3 shards, and nothing deletes it by default. After a few months you have hundreds of shards on a small heap. An ISM policy deletes indices after N days, and its ism_template applies it to new indices automatically:

PUT _plugins/_ism/policies/wazuh-retention-30d
{"policy":{"description":"Delete Wazuh indices after 30 days","default_state":"hot",
 "states":[
  {"name":"hot","actions":[],"transitions":[{"state_name":"delete","conditions":{"min_index_age":"30d"}}]},
  {"name":"delete","actions":[{"delete":{}}],"transitions":[]}],
 "ism_template":[{"index_patterns":["wazuh-alerts-4.x-*","wazuh-monitoring-*","wazuh-statistics-*"],"priority":100}]}}

POST _plugins/_ism/add/wazuh-alerts-4.x-*,wazuh-monitoring-*,wazuh-statistics-*
{"policy_id":"wazuh-retention-30d"}

The add call attaches the policy to indices that already exist. Note that min_index_age counts from the index creation date, so check it matches the date in the name before you apply. Deleted indices are gone for good unless you have snapshots. In my case the shards went from 287 to 135 and the indexer from 22 GB to 8 GB.

4. Clean the raw alert logs with cron

The manager also keeps compressed copies under /var/ossec/logs/alerts/YYYY/Mon/, and nothing rotates them away (here, about 3.5 GB a month). Two gotchas: minimal Ubuntu images may not have cron installed at all (systemctl status cron), and files in /etc/cron.d must end with a newline, otherwise cron ignores them silently (“Missing newline before EOF” in the journal).

# /etc/cron.d/wazuh-alerts-retention : keep 45 days
SHELL=/bin/sh
PATH=/usr/sbin:/usr/bin:/sbin:/bin
30 3 * * * root find /var/ossec/logs/alerts -mindepth 3 -type f -name 'ossec-alerts-*' -mtime +45 -delete; find /var/ossec/logs/alerts -mindepth 2 -type d -empty -delete

Keep the raw logs a little longer than the indices. They are your fallback copy after ISM has deleted an index.

5. Know what is supposed to be big

/var/ossec/queue/vd/feed (about 13 GB on 4.14) is the vulnerability-detection CVE database (RocksDB, updated hourly). It’s normal, and it doesn’t grow with your number of agents. Leave it alone.

Checklist

  • Heap below about 75% and no circuit-breaker errors
  • vm.swappiness=1
  • An ISM retention policy on alerts, monitoring and statistics indices
  • cron installed, with a cleanup job for /var/ossec/logs/alerts
  • Disk below 80%: the indexer stops allocating at 85–90%
  • Back up every config before editing, and verify after each restart