How to install OSSEC – The Server Room

This guide covers installing OSSEC 4.4.1 as a standalone host-based intrusion detection system on Ubuntu 24.04. It explains installing dependencies, verifying the release signature, running the installer, starting the service, and configuring file integrity monitoring and rootkit detection.

It also covers SMTP alert delivery, real-time monitoring limitations, safe testing, troubleshooting, and configuration practices that reduce alert noise and protect sensitive data. Regular alert review, software updates, backups, access controls, and firewall maintenance remain important.

How to Install OSSEC on Ubuntu 24.04

OSSEC is an open-source host-based intrusion detection system that helps you monitor security events, investigate suspicious activity, and detect changes to important files.

This guide covers a standalone installation on an Ubuntu 24.04 server, including file monitoring, alerts, and basic troubleshooting.

What does OSSEC monitor?

Changes to monitored files and directories.
Security events in configured system and application logs.
Indicators of rootkits and suspicious system configurations.

An alert is a reason to investigate, not proof of a compromise. Software updates, deployments, and routine administration also change files.

Before you begin

An Ubuntu 24.04 server with SSH access.
A user with sudo privileges.
A recent backup or server snapshot.

These commands are for Ubuntu. AlmaLinux and other distributions use different package names and installation commands.

If OSSEC or Wazuh is already installed, review its upgrade instructions before continuing. Do not install a second monitoring system over an existing installation.

1. Install the build dependencies

sudo apt update
sudo apt install build-essential zlib1g-dev libpcre2-dev libevent-dev libssl-dev libcurl4-openssl-dev libmagic-dev libsystemd-dev curl ca-certificates gnupg

2. Download and verify OSSEC

Download the current stable Unix release and its matching GPG signature from the official OSSEC download page.

At the time of this update, the release is OSSEC 4.4.1. Check the official page before installing, as newer releases may be available.

Work in a dedicated directory:

mkdir -p ~/ossec-install
cd ~/ossec-install

Place the release archive and matching signature in this directory. Import the official signing key:

curl -fLO https://www.ossec.net/files/OSSEC-ARCHIVE-KEY.asc
gpg –import OSSEC-ARCHIVE-KEY.asc
gpg –fingerprint EE1B0E6B2D8387B7

Compare the fingerprint with the one published in the official installation documentation:

B50F B194 7A0A E311 45D0 5FAD EE1B 0E6B 2D83 87B7

For the 4.4.1 release, verify the archive with:

gpg –verify ossec-hids-4.4.1.tar.gz.asc ossec-hids-4.4.1.tar.gz

Continue only after the signature verifies successfully. Adjust the filenames if you downloaded a newer release.

3. Run the installer

tar -xzf ossec-hids-4.4.1.tar.gz
cd ossec-hids-4.4.1
sudo ./install.sh

For monitoring one server independently, choose local when asked for the installation type. Keep the default installation directory, /var/ossec.

Enable file integrity monitoring and rootcheck. Leave active response disabled during initial setup so you can review alerts before allowing automated blocking.

If you enable email notifications, enter your own delivery settings. Keep SMTP passwords out of tutorials, screenshots, and support messages.

4. Start OSSEC

sudo /var/ossec/bin/ossec-control start
sudo /var/ossec/bin/ossec-control status

Check for startup errors:

sudo tail -n 50 /var/ossec/logs/ossec.log

Confirm that your installation starts automatically after a reboot. Startup integration depends on the installation method and operating system.

5. Configure file integrity monitoring

Back up the configuration before editing it:

sudo cp -a /var/ossec/etc/ossec.conf /var/ossec/etc/ossec.conf.bak
sudo nano /var/ossec/etc/ossec.conf

Find the existing <syscheck> section. Merge the following settings into it, preserving other options you need and avoiding duplicate directory entries:

This example schedules a full scan every hour and enables near-real-time monitoring for existing file content changes under /etc. Adjust the scan interval to suit your server’s size and workload.

To monitor website files, add an entry for your actual document root:

<directories check_all=”yes” realtime=”yes”>/var/www/example.com/public</directories>

Exclude busy cache, session, and temporary directories where appropriate. Monitoring constantly changing files creates noise and makes useful alerts harder to spot.

Leave report_changes disabled for directories containing passwords, API keys, or other secrets. Enabling it can retain copies of file contents and include sensitive changes in alerts.

6. Understand real-time monitoring limits

Real-time monitoring begins after the initial scan establishes a baseline. It does not make every check instantaneous.

New-file alerts require a scheduled scan, and permission or ownership changes may also wait for that scan. Do not assume every event will generate an immediate notification.

7. Configure email delivery

OSSEC 4.1 and later support authenticated SMTP and TLS when built with the required libcurl support. Use the current OSSEC documentation to locate the SMTP configuration instructions for your release.

You can also deliver alerts through a properly configured local mail relay. Do not assume that setting the SMTP hostname alone is enough for an external email provider.

Test delivery separately from event detection. An alert appearing in the local log does not prove that an email reached your inbox.

8. Test monitoring safely

Create a dedicated test file:

printf ‘OSSEC monitoring test\n’ | sudo tee /etc/ossec-monitoring-test.txt

Allow a scheduled scan to discover it, then change its contents:

printf ‘OSSEC monitoring test changed\n’ | sudo tee -a /etc/ossec-monitoring-test.txt

Review the alerts:

sudo tail -n 100 /var/ossec/logs/alerts/alerts.log

Remove the test file when finished:

sudo rm /etc/ossec-monitoring-test.txt

Troubleshooting

Check /var/ossec/logs/ossec.log for startup, configuration, and delivery errors.
Confirm that monitored directories exist and match your server’s actual paths.
Allow the initial scan to finish before testing real-time changes.
Review alert thresholds if events appear locally but do not generate email.
Restart OSSEC after configuration changes and check its status again.

sudo /var/ossec/bin/ossec-control restart
sudo /var/ossec/bin/ossec-control status

Keep monitoring useful

Review alerts regularly, tune noisy rules, and keep OSSEC updated. File monitoring works best alongside tested backups, timely software updates, restricted administrative access, and a maintained firewall.

For current releases and configuration details, visit the official OSSEC documentation.

Source link