Ever tried to hunt down why a service suddenly stopped, only to stare at an empty /var/log directory?
You’re not alone. In a lot of labs—especially the 8.4.9 “Configure Logging on Linux” exercise—students spend more time Googling “where does syslog write?” than actually fixing the problem.
The short version is: if you don’t tell Linux where to put its messages, you’ll be chasing ghosts. Below is everything you need to know to ace that lab, keep your logs tidy, and actually use the data when something goes sideways Worth keeping that in mind..
What Is Logging on Linux
Linux logging is the system’s way of whispering (or shouting) everything that happens: kernel boots, daemons start, users log in, files get accessed. In practice, those whispers end up as plain‑text files under /var/log, but the magic happens behind the scenes with a daemon called rsyslog (or sometimes systemd‑journald on newer distros).
Think of rsyslog as the post office. That said, applications drop “letters” (log messages) into a mailbox, rsyslog reads the envelope, decides where to deliver it, and then stamps it into the right file. You can also tell it to forward messages to a remote server, filter out noise, or change the format entirely Turns out it matters..
It sounds simple, but the gap is usually here.
The Core Pieces
- syslog facility – a tag that groups similar services (e.g.,
auth,cron,daemon). - syslog severity – how urgent the message is (
debug,info,warning,err,crit). - rsyslog.conf – the rulebook that maps facility + severity to a destination.
- journalctl – the command‑line viewer for systemd’s binary journal (if you’re on a system that uses it).
If you’ve ever edited /etc/rsyslog.conf or dropped a file into /etc/rsyslog.d/, you’ve already been playing with the same tools the 8.Also, 4. 9 lab expects you to master.
Why It Matters / Why People Care
When you’re troubleshooting a flaky web server, the first place you look is the log. No log, no clue, and you end up restarting services blindfolded.
In a production environment, logs are the raw material for compliance audits, security forensics, and capacity planning. Miss a single line about a failed SSH login and you could be blind to a brute‑force attack Simple, but easy to overlook. Took long enough..
And here’s the thing most newbies miss: logging isn’t just “turn it on”. So it’s about making the right data visible, searchable, and retained for the right amount of time. Over‑logging fills disks; under‑logging leaves you clueless. The 8.4.9 lab forces you to strike that balance That's the part that actually makes a difference..
At its core, where a lot of people lose the thread.
How It Works (or How to Do It)
Below is a step‑by‑step walk‑through that covers the typical lab requirements: installing rsyslog, configuring local files, rotating logs, and sending a copy to a remote host.
1. Install and Verify rsyslog
# Debian/Ubuntu
sudo apt-get update && sudo apt-get install -y rsyslog
# RHEL/CentOS/Fedora
sudo yum install -y rsyslog
After installation, make sure the daemon is running:
systemctl status rsyslog
You should see “active (running)”. If it’s disabled, enable it:
sudo systemctl enable --now rsyslog
2. Understand the Default Configuration
Open the main config file:
sudo less /etc/rsyslog.conf
You’ll notice three sections:
- Modules – load input/output plugins (
imuxsock,imjournal, etc.). - Global directives – things like
$RepeatedMsgReductionthat control behavior. - Rules – the heart of the matter, e.g.:
*.info;mail.none;authpriv.none;cron.none /var/log/messages
authpriv.* /var/log/secure
mail.* -/var/log/maillog
The asterisk (*) means “all facilities”. After the semicolon you can exclude specific ones (mail.none). The path at the end tells rsyslog where to write Took long enough..
3. Create a Custom Rule File
Best practice (and what the lab expects) is to drop a new file into /etc/rsyslog.d/. Let’s call it `10-custom.
sudo tee /etc/rsyslog.d/10-custom.conf > /dev/null <<'EOF'
# Log all cron activity to its own file
cron.* /var/log/cron.log
# Separate authentication logs
auth,authpriv.* /var/log/auth.log
# Capture kernel messages at warning level or higher
kern.warn /var/log/kern.warn.log
EOF
A few notes:
- No spaces before the destination path, otherwise rsyslog will ignore the line.
- Prefixing the file with a low number (
10-) ensures it’s loaded early, before any catch‑all rules that might otherwise swallow the messages.
4. Test the New Rules
Restart rsyslog so it picks up the new file:
sudo systemctl restart rsyslog
Now generate a test message:
logger -p auth.info "Test auth log entry"
logger -p cron.notice "Test cron log entry"
logger -p kern.err "Kernel error test"
Check the files:
tail /var/log/auth.log
tail /var/log/cron.log
tail /var/log/kern.warn.log
If you see your messages, you’ve nailed the basic configuration Simple, but easy to overlook..
5. Set Up Log Rotation
Linux doesn’t keep logs forever; logrotate handles that. The default package already ships a config for most standard logs, but you’ll want a custom rule for the files you just created Not complicated — just consistent..
Create /etc/logrotate.d/custom:
sudo tee /etc/logrotate.d/custom > /dev/null <<'EOF'
/var/log/cron.log /var/log/auth.log /var/log/kern.warn.log {
weekly
rotate 4
compress
missingok
notifempty
create 0640 syslog adm
sharedscripts
postrotate
/usr/lib/rsyslog/rsyslog-rotate
endscript
}
EOF
Explanation in plain English: keep four weeks of logs, compress old ones, and tell rsyslog to reopen its file handles after rotation (the postrotate script does that) That's the part that actually makes a difference..
Run a manual rotation to be sure:
sudo logrotate -f /etc/logrotate.conf
6. Forward Logs to a Remote Syslog Server
Many labs ask you to ship logs to a central host for aggregation. Because of that, add this line to a new file, say `/etc/rsyslog. d/30-remote.
*.* @192.0.2.10:514
@= UDP (use@@for TCP).- Replace
192.0.2.10with your remote server’s IP.
Restart rsyslog again, then verify with tcpdump or by checking the remote server’s /var/log/syslog. If you see the lab’s test messages arriving, you’re golden Not complicated — just consistent..
7. Using systemd‑journalctl (Optional)
If your distro uses systemd-journald (most modern ones do), logs are also stored in a binary journal under /run/log/journal (volatile) or /var/log/journal (persistent). You can view them with:
journalctl -u rsyslog
journalctl -p err
To make the journal persistent, create the directory and restart:
sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald
The lab may not require this, but it’s good to know the two systems can coexist.
Common Mistakes / What Most People Get Wrong
-
Editing the wrong file – The default
rsyslog.confis still read, but any mistake there can override your custom rules. Always add new rules in/etc/rsyslog.d/unless you have a reason to edit the main file The details matter here.. -
Forgetting the dash before a log path – A leading
-tells rsyslog to use asynchronous writes (faster, less safe on power loss). If you accidentally put a space before the dash, rsyslog treats it as part of the filename and silently drops the rule Less friction, more output.. -
Not reloading after changes –
systemctl restart rsyslogis required. Some people thinkservice rsyslog reloadworks, but on newer systems it only reloads the configuration if the daemon supports it, and rsyslog prefers a full restart Turns out it matters.. -
Mismatched facility/severity syntax – Using
authpriv.*is fine, butauthpriv.*.*will break the parser. Stick tofacility.severityor the wildcard*That's the part that actually makes a difference.. -
Logrotate not notified – If you forget the
postrotatescript, rsyslog will keep writing to the old (now renamed) file, causing duplicate entries and huge disk usage That's the part that actually makes a difference. That's the whole idea.. -
Remote logging over UDP without firewall rules – UDP is connectionless; if the firewall blocks port 514, your packets vanish. Double‑check
iptablesorfirewalld.
Practical Tips / What Actually Works
-
Keep it simple – One rule per file is easier to audit. If you need many destinations, consider using a template (
$Template) instead of copy‑pasting lines. -
Use timestamps – Add
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormatat the top of your config if you need the classic “MMM DD HH:MM:SS” format. It makes grepping far easier Nothing fancy.. -
Separate noisy services – Applications that spew debug info (e.g.,
cronon a busy server) should have their own log file with a higher severity threshold (cron.warn). -
Compress early – Adding
compressto logrotate saves space, but remember that compressed logs are not readable bytail. Usezcatorzlesswhen you need to peek. -
Test with logger – The
loggercommand is your best friend. It lets you fake any facility/severity without needing to restart services. -
Monitor disk usage – Set up a simple alert with
df -h /var/login a cron job. Nothing is more frustrating than a full root partition because logs weren’t rotated Nothing fancy.. -
Document your changes – Add a comment at the top of each custom config file: who added it, why, and the date. Future you (or a teammate) will thank you when the lab turns into a production environment Worth keeping that in mind..
FAQ
Q: Do I need both rsyslog and systemd‑journald?
A: Not necessarily. Most distros run both; journald captures everything first, then forwards to rsyslog if configured. You can disable journald forwarding by setting ForwardToSyslog=no in /etc/systemd/journald.conf, but keep it on for redundancy.
Q: How can I view only kernel warnings in real time?
A: journalctl -k -p warning -f or tail -f /var/log/kern.warn.log if you’ve set up a separate file.
Q: My remote syslog server isn’t receiving anything. What’s the first thing to check?
A: Verify network connectivity (ping/nc -vz <IP> 514), then confirm the firewall allows UDP/TCP 514. Finally, look at /var/log/syslog on the client for “error sending” messages Worth knowing..
Q: Can I limit log size without using logrotate?
A: Yes, rsyslog has a built‑in omfile option FileSize that stops writing once a file reaches a threshold, but logrotate is more flexible and widely used And that's really what it comes down to..
Q: Is it safe to use UDP for remote logging?
A: UDP is faster but unreliable—packets can be dropped. For critical security logs, use TCP (@@remote:514) or even TLS (@@@remote:6514) if your server supports it.
Logs are the lifeblood of any Linux system. Once you’ve got the 8.4.9 lab down—custom rules, proper rotation, and remote forwarding—you’ll find yourself spending far less time guessing and a lot more time actually fixing problems That's the whole idea..
So next time a service misbehaves, open the right log file, read the line, and let the data tell you what’s wrong. Happy logging!