NETWORK TECHNOLOGY
Comprehensive IT solutions, support, security and automation for your business
Tech support: 24/7
instant response for contract clients +359 2 958 6535
Network Technology

An attack detected, and missed: when your protections are not connected

You have a firewall, fail2ban and monitoring, yet an attack can still get through. How to connect servers, router and monitoring so they block together and send…

Eng. Svilen Arsov ·
In short: you probably already have a firewall, fail2ban and some monitoring. This article is about making them work together instead of each on its own.

An attack that was detected, and missed

Why separate protections are not enough, and what protection looks like when the servers, the router and the monitoring know about each other.

Diagram: router, shared list of blocked addresses, servers, monitoring and Telegram alerts

In most small companies the security setup looks fine. The router has rules, the servers run fail2ban, and there is some kind of monitoring.

The problem is that these pieces don't know about each other. Monitoring sees the attack in the logs of one server. The router, which could stop it for every machine, never hears about it. The attacker simply moves on to the next server.

The worse case

Protection that has stopped working looks exactly like protection that has nothing to report. The agent crashed after an update, the service didn't start after a reboot, the log disk is full.

For days there are no alerts and everyone assumes all is well. In reality nothing is being watched.

This isn't the admin's fault. It is what happens when every tool is added separately, at different times, for a different problem. Each one does its job, but nobody looks at the whole picture.

What's missing: the link between the tools

Usually there is a firewall and log-based blocking. What's missing is the connection between them. If you recognise at least two of these, this article is for you:

🛡️
Each server protects itself and doesn’t know what the others have blocked.
🧩
Blocked addresses live in different places and nobody has the full list.
🔌
Blocks on the router disappear after a reboot or a power cut.
🔔
Alerts are either zero or hundreds a day, and nobody reads them.
❓
Nobody checks whether the protection itself is still working.
✍️
Settings are changed by hand and not recorded anywhere.

Each of these looks minor on its own. Together they mean that during an attack everything depends on someone happening to look at the right log. And when something breaks, nobody remembers what was changed and when.

Today Connected Router Server Server Server no link each machine protects only itself Router Shared list Server Server Server
On the left, each server protects itself and the router has no idea what is happening on them. On the right, the servers write blocked addresses to a shared list and the router blocks them for the whole network.

What connected protection looks like

This isn't a new product. It uses the familiar tools, connected to each other. Piece by piece:

Router with rules and blocklists

Stops the attack before it reaches the servers. Obviously invalid traffic is dropped right at the entrance. An address that keeps trying is blocked first for a short time, then longer, and finally for good.

fail2ban on every server

Each server reads its own logs and blocks on its own. The rules are tuned for the services that actually run there, not left at the defaults. On its own, though, fail2ban only protects its own server: a blocked address keeps trying on the next one.

Shared list of blocked addresses

Every server records every block in one place, so you can see which address tried where. The router restores its rules from this list after a power cut or reboot, without anyone re-entering them.

Monitoring and log collection

Agents on every machine, including the hypervisors, plus the router logs, collected in one place. Ten failed logins on one server are nothing unusual. The same ten spread across five servers in a minute are an attack no single server can see.

Automatic blocking

When it is clear that it’s an attack, the address is blocked automatically through the same shared list and router. No separate new tool is added. Unclear cases are not blocked but sent to a person.

Checking that protection works

The servers check each other: are the agents running, are the logs arriving. If something stops, that is treated as a problem and reported right away.

Detectionon any server Decision Block on the router Write to the shared list Checklist ↔ router
An address caught on one server is blocked for all of them. The list and the router are compared regularly, so nothing is lost after a restart.

Telegram alerts only for what matters

Alerts come at two levels. Telegram only gets what the system did not stop on its own and needs a person. Everything else arrives once a day by email, grouped and counted.

An attack that was stopped automatically is not a reason for a Telegram alert. It shows the protection is working, and you see it in the daily email.

All eventse.g. tens of thousands a day Logeverything is recorded, no alerts Daily emailone, grouped and counted Telegram: only what was not stopped
Almost everything stays in the log and the daily email. Only what was not stopped automatically goes to Telegram. Numbers are examples.

Backups and rollback

Before every change the settings are backed up. Virtual machines get snapshots, and backups are kept somewhere else, not on the same server. A wrong rule on the router can cut off access for everyone, so every such change comes with a ready way back.

Recorded changes

Every settings change is recorded: what, when and why. The record is regularly compared with the real state. If someone changed something by hand and didn’t record it, it shows up.

How it's rolled out without stopping work

The most common worry is that the protection will block your own staff or stop a service during working hours. That's why it is done in steps:

1
Review what you have. Something that works is not replaced just because it could be done another way.
2
Use what is already there. We connect the tools you already have instead of adding new ones.
3
Record only, at first. Every new rule first only records what it would block. After one day of real traffic it is enabled.
4
Backup and check. A backup before every change and a check of the settings before they are applied.
5
Everything is recorded. It is always clear what changed and how to roll it back.
Review Recordonly One day ofreal traffic Enable backup · rollbackbackup · rollbackbackup · rollbackbackup · rollback Nothing is enabled before it has been tried on real traffic.
Every rule records first and blocks later. Every step has a backup you can roll back to.

This way everything is introduced in parts, without interrupting services. Your team sees each step before it starts acting and can stop it. If something doesn't behave as expected, rolling back takes minutes.

What you end up with

✓
One list of all blocked addresses: who, where, how many times.
✓
Automatic blocking for the whole network when an attack is caught on one server.
✓
A Telegram alert only when a person needs to step in.
✓
One email a day with the statistics, instead of hundreds of alerts.
✓
Router rules restored automatically after a power cut or reboot.
✓
If monitoring stops, you find out right away.
✓
Backups and snapshots you can roll back to.
✓
A recorded history of every settings change.

Who it's for, and who it isn't

A good fit for
  • companies with their own servers or colocation;
  • small hosting providers;
  • offices with a router, VPN and a few services reachable from outside.
You don't need it if
  • everything is in the cloud and you have no servers of your own reachable from the internet;
  • you have one server with one service. Well-tuned rules on that server and regular backups are enough.

First step: a free review

If you recognised your own setup at the start of this article, begin with a review: what you have, what isn't connected and what's missing. Afterwards you'll know what needs to be done, whether or not we work together.


Author photo
Author

Eng. Svilen Arsov

Head of IT Infrastructure & Security at Network Technology.

Comments

Loading…
Only registered and logged-in users can comment.