Networks and perimeter security
We manage, almost always remotely, the routers, firewalls, switches and Wi-Fi networks of companies in industry, distribution, transport and services: site-to-site links, VPNs across multiple brands, firewall tuning, diagnosis of backbones with devices from different manufacturers, temporary networks for events.
We work on devices in production, so we proceed with caution: first we measure, then we propose; every change starts out with its own way back; at the end we leave runbooks, saved configurations and, when needed, an account of what went wrong and why.
Site-to-site links and network extensions
We connect sites, plants and data centres with WireGuard, EoIP and OpenVPN; we configure bridges with VLANs, MTU and MSS clamping, policy routing across multiple lines and router redundancy with VRRP. When a network has to be extended at layer 2 all the way to a data centre, we do it knowing where the pitfalls lie: fragmentation, NAT dropping the replies, allowed addresses in the tunnels.
Technologies and products
- MikroTik RouterOS 7
- WireGuard
- EoIP and GRE
- OpenVPN
- VRRP
- IEEE 802.1Q
Examples
-
IT services company In production
A site network extended all the way to the data centre
- The problem
- The LAN and two VLANs of the site had to reach, at layer 2, the servers hosted in the data centre; the previous solution over L2TP and IPsec was not enough.
- What we did
- Three EoIP tunnels over WireGuard, with a usable MTU of 1378 bytes. Two pitfalls solved along the way: the site firewall’s NAT dropped the replies when the WireGuard source port was the same as the destination port, and the allowed addresses in the tunnel had to include all the routed networks.
- Result
- The extension is in production. Automatic high availability between the two site routers was studied and, for now, judged not ready: we wrote that down instead of enabling it.
-
Group in the distribution sector In production
A site connected over 5G with two operators
- The problem
- A site had to work over 5G lines while waiting for fibre, without losing remote management.
- What we did
- Two routers and two 5G lines from different operators: browsing traffic goes out over the second line through policy routing, and the first stays with the tunnels and management. An on-board script saves a restore point after 30 minutes of healthy operation and reloads it if management and the internet go silent. The bandwidth measurement showed that the limiting factor was the router’s CPU (97%), not the line.
- Result
- The site worked over 5G and today 5G acts as backup to the fibre; switching between the two follows a five-step procedure with nine preliminary checks.
VPNs between sites, to the cloud and for remote working
We configure IPsec VPNs (IKEv1 and IKEv2, policy-based or route-based) between devices from different brands and to public clouds, as well as WireGuard and OpenVPN concentrators and remote access with centralised authentication. We also design the switchover to the disaster recovery site and test it in both directions.
Technologies and products
- IPsec and NAT-T
- WireGuard
- OpenVPN
- SoftEther
- FortiClient
- SAML with Microsoft Entra ID
- Oracle Cloud and Google Cloud VPN
Examples
-
IT services company In production
VPN to the disaster recovery site in the cloud
- The problem
- The disaster recovery site in the cloud had to be reliably reachable, and its configuration was never to put production at risk.
- What we did
- Two tunnels to the two endpoints of the DR site. The script that configures them has preview, verification, rollback and a lock that protects production.
- Result
- The VPN has been tested in both directions on both tunnels (16-18 ms) and manual switchover to DR is ready. Automatic switchover, a three-state machine with hysteresis, is a proposal; having both tunnels active with BGP is a study.
-
Office of an IT services company In production
Remote access on a firewall without SSL-VPN
- The problem
- That firewall model has no SSL VPN, but users had to work from outside as if they were in the office.
- What we did
- FortiClient over IPsec IKEv2 with EAP: the clients get a LAN address (with proxy ARP) and also reach the VPNs to the other sites without changes at the other end. Behind a hotspot the return traffic was getting lost in the NAT: we forced NAT-T.
- Result
- Remote access is in production; sign-in with company Microsoft Entra ID credentials via SAML is configured and being tested.
-
IT services provider In production
A VPN server that ran out of memory six times a day
- The problem
- The VPN server used for a client’s backups was being terminated for running out of memory up to six times a day.
- What we did
- The cause was a bridge on a network card that was switched off, which opened one socket per second: 27,233 orphaned sockets, about 7 GB of kernel memory. We removed the bridge and added an automatic check.
- Result
- Memory in use fell from 95% to 3.7%; an automatic check monitors the service.
Perimeter firewalls and segmentation
We analyse and tune Palo Alto, FortiGate and Check Point firewalls and MikroTik and Linux filters: zones, rules, NAT, intrusion prevention profiles. For Palo Alto firewalls we have our own tools that, through the APIs, show which rule a flow hits, which rules allow a given type of traffic and what is published on the internet.
We segment networks to reduce the potential damage: guest Wi-Fi on its own VLAN with the clients isolated from one another, outdated servers published only to a few sources (and the restriction tested from an address that is not allowed), cameras on a separate network.
Technologies and products
- Palo Alto PAN-OS and Strata Cloud Manager
- FortiGate (FortiOS)
- Check Point Gaia
- WatchGuard Fireware
- RouterOS firewall
- iptables and nftables
Examples
-
Industrial group In production
An IPS signature that isolated a server every hour
- The problem
- A server kept losing its connection to the domain at regular intervals, with no obvious trace in the firewall’s traffic logs.
- What we did
- An intrusion prevention signature fired on ONVIF traffic to two cameras, and its block action dropped all of the server’s traffic, 3,700 packets per second; the block renewed itself every time it expired. We created the exception through the APIs of the cloud management system.
- Result
- The server is stable. An automatic check now reads the block tables of five firewall clusters every 3 minutes and sends an alert with an unblock button.
-
Industrial group with cloud-managed firewalls Delivered
Rules that seemed to block but did not
- The problem
- The rules distributed by the cloud management system cannot be read in the local configuration, and the policy match test always returned the same rule.
- What we did
- We used the logs and sessions as our reference, which do not lie, and took an inventory of the services published on the internet. This revealed that the block on malicious addresses was ineffective: an allow rule preceded it by 355 positions.
- Result
- The block was moved ahead of the allow rule; the inventory of published services has been delivered.
Design and commissioning of site networks
We design a site’s network (firewalls, switches, Wi-Fi, VPNs), prepare it alongside the existing one and carry out the cutover in an agreed window, with a runbook and rollback ready. At the end a script checks the installation point by point.
Technologies and products
- FortiGate in high availability (FGCP)
- Aruba CX (AOS-CX)
- UniFi
- Google Cloud VPN
- rsyslog
Examples
-
Office of an IT services company In production
A site's new network, cut over in a single day
- The problem
- Replacing the firewalls, switches and Wi-Fi of a site in production, together with the VPNs to the cloud and to the other sites.
- What we did
- An active-passive FortiGate cluster, two Aruba CX switches, a UniFi controller and three access points, prepared alongside the existing network. Then five site-to-site VPNs (three route-based to Google Cloud, two to Check Point), remote access and the corporate and guest Wi-Fi.
- Result
- Cutover in a single day, high availability tested, rollback not needed. The next day the check script reported 38 checks passed out of 38; the counters brought to light a faulty port that caused a spanning tree recalculation every time it came back up, and it was shut down.
-
Group in the distribution sector In production
The distribution node of a logistics site
- The problem
- The distribution node of the site network was unstable and the uplink ran through a faulty optical module.
- What we did
- A new Aruba switch as the MSTP root, the uplink moved off the faulty module, a syslog server and an availability dashboard for 22 devices.
- Result
- The site network is stable and observable; the spanning tree study has been partly applied.
Switching, backbones and spanning tree
We read and fix HPE and Aruba, Dell, Cisco and MikroTik backbones: LACP aggregation, MLAG in its variants (VSF, distributed trunking, VLT), STP, RSTP and MSTP, VLANs per link. We track down intermittent loops, the ones that appear and disappear, with repeated measurements and tools written for the purpose.
Technologies and products
- HPE ArubaOS-Switch
- Dell OS10 (VLT)
- Aruba CX
- Cisco Catalyst
- MikroTik CRS and SwOS
- LLDP and CDP
- IEEE 802.1D, 802.1w, 802.1s, 802.3ad
Examples
-
Industrial group with several campuses In progress
An intermittent loop between two aggregation technologies
- The problem
- Some servers were losing traffic intermittently, with no obvious fault, on a backbone of eight devices from two manufacturers spread across three campuses.
- What we did
- We wrote a detector based on a simple principle: the two members of a VLT pair cannot disagree about where a MAC address is. It identified the affected servers, and a study of the configurations located the loop at the point where two different MLAG technologies meet, on a link with spanning tree filtered.
- Where we have got to
- We removed 28 VLANs from one link (CPU on a core switch from 27% to 12%), re-enabled a second link with no moves on 578 MAC addresses, reduced MAC moves from 12 to 2 per minute and brought VLT inconsistencies down to zero with a temporary fix. The structural solution has been studied, with the hardware constraints verified.
-
Production site In production
Router redundancy at a production site
- The problem
- The site’s pair of routers did not guarantee a clean failover: BPDUs were not being exchanged because VLAN 1 was untagged on one side and tagged on the other.
- What we did
- We rebuilt the redundancy with VRRP and an automatic check against loops.
- Result
- Measured failover: about 5 seconds for a router failure, 10 for an unplugged WAN cable, failback with no overlap.
Device fleets, hardening and automation
We keep a single inventory of network devices and query it with our own tools (SSH, RouterOS API, FortiOS REST API, PAN-OS XML API): versions, exposure, management rules applied with automatic rollback on the device, searches for indicators of compromise, full backups before and after every change.
Technologies and products
- Python (paramiko, routeros_api)
- RouterOS API
- FortiOS REST API
- PAN-OS XML API
- AOS-CX checkpoints
Examples
-
MikroTik fleet across several clients In production
Management locked down on 26 routers after a security advisory
- The problem
- A CERT advisory reported RouterOS vulnerabilities being exploited through SSH. We needed to know at once which devices were exposed and to lock them down, without losing access to any of them.
- What we did
- An inventory of 67 devices across 11 projects, with a traffic-light rating based on exposure to known vulnerabilities, not on version numbers. On 26 devices we restricted management access to allowed addresses and switched off unnecessary services, with a mandatory preview, an on-board scheduler that restores everything within 10 minutes if the change is not confirmed, and a check over a new connection.
- Result
- Measured result: no device with SSH exposed. The six-wave upgrade plan has been written and is in progress.
-
Network infrastructure for services Delivered
Compromise check on a VPN concentrator
- The problem
- Ruling out with certainty that a VPN concentrator had been tampered with.
- What we did
- A complete check of users, keys, scheduled tasks, scripts, tunnels, firewall, routes, DNS, proxy, certificates and change history.
- Result
- No implant found, with an itemised list of what was checked.
Corporate and warehouse Wi-Fi
We reconstruct the actual architecture of a Wi-Fi network (what acts as the gateway, what hands out the addresses, where authentication goes), fix roaming and radio parameters and analyse misbehaving clients starting from the controller’s data, not from impressions.
Technologies and products
- UniFi
- Alcatel-Lucent OmniVista Cirrus
- Cisco WLC
- IEEE 802.11k, 802.11r, 802.11v
- 802.1X and RADIUS
Examples
-
Industrial group with several sites In production
Roaming fixed across more than a hundred access points
- The problem
- Slow roaming and devices “connected but with no address” at several sites.
- What we did
- We documented that the controller works only at layer 2, while the gateway and DHCP relay sit on the firewall and 802.1X authentication goes to the RADIUS servers. We applied the fixes (802.11r on 20 SSIDs, roaming assistant, DTIM, minimum data rate) with a script and verified them with 57 automatic checks (56 passed), with a targeted rollback on three SSIDs.
- Result
- Roaming has been fixed. For the devices with no address, we also corrected our first hypothesis in writing: what unblocked them was restarting the access points, not disabling 802.11r.
-
Distribution with several logistics sites Study
Warehouse terminals that changed access point five times an hour
- The problem
- The older warehouse terminals were losing their connection while in use.
- What we did
- From the controller data: on the SSID with 802.11r those terminals made 4.8 roaming events and 3.8 disconnections per hour, against 0 and 0.2 on the SSID without it. There is more than one cause: the threshold for “sticky” clients disabled, an outdated driver, the lack of 802.11k and v, known issues with the FT-PSK combination.
- Where we have got to
- Recommendations delivered in order of priority.
Event networks
We put temporary networks into service and look after them for the whole duration of the event: redundant gateways, multiple internet lines, VLANs per area with bandwidth limits, high-density Wi-Fi with a channel plan that respects the constraints that apply in Italy, devices prepared on the bench and managed remotely.
Technologies and products
- Pacemaker and Corosync
- Policy routing on Linux, iptables, tc
- ISC DHCP
- MikroTik, Cisco Catalyst, UniFi
- SNMP and ONVIF
Examples
-
Major international cultural event Delivered
The network of a major event, from the bench to the close
- The problem
- A temporary network with many different areas, guests and productions, which had to hold up for the whole event and be manageable remotely.
- What we did
- Gateway and DHCP on a high-availability Linux cluster, expanded from two to four nodes without downtime; two internet lines with policy routing per VLAN; 16 VLANs, some with a bandwidth limit. Seven routers connected to the management VPN: some cloned units had the same bridge MAC and pushed packet loss up to 100%. Eleven access points with the channel plan built around the constraints that apply in Italy (DFS channels, weather radar zones). A web panel for moving a VLAN from one line to the other, tested with 10 trials that brought three defects to light, all of them fixed.
- Result
- The network was in service for the entire event, with redundant monitoring: automatic failback measured at about 18 seconds.
-
Major international cultural event Delivered
Access points prepared on the bench
- The problem
- Preparing new access points in a short time, with the correct firmware, names and addresses.
- What we did
- A bench with the controller on a laptop and the access points connected one at a time over PoE; a PowerShell script adopts them, updates the firmware, renames them, assigns the address and sets the controller address via SSH.
- Result
- About 5 minutes per access point, 8 prepared in one day, with the pitfalls documented for next time.
Secure publishing of web services
We publish applications behind reverse proxies, exposing the minimum; we manage certificates and DNS and restrict inbound access to the necessary ports. The principle is to look like nothing: no version in the headers, generic errors, silent closure for anyone who does not know the site’s name.
Technologies and products
- nginx and Nginx Proxy Manager
- Let's Encrypt
- fail2ban
- nftables
- Cloudflare
- DNS blacklist checks
Examples
-
Web services provider In production
A web application exposed only where needed
- The problem
- An application reachable from the internet was receiving scanner traffic and login attempts.
- What we did
- nginx forwards only two paths and closes everything else without replying; access allowed to a list of sources, a limit of 10 requests per second per address, 21 scanners recognised and rejected, four fail2ban rules fixed because they were not reading the logs.
- Result
- Attack surface reduced to two paths. When an automatic update crashed a third-party module, we identified it from the kernel log and disabled it.
-
Group in the distribution sector In production
Outlook getting the office blocked by the brochure website
- The problem
- The office’s internet address kept ending up on the blacklist of the hosting service for the company website.
- What we did
- We discovered that Outlook was querying the autodiscover service on the brochure website 378 times a day, and the hosting provider’s anti-bot filter mistook it for an attack. A Cloudflare rule, verified against 16 variants, answers those requests before they reach the site.
- Result
- Outlook’s requests no longer reach the site and the office no longer ends up on the blacklist.
Diagnosis and traffic measurement
We separate causes with tests that rule things out, counters and device logs. If there is no flow collection, we put it in place.
Technologies and products
- ntopng and nprobe
- NetFlow v9
- ClickHouse
- SNMP
- tcpdump and RouterOS sniffer
Examples
-
Company with video surveillance across several sites Delivered
Video surveillance events that did not reach the monitoring centre
- The problem
- The camera events were not reaching the monitoring centre, yet the links between the servers were all working.
- What we did
- The firewall logs showed that the event channel was travelling over UDP to APIPA addresses, advertised by the software of the servers’ USB management adapters. We disabled those adapters and restarted the service only where needed.
- Result
- Events are arriving regularly again, without touching the rest of the system.
-
Office of a service company In progress
PCs that would not update, due to three overlapping causes
- The problem
- Windows updates were failing on many of the office’s PCs.
- What we did
- There were three causes. We resolved two of them on the PCs; using TTL values and response times, we located the third in a transparent proxy run by the operator, after clearing the router and firewall with their counters.
- Where we have got to
- Two causes resolved, the third documented and in the operator’s hands.
-
Cloud services provider In production
Network flow collection for a cloud provider
- The problem
- There was no view of who talks to whom and how much.
- What we did
- nprobe as a NetFlow v9 collector feeding ntopng, with the archive on ClickHouse; we defined the local networks.
- Result
- Hosts recognised as local went from 0 to 82, with no flows lost.