Is it broken on my side or theirs? A short story of one outage
Everyone knows this moment.
You try to open a website, training platform or customer panel and suddenly nothing happens. The browser keeps spinning, then an error appears, and the classic end-user diagnostic procedure starts in your head:
Is it my internet connection? Is Wi-Fi pretending to work again? Is it DNS? Is the website down? Or am I clicking something wrong?
In the case of one e-learning platform, the situation looked exactly like that: the service stopped responding, and from the outside there was no clear information whether anyone was already aware of the issue.
First question: is it on my side?
Before blaming the server, hosting provider, DNS, internet operator, weather and phases of the Moon, it is worth checking the basics.
First, a simple reachability test for the IP address:
ping IP_ADDRESS
Result: no response.
ping alone does not prove everything, because ICMP may be blocked. Some servers simply do not respond to ping and continue living happily.
That is why the next step was testing HTTP/S.
curl -v https://example.com
In this case, the connection stopped while trying to connect to IP_ADDRESS:443.
So the browser was not just “having a bad day”, and the issue did not look like a typo in the address. The client was trying to connect to the server, but was unable to establish the connection.
Second step: are the ports alive at all?
Next, a simple port test:
nc -vz example.com 80
nc -vz example.com 443
Result: timeout on ports 80 and 443.
At this point, it looked less like a local browser problem and more like an issue with the service, server, hosting provider or the network path to it.
Third step: test from another location
A very important step: test from another connection, preferably from a completely different location.
If you test only from your home connection, the issue may still be:
- your router,
- your internet provider,
- local DNS,
- a routing issue between you and the server.
That is why it is useful to run the same test from an external VPS.
From an external server, the connection attempt to port 443 returned:
No route to host
And that was a strong hint: the issue did not look like a local user-side problem. Since another location on the internet also could not connect, the outage was most likely related to the service infrastructure, hosting, routing, firewall or the server itself.
So where was the problem?
Based on these tests, it was reasonable to say:
This is probably not my laptop, browser or home Wi-Fi.
It still did not tell us the exact root cause. It could have been many things:
- powered-off server,
- hosting outage,
- routing issue,
- misconfigured firewall,
- VPS provider issue,
- reverse proxy failure,
- failed configuration change,
- network problem in the data center.
But from the customer’s point of view, something else mattered more: the service was unavailable, and the user had no idea whether anyone already knew about it.
The worst part of an outage is not always the outage itself
Outages can happen to anyone.
Servers fail. Routing can play tricks. A firewall can become a little too ambitious. A TLS certificate can expire at the worst possible moment. And a “quick update” can sometimes turn into an evening with logs.
The real problem starts when the customer learns about the outage before the service owner does.
This is the unpleasant scenario:
“Hello, is your platform working?” “Is it not working?” “No.” “Oh…”
And suddenly the customer becomes the monitoring system.
But the customer should not be the monitoring system. The customer should be the customer.
What could have been done better?
It is not always possible to avoid an outage, but it is possible to handle it better.
1. Availability monitoring
The foundation should be simple external monitoring:
- does the website respond over HTTPS,
- is port
443reachable, - does the HTTP response return a reasonable status code,
- is the TLS certificate not about to expire,
- does DNS point to the correct address.
It does not have to be a huge enterprise monitoring system from day one. Sometimes checking every minute or every few minutes and sending an alert to email, Discord, Slack or a phone is enough.
The important part is that the service owner should know about the problem before the customer writes.
2. A simple emergency status page
The second thing: an emergency placeholder page.
Nothing fancy. A simple message is enough:
We are experiencing a technical issue. We are aware of the outage and are working to restore the service. Sorry for the inconvenience.
That is much better than silence, timeout and uncertainty.
Such a page can be hosted separately, for example on another hosting provider, as a static file, behind a simple reverse proxy or as an emergency DNS/Cloudflare switch.
Even a manual switch to such a placeholder page is better than a situation where the user only sees a connection error and does not know whether the service still exists.
3. A “what to do when it breaks” procedure
Small companies often do not need to start with a huge business continuity plan. One short instruction is enough:
- where the server is hosted,
- who has access,
- where DNS is managed,
- where the hosting panel is,
- how to check logs,
- how to enable the emergency page,
- where backups are,
- how to verify that a backup can be restored.
This can be a single page of documentation. During an outage, that one page can be worth more than five years of “it somehow worked”.
What would I suggest after such an outage?
In a case like this, I would not start with a revolution. First, the basics:
- external website availability monitoring,
- alerts so the service owner knows about the outage immediately,
- a simple emergency status page,
- short infrastructure documentation,
- backup and restore procedure verification,
- DNS, TLS, firewall and hosting review,
- a minimal incident response procedure.
Only after that does it make sense to think about more advanced things.
Because sometimes the problem is not the lack of Kubernetes, clusters or a cosmic architecture. Sometimes the problem is the lack of one simple question:
“How do we know it is down?”
And the second one:
“What do we show the customer when we already know?”
Moral of the story
Not every outage is a disaster. But every outage without information looks worse than it has to.
If a service is unavailable, the customer should see a clear message instead of guessing whether the problem is on their side, with the internet provider, in the browser, in the cable, in DNS or maybe in Mercury retrograde.
Monitoring, alerts, backups and a simple emergency status page are not luxuries. They are basic technical hygiene.
If you want to organize this area in your own environment, see the Server Backup and Monitoring service page.
Especially when a platform is used for customer communication, sales, training or everyday business operations.
Update: My own DR test
This post did not end with just a comment on someone else’s outage. I treated it as an impulse to check my own infrastructure.
My website, logos.net.pl, runs on a Raspberry Pi 4, and I prepared a VPS as a backup. I tested a manual traffic switch through Cloudflare Tunnel from the Raspberry Pi to the VPS and back.
I described it here:
The most interesting conclusion? The failover itself worked almost immediately. The actual problem turned out to be my local Pi-hole, which had cached DNS records and, for a while, pretended that the procedure was not working.