IT Notes - horrorstories

The Email They Shouldn't Have Read

Stefano Marinelli — Wed, 08 Oct 2025 14:25:36 +0200

Author's Note: Before we begin, an important clarification. What follows is a horror story based on real events from my career. However, to protect the privacy of the people and companies involved, I have deliberately mixed things up: technologies, contexts, and specific details have been modified or merged with other experiences. I therefore invite you to read this story not as a strict chronicle of a single event, but as an archetype of a widespread problem in the IT world: vendor lock-in and predatory business practices. Any attempt to identify the specific company or software described would lead to an incorrect conclusion. More about it here.

When the phone rang, I was in a meeting - so I didn’t answer. But I recognized the number and sent a quick message:

"I’m with a client right now. If it’s not urgent, please send me an email - otherwise I’ll call you ASAP".

The reply, via SMS, left me speechless:

"That’s exactly the problem. I can’t send you an email. Call me as soon as you can".

From that moment on, my perception of a certain kind of world changed forever.

A few years earlier, a major public institution - let’s call it Agency A - was still running an ancient Exchange mail server. It hadn’t received security updates for ages, the anti-spam was completely ineffective, and the new regulations were clear: embrace Open Source solutions whenever possible.

They had already received a proposal - expensive but, when compared to similar offers made to other organizations, apparently reasonable - for a managed service hosted by an external provider and based on an open source mail stack. The company offered a managed version with its own proprietary additions and enterprise support.

The catch? While such pricing had become almost "normal" in the market, it was still wildly inflated considering what was actually being delivered. Agency A already had solid infrastructure - reputable IP classes, redundant datacenters, everything running smoothly. We had built and maintained that environment for years, and it was still performing perfectly.

The request was simple: “Evaluate this solution, and if it’s suitable, we’ll migrate.”. About 500 active mailboxes, roughly the same number of aliases. Manageable, but far from trivial.

So I started experimenting. I had heard of that stack before but never used it directly. I deployed it in some non-critical environments - ours, and a few test clients who agreed to try it at a discounted rate. Everything worked flawlessly for almost a year. I began to appreciate its design and flexibility. Confident, I told Agency A we could proceed with a pilot migration.

We built a new server, deployed the stack, and assigned a few secondary domains for early adopters. The feedback was great - so good that users started pushing for a full migration.
The IT team planned carefully: created accounts and aliases, migrated selected mailboxes, and kept the old Exchange server online (hidden, for legacy access).

The morning after the MX switch I was tense, waiting for trouble - but it never came. A couple of small questions, nothing serious. The internal team handled everything perfectly. It was a success.

Word spread quickly.

Agency B - smaller, but in some ways more influential - contacted me. They were customers of the same managed-service company that had pitched to Agency A. Once they saw the potential savings (at less than a tenth of the annual cost), the stability, and the freedom of keeping their data on their own servers, they became very interested. Their contract, however, was a five-year deal with automatic renewal - two years left. The legal office said the notice period was six months, so there was time.

They wanted to prepare silently. Their supplier was known for aggressive commercial behavior and often retaliated when customers tried to leave. So we built everything quietly - users, aliases, test setups - and froze the system, waiting for the official termination notice.

That day finally came. The notice was sent, about eight months before expiration. Migration would begin upon confirmation of receipt - or the following month at the latest.

Meanwhile, I learned that Agency C - another institution - was also planning to leave the same provider. They wanted to keep the same software stack for consistency, so I told them about our experience. They asked for a quote, which I prepared (without mentioning Agency B, of course). My margin would have been small, but the project made sense: it was about owning your data, not making money.

Everything seemed to move smoothly - until that SMS.

I called back immediately.

"There’s been a problem with the termination" said the IT manager of Agency B.
"Somehow they found out what we were doing. There are hidden clauses we didn’t know about, and now we can’t leave - at least not for another five years. They know everything. Even your quote.".

I was stunned. How could they possibly know?

Minutes later, my phone rang again - Agency C this time.

"Forget the proposal", they said. "They called us. Threatened us, actually. They even mentioned your name and said they might take legal action against you for unfair competition - claiming they’re the only ‘authorized’ installers of that software. Which is absurd, of course. It’s open source. But our director doesn’t want trouble.".

Sometimes, when public money is involved, people prefer avoiding troubles over doing what’s right.

Something didn’t add up.
Then someone at Agency C noticed a clue: a former interim IT manager still had an email client connected via token authentication - with access to all messages. And that person had signed the original contract with the provider years before. Informally questioned, he admitted contacting them "to warn them" but claimed it was harmless. He never mentioned me - supposedly.

That still didn’t explain how they knew about Agency B’s internal steps. To test a theory, we set a trap: I asked a friend abroad to send Agency B a fake quote, from a company outside the EU.
The following Monday, the provider called Agency B and said, "We advise against working with non-EU companies - compliance can get tricky.".

That strongly suggested it: it looked as if they might have been reading the emails.

The IT manager exploded and confronted them. The response was chilling:

"I’m not saying we do - but we could. It’s in the contract. You should read the fine print, especially the unilateral amendment from two years ago.".

That amendment, quietly accepted, included horrifying clauses:
- notice period extended from 6 to 12 months
- formerly free services could become paid at the provider’s discretion
- and, "for security reasons", they could disable any access other than the webmail - which they promptly did.

All of this happened before the GDPR era, when certain practices could still slip through.

I tried to contact the company’s owner directly - no reply. Calls, emails, nothing. Their support lines were "not authorized to forward requests". I wanted to confront them about the “not accredited” nonsense and the so-called unfair competition. But bullies never like a fair conversation.

I urged Agency B and C to investigate - not only legally, but ethically.
They were horrified, yes - but in the end, nothing changed.
Worse: the provider, invoking that same contract amendment, made previously free features paid ones, increasing their costs by another 30%.
Management wasn’t outraged by the abuse - just by the extra expense, "hard to justify in the budget".

Years later, those directors were gone. The technical staff remained - older, wiser, and determined not to repeat the mistake. They eventually switched providers, though to something "safer", not necessarily better.

I couldn’t solve that problem. The battle had to come from them, and I would have supported them all the way - not for profit, but for principle.
Because when a company that claims to “support open source” behaves like that, we all lose.
We all get labeled the same way.

And that’s the real horror of the story - not the software, but what people do with it.

The Day GlusterFS Tried to Kill My Career

Stefano Marinelli — Wed, 21 May 2025 12:55:00 +0200

I was at a client, a healthcare facility, to replace some hard drives. The official line was a familiar one: no budget for significant upgrades. This meant we had to keep the current setup running, a system that was both outdated and, frankly, unreliable by modern standards. We were holding everything together with metaphorical duct tape and sheer willpower, but somehow, it was stable and functional. I had managed to set up a makeshift "cluster" using OpenNebula with GlusterFS for storage (which I later, through hard lessons, replaced with MooseFS and eventually Ceph), squeezing every last drop of performance from the available hardware.

The IT manager, a decent fellow, seemed to genuinely believe the purse strings were just exceptionally tight. However, another company, known to be friends of the general manager, were already providing support on two specific VMs. Their presence felt disproportionate to their limited scope, and there was a subtle, persistent pressure from the GM to involve them more, a context that would only become clear later.

We scheduled an intervention and meticulously notified everyone to disconnect and shut down their machines by 12:30. By 13:00, we had completed the necessary backups, aiming to start the core intervention by 14:00 and have the essential systems back up and running by 15:30. The goal was straightforward: update the underlying systems and check the health of the disks. We shut down all the VMs; everyone disconnected. It was lunchtime.

We updated the servers, rebooted them. And then, one of the disks started throwing errors. GlusterFS, for reasons I never fully pinned down but have strong suspicions about, decided in that critical moment to overwrite both the failing disk and its supposed replica with zeros. I hadn’t changed a single configuration file related to it. From that day forward, GlusterFS, for me, simply ceased to exist as a viable option.

Panic. Pure, cold panic – there were backups, of course, but on a USB 2.0 disk (these were old servers, no USB 3.0 in sight)! The restore time would be agonizingly slow. I immediately stopped everything, feeling like I was about to faint. The IT manager, seeing my face, didn’t immediately grasp what had happened, so I quickly explained the catastrophic data wipe on that specific storage volume. He composed himself and announced to the waiting staff that we would cancel the remainder of the planned intervention, restore from backup, and still aim to have the critical services back up by 15:30 as originally planned, prioritizing the most essential VMs.

Just as we were about to begin the painstaking restore, the "competing" company – the GM’s friends – reached out. They had, against all explicit instructions, powered on one of their VMs and started "doing their own interventions". And now, they were complaining that we had lost some of their data.

This was the moment the general manager and those guys went full "carpe diem". They immediately put me under accusation, loudly proclaiming I had undoubtedly made a critical error that led to this "lost data". It was then the penny truly dropped for the IT manager, and the GM's earlier insistence on involving his 'friends' company took on a sinister new light. Now I can say it: his primary goal hadn't just been about perceived cost-saving; it was to systematically give work to this company, to hand everything over to them, paving the way for them to take over the entire infrastructure, regardless of competence or genuine need.

I wrote a detailed technical report explaining exactly what I had observed, noting various unauthorized SSH logins from the IP addresses associated with those individuals during our clearly demarcated intervention window. Conveniently, the command history on the affected systems had been wiped clean. Of course, not by me.

They continued to harass me for a while. The last, most audacious thing they asked was for me to attend a meeting "to explain in person". They deliberately tried to schedule this meeting for the day before my wedding. And they knew it.

They threatened to demand unspecified (and undoubtedly high) financial compensation for "the lost data". What lost data, you ask? The data they claimed to have entered onto their VMs during our intervention window – the very window they'd been explicitly warned to stay out of and during which they had no business touching the systems.

Final result: no tangible problem for me. I hadn’t done anything wrong. The backups, though slow to restore, were intact and complete. They only calmed down when I proved (with network logs and firewall records in hand) that I wasn’t the only one connected to those machines during the critical period, and my witnesses (two colleagues present with me and the IT manager himself) had seen all my actions and could confirm I hadn’t deviated from standard, careful procedure.

In the end, I realized they would undoubtedly try something like this again. I made the decision to leave the client. Even the IT manager, who had initially tried to believe in the good faith of the leadership and had then witnessed firsthand the technical realities and the subsequent malicious accusations, decided he’d had enough and resigned to change jobs. He understood the financial and political interests at play were far beyond his control.

Unsurprisingly, the general manager managed to install the company he wanted, at astronomical figures. After just two months, true to form, they got their hands fully on the system I had painstakingly kept alive… and broke it. They actually had the gall to ask me for assistance, which I refused. At any price.

After a few years, I found out through the local press that the general manager had ended up in jail for corruption, bribes, and for systematically favoring his friends' companies across many sectors, not just IT. Don't ask me why, but I wasn't surprised at all.

That day, I celebrated. It was a stark reminder that while integrity might not always win the immediate skirmishes in such environments, the rot of corruption often consumes itself in the end. But the cost, for those caught in the interim, can be immense.

The Server That Wasn't Meant to Exist

Stefano Marinelli — Tue, 13 May 2025 16:13:36 +0200

Yesterday I read a piece of news that brought back an important - and painful - episode from my career.
A story about trust, technology... and the kind of problems that can't always be solved.

About 16 years ago, I was contacted by an old friend. He was worried about a situation involving some mutual acquaintances.
To keep it short: an entrepreneur - administrator and owner of several companies - had died suddenly.
He was the kind of man who centralized everything, and his wife and children found themselves struggling to manage things.
One of the sons decided to cash out and leave the family business (focusing on his own career), while the others chose to stay involved in day-to-day operations.
The wife, elderly and retired for years, ended up at the helm, but she was clearly out of her depth.

The main issue was the complete lack of information flow: no digital systems of any kind were in place.
Employees had their own PCs (sometimes even personal laptops), and there was zero control over anything.
All the accounting and administrative data were scattered across individual machines, often taken home at the end of the day.
From the owners’ perspective, all they saw was a huge cash flow coming in - yet the accounts were always in the red.

"If we keep going like this, we’ll be bankrupt in just a few years", I was told.
What I could do was set up a proper IT system, structured to make data management transparent and traceable.

I planned - and got immediate approval for - the purchase of routers, switches, various networking devices and a server with several disks.
The OS of choice, as was my habit at the time, was NetBSD. Thanks to XEN, I set up multiple VMs.
One handled the NAS duties (using Samba, so PCs could connect and store files directly there), another ran Archivista.
I even worked on translating Archivista’s interface into Italian, since it wasn’t yet localized, just to make it easier for users.

As usual in those days, I added a caching proxy (Squid) and a content filter (DansGuardian), to ensure proper usage.
The internet connection was very slow and often collapsed under heavy load - mostly recreational use, as logs revealed.
There was no supervision, and many people were downloading movies and such on company time.

As often happens, not everyone was happy.

One figure in particular - the late owner's former right-hand man - opposed the new system in every possible way.
According to him, none of this was necessary. But the real alarm bell had been his sudden change in lifestyle.
He’d made purchases that didn’t remotely align with his salary.
At the time, the company had no oversight and dealt with a lot of cash. It was all technically legal, especially given the nature of the business. I won’t go into detail - privacy matters.

Once everything was up and running, we trained the employees to use the new system.
Most were thrilled - finally able to work properly, with files in the right place and centralized document management.
OCR, archiving, etc. were all seen as major time-savers and a big boost in efficiency.
Some of the accounting staff remained skeptical, of course.

Since all of this was far from where I lived at the time, I went back to my life once the system was stable and everything in place.
A few quiet days passed - then one morning, my phone rang:

"Good morning, this is XYZ - I handle some technical aspects of a software suite used by the company where you've just installed everything.
We need to install our software, and I understand you set up the server. I’ll need the full server diagram and all the admin passwords".

I explained that it wasn’t a Windows machine, as he assumed (without even having seen it), but NetBSD, running NetBSD and Linux VMs.

A few seconds of silence.

"I see. Then I’ll have to wipe it and install Windows. I need Windows, and I don’t have time to wait for a new server - I’ll proceed tomorrow morning".

I froze. I told him that was not possible - the entire workflow now depended on that machine, and erasing it would be catastrophic.

"I’ll speak with the owners", I said, "and I’m sure they’ll provide you with a separate server within hours".

No use. He started to backpedal.
To my (young) eyes, the goal was now obvious: that server had to disappear, and fast.
He said he would "restore the previous situation", and claimed the server couldn’t remain as-is because he needed it.

I immediately called the owners. Sadly, due to inexperience and inability to handle the situation, they panicked.
They asked me to consider letting him do it, and then redoing the setup later, covering the cost of new hardware and my time.
I refused. I was young, but I already had this mindset: do what’s right, even at the cost of profit.
This was clearly a maneuver to eliminate controls - the server, the centralized filesystem.
The goal was to hide the real accounting data from owners and auditors. Thousands of euros vanished every day through "transactions".
The owners had started to understand, and this new pressure confirmed just how rotten things really were.

I called that man back and told him clearly: the server I’d built wasn’t to be shut down.
If he needed one, I’d deliver a new server by that evening, just for him.

At that point, he finally spoke more openly:

"You don’t get it, do you? I need that server. Not a server.
You’d better go along with this — or you’ll have serious trouble working in this area again".

And other similar, "nice" sentences. I replied calmly:
"Look, I’m just doing a favor for some friends. I don’t have clients in your area — and I don’t want any.
I’d rather do a good job for them than gain new clients".

No way through. He kept pushing - confident in his own sense of "power" - until I said what I had been trying to avoid.

Because I had recognized who he was.
And the disappointment hit me twice as hard - I used to admire him.
He, however, had not recognized me.

"Excuse me, but why are you talking to me like this? You’ve known me since I was a child. Don’t you remember? I’m the nephew of..."

He froze.
He understood immediately.
He connected the dots and knew full well that a single phone call to someone extremely close to me - someone he owed a great deal, both personally and professionally - would have the opposite effect he was aiming for.

That person had helped him greatly over the years. So much so that he folded:

"Oh... I’m so sorry... I didn’t recognize you. I’ll find another solution. Sorry again".

He hung up.
Never heard from him again.

I informed the owners that the issue was resolved (leaving out most of the details), but it didn’t last long.
Within days, a series of "unfortunate events" hit the server: the UPS failed, the server was "accidentally" unplugged and plugged back in incorrectly, and finally... it stopped responding on the network.

It was dead.
And when we opened it, the hard disks were just... gone.

But there was one thing nobody (but the owners) knew.
The server - slowly but surely - had been backing up externally.
All the data up to that point had been copied to a device we’d quietly installed at the owners’ home: a tiny PCEngines Alix, running NetBSD with two USB drives.
It was slow, yes - slow hardware, slow disks - but reliable.
That very device still works today (with FreeBSD) and provides services elsewhere.

I handed all the data to the owners and asked what they intended to do.
They took some time - days, then weeks.
Eventually, they said they’d probably investigate whether there were grounds for a theft report.

I never heard more about it.

But then came a tempting offer:

"Come work for us. Manage our network infrastructure and help us overhaul our internal procedures.
Even if you’ve just bought a house far from here, even if you’d have to leave your other clients -
we’ll pay you enough to forget everything else. Name your price".

They would’ve done it, too.
Our mutual friend urged me:
"They’ve got a huge cash flow, but too many people are taking advantage of them due to lack of control.
Take the job - they’ll treat you like gold, and you’ll really help them".

I didn’t think twice.

I turned it down.

I like my work.
I like doing what I do - and the income is a consequence, not the cause.
I would’ve had to give up my life, my path, to fight battles I don’t enjoy - and that I might not even win.

Because sometimes, dishonest people do win.

I’ve never regretted declining that offer.
I lost touch with all of them years ago, but I later heard things went as I predicted:
the owners gradually backed out.

They made another request later on, which I tried to fulfill — but even that was blocked, just when everything was ready.

At some point, I had to walk away.
Not because I wanted to abandon them in a time of need, but because they weren’t giving me the tools to do what was necessary.
They were so overwhelmed, so unprepared, that they ended up yielding to pressure - often from the very people who were hurting them.

And of course, I’ve left out the worst parts of the story.

(Author's note: Many readers, understandably struck by the severity of the events, have speculated about the involvement of organized crime. I want to clarify that, while the situation was extremely problematic and dishonest, that wasn't the case. The "worst parts" I alluded to referred to other internal dynamics, abuses of trust, and improprieties that I prefer not to detail further for privacy reasons and to avoid weighing down the narrative.)

That’s when I realized:
Some situations are so rotten, they simply can’t be salvaged.

And that’s okay.

I solve problems - it’s what I do best.

But I can’t solve every problem.
Especially not when those involved choose to protect the problem instead of fixing it.

I Almost Died for a Full Sentry Database

Stefano Marinelli — Sat, 28 Dec 2024 09:00:00 +0100

Yesterday afternoon, a developer notified me that Sentry is down again. I check: 200 gigabytes of database filled in just a few hours. And it reminds me of the night, years ago, when I almost "died" because of a full Sentry.

A client had launched a new service with great fanfare. After a few days, the attention was through the roof. While some of his devs were pushing to move "to the Cloud", he had decided against it. He always had a vision similar to mine. Thanks to his competence and clear ideas, he had managed to convince several investors to sponsor the project. Things were going well. This person was highly skilled, both technically and administratively, but, after the initial development, he became very absorbed in everything surrounding the project.

The two dedicated servers in place were handling the load with ease, even under heavy traffic. Just over €100 per month for thousands of concurrent users. Good implementation, good optimization, backups every 5 minutes via zfs-send and zfs-receive to a third backup server.

The only issue: some devs had requested Sentry for monitoring. In this case too, we opted for a self-hosted setup. A dedicated server was set up just for Sentry: 2 TB of fast storage. After a week, it had used just 2 GB of disk space. Everything was fine, I was feeling confident. But...

...one afternoon, something went wrong. Suddenly, the web apps on both dedicated servers became extremely slow. Then unreachable, despite their minimal load. Panic ensued. Some devs started chanting the usual mantra, "we need more powaaaar!!!" but, in reality, the servers were under zero load but had a ton of outgoing network connections.

I investigated and realized that everything was grinding to a halt when calls to Sentry were made. I logged into the Sentry server: full. And this wouldn't have been an issue if it weren't for the fact that the production apps would freeze if Sentry didn’t respond. It had been implemented in a blocking way, with a timeout of 300 seconds. The notifications about that server were sent directly to the devs, but it seems that the common alias they provided me actually pointed to nothing more than a test mailbox, which was abandoned right after the account was created.

The project manager identified the issue as a deployment made just a few hours earlier by a dev. They rolled back the changes. Everything returned to normal. We chalked it up as a bad experience. The dev in question began arguing that we had taken the wrong approach because, in the cloud, "this wouldn’t have happened". I thought to myself (but kept it to myself) that if he had written proper code, this wouldn't have happened either. But in his eyes, I was probably just the sysadmin being a pain and judging the "quality" of his work. In the cloud, much of this is often hidden by autoscaling or other abstractions.

Two days later, I had a high fever. The doctor prescribed some medication, but I had a pretty bad allergic reaction to one of them. By the evening, I was feeling awful and went to bed. I was exhausted, so I fell asleep immediately. In the middle of the night, a series of notifications woke me up: everything was down again. Even though I was sick, I got out of bed and rushed to the computer. Another deployment had been made two hours earlier, again by that same dev. Sentry was down again. Full again.

This time, I acted urgently and independently. I shut down Sentry and configured the reverse proxy to respond with a "200" to every call, just to keep the application running. Emptying Sentry would have been complicated, given that a vacuum full in PostgreSQL requires space - which we didn’t have - and time. But I wasn't feeling well. Not at all. I got up, and my wife followed me, seeing how unwell I was. I didn't even check if the alerts had cleared; I ran to the bathroom. That's all I remember. My wife heard a thud; I had fainted and hit my head on the floor. Luckily, I had been sitting when I fell, so the bump wasn't too bad. Shortly after, I woke up. The first thing my wife, worried but knowing me well, said was: "The alerts are clear; the servers are up". I relaxed. I had a double espresso, pulled myself together, and went back to bed, though I couldn't sleep.

I sent an email to the project manager and told him: I can't keep this setup running. I'm not capable of jumping out of bed every time Sentry crashes and everything stops because of it. It fills up too quickly, and I don't think there's any reason to log and retain that much data. Even less so for the calls to be blocking. In my opinion, that whole part of the development needs to be reviewed.

The dev kept insisting that we needed to move to the cloud to avoid these issues. I pointed out that in the cloud, it would have been the same unless we set extremely high limits. In that case, it would have been costly, and we'd still just be kicking the problem down the road. Code needs to be written properly; you can't just waste money and resources endlessly to cover up inefficiencies.

Since replacing the dev wasn't an option for various reasons, they decided to humor him and moved Sentry to a cloud instance, partly to free me from the situation. I was happy to only manage the production servers. It's funny how nowadays, "metrics" and "error tracking" seem to matter more than service stability and efficiency. Anyway, they didn't set strict limits - but they did fix the code. But... a few days later, the dev once again made an "incorrect" commit, and the wild logging resumed. This time, no one noticed: the cloud provider's alerts were ignored, ending up in that secondary project mailbox which, as it turned out, no one checked anymore after the accounts were created.

This went on for a week. Then two. Then three. The dev, at this point, sent an email "boasting" about how his idea of moving to the cloud had been successful for service continuity. After a month, the bill arrived. No one had realized what was happening. They never told me the amount, but I know a good portion of the remaining budget for the project's development and promotion went up in smoke. And when an investor found out what had happened and how all that money had been burned, they pulled out. I can only imagine: if in the early days they filled 2 TB in just a few hours, with traffic increasing daily over a month...

In the end, the project failed, and this event, while not the sole cause, significantly undermined the credibility of how the money was managed.

Basically, I almost died for nothing. :-D