How to automate in an MSP when there is no real validation environment

Here we always talk about best practices, because it is the right thing to do and we were always good at school. But an enormous ocean separates IT theory from day-to-day practice… And it is full of monsters. For example, most MSPs do not have a validation environment that reproduces their customers’ reality.
They have not set up a twin staging environment or a reliable replica, and they do not have that pristine laboratory where they can test automations before touching what matters. They have production and an intern made of caffeine with a finger on the push button. But production is where the paying customer lives, does not care about excuses and calls breathing fire.
The instinct in this case is usually either to give up on automation “just in case” or to jump in headfirst and pray. The first of those doors condemns us to keep doing manually everything that erodes margin and consumes the team. The second is piloting at warp speed while blind, so you go very fast… until you stop going.
Here we will open a third door, because not having a perfect validation environment does not mean giving up on automation, but it does mean designing it differently. In a more cautious, segmented and, above all, reversible way.
Let’s see how.

Why a real staging environment rarely exists in an MSP

For once, this issue is not our negligence, seriously. So it is worth normalising it because, in reality, it is the very nature of the business.
An MSP manages heterogeneous customers. Each one has its own operating system, its own version, its patch pending since the Cretaceous, its particular topology… Replicating that in a test environment would mean twenty different laboratories and this is where perfect theory does not fit into IT reality.
To that we must add legacy infrastructures. Those systems that nobody dares to touch because nobody really knows how they work or which legendary character set them up, so they can rarely be cloned with guarantees.
But above all stands the supreme ruler, money.
Replicating representative environments requires hardware, licences and hours that almost no MSP can allocate per customer.
To rub salt into the wound, many parts of the managed infrastructure depend on external services (third-party APIs, integrations, the customer’s own systems…) that are simply impossible to reproduce in an isolated environment.
Deep down, there is a matter of control, because an MSP does not own the customer’s architecture. It manages it, yes, but it does not own it, which limits what it can build as a testing mirror.
In that real life, so different from theory, reliable staging, when we talk about multi-customer environments, is the exception rather than the rule, so, more than a shameful secret to hide, we are talking about a design constraint we need to work with.
But of course, that involves danger…

What risks appear when automating directly in production

Chuck Palahniuk wrote Fight Club and said that the only way to find happiness was to risk being cut open. But Palahniuk did not work in IT, and the first key to automating directly in production is understanding those risks, or we really will end up cut open.
The first and most feared one is the rapid propagation of errors.
Automation is, by definition, something we execute many times and very quickly. Wonderful when it works and a complete disaster when it does not, because a wrong script breaks two hundred servers before our jaw hits the floor when we see it.
Second, there are changes that are difficult to diagnose.
When something is applied massively and simultaneously, it is also likely to start failing massively and simultaneously.
In those cases, it is hard to know exactly what changed, in what order and where things started to go wrong.
And when what goes wrong is critical, it becomes a business challenge for the customer we promised their IT would run smoothly. We lose their trust, and trust is a porcelain vase: even if we painstakingly glue the pieces back together, it is never the same.
The rest of the bill arrives later, because those risks make us fearful and inefficient.
So we increase reactive supervision, because nobody can really trust it and we are watching everything all the time. Little by little, SLAs also degrade, and rollback (that step back nobody designed because “come on, what could possibly fail?”) becomes very expensive exactly when we need it most.

Which automations are especially delicate without real validation

After understanding the risks, we must understand the specific automations in the customer environment, because they are not all the same and our first exercise in caution is distinguishing the ones that bite from the ones that do not.
The group of those we should treat as if we were defusing a bomb includes:

• Massive configuration changes, which apply the same thing to many assets at once. If the template is wrong, the error will be massive.
• Automatic remediations in critical systems, where self-healing is great, except when the cure is worse than the disease.
• Automations that depend on specific topologies, which work brilliantly in customer A, the one they were built for, but land in customer B’s infrastructure as if it were Normandy, and Palahniuk is right because you end up cut open.
• Processes with unreliable thresholds (or thresholds not adapted to each customer’s reality) that trigger actions based on poorly calibrated metrics and act when they should not.
• Irreversible actions or actions with no possible rollback : deletions, migrations, overwrites… Here there is no second chance.
• Changes in services where a failure has a direct impact on the customer’s business. Here, an error turns into CEO calls, first from the customer’s CEO and then from ours talking about severance packages.

Now, we are not identifying what is critical in order to give up on automating it. The key is to reserve the most careful treatment for it out of all the approaches we will see below.

How to reduce risk when we cannot test in a twin environment

Let’s get to the heart of the matter. If we cannot test in a twin or similar environment, the strategy consists of turning production itself into our test bench, but in a controlled way, exposing each time only a small and recoverable portion of the total.
Life in IT is a casino, and it is worth following the golden rule for surviving them: Never bet more than we can afford to lose in each move we make.
That is why we do not act like James Bond and go “all on red”, but instead apply progressive deployment, hoping for the best but preparing for the worst.
This mindset is essential and, instead of applying that change suggested by the LLM to a thousand assets at once, we start with a small pilot group, observe while wiping away the cold sweat and, only if our nightmares do not come true, expand that deployment.
We will never be 007 with that attitude, but a phased activation (where each phase confirms that the previous one did not go up in flames) will make the Mandalorian nod and say: “This is the way”.
That piloting is based on segmentation, and this is where an MSP paradoxically has an advantage (about time).
That MSP works by customers and criticality levels (or at least, it should), which makes it possible to order deployments by starting with the least critical and most tolerant environments, leaving the services that cause the most heart attacks for last.
This progressive deployment is a horse we ride carefully, adding a series of additional security safeguards, such as:

• Defined change windows, so that, when something goes wrong, it does not catch us sleeping at three in the morning or happen during the customer’s sales peak.
• Minimum prior validations: even if we cannot simulate everything completely, we can always check preconditions before jumping in (does the service exist?, is there space?, is the asset where I think it is…?). Better a basic verification than a ready-made catastrophe.
• Automated post-change verification: after the change, the first step is to automatically confirm that the system is still healthy. If those checks pass, we review it manually in depth, with a level of dedication proportional to how critical the system is.
• Rollback designed from the start and with a cool head. Before launching anything, the question is not “will it work?”, but “how do I undo it if it does not work?”.

If we look closely, none of these measures requires staging, only judgement based on a doomsday prepper mindset and a little patience.
The bad part is that we often automate because of trends and FOMO, which makes us run around like headless chickens because “we do not want to fall behind and the competition has surely already applied it”.
The competition is going through the same thing as we are, and this connects with the typical mistakes when automating processes in an MSP. Spoiler: almost all of them have the same father, wanting to go too fast.

What role the observability of automation itself plays

Automation, by definition, makes us lose direct control, because we take our hands off the wheel to focus on other things and pray that self-driving does not end up wrapped around a lamppost.
But letting go of the wheel does not mean closing our eyes.
Hence the need to monitor automation so that it does not gradually drift towards that lamppost.
Spock said: “Logic is the beginning of wisdom, not the end”. Automation is also only the beginning, and final wisdom is knowing what the thing we set running on its own is actually doing.
For that, we need answers to very specific questions:

• Whether the automation executed correctly or got stuck.
• Which assets it acted on exactly.
• With what result.
• What actually changed in the system.
• What failed and where.
• What exception appeared that we had not anticipated.
• How much supervision work it generates, because an automation that forces you to constantly monitor it has not freed us from anything.

The point is that the quality of subsequent traceability partly defines how much validation in a previous environment (which we do not have) we can skip.
In the end, even with that previous laboratory, things will not turn out exactly the same, and the validation that counts is the one performed against reality.
But without visibility, automating production without staging is Bond’s “all on red”, which only works in the movies.

What changes when an MSP automates production with proper criteria

When we assemble everything above like the Avengers (progressive deployment, segmentation, planned rollback and observability), staging becomes less necessary and there will be signs that we are doing it right.
These are:

• Less improvisation. Because every change follows a known pattern, instead of depending on the inspiration of the technician on duty.
• Fewer massive errors, simply because we are no longer betting everything on red, but placing smaller bets.
• More confidence to scale. Because when we know that a deployment is gradual and reversible, we dare to operate more customers with the same team without our stomach tightening.
• And above all, there is better traceability and change control. Automation stops being a black box that depends on the whims of silicon and becomes a governed process.

The result is sustainable automation even when no real staging environment exists, which was exactly what seemed impossible at the beginning.

How Pandora FMS helps automate in a controlled way without a full validation environment

The most practical question is: what tools do we use to do all this without losing our minds?
Pandora FMS fits as the answer, but not because of magic or marketing, but because we designed and built it for multi-tenant battles.
Thus, segmentation by groups and customers is native. Organising deployments by criticality or environment (those pilot groups we talked about earlier) goes from being a matter of hacks and workarounds to the natural way of working with our tool.
On that basis, pre-change and post-change monitoring gives us the validations we saw could replace (at least partly) the previous laboratory. With Pandora FMS, we check the status before acting and confirm the result afterwards… automatically.
Controlled automation makes it possible to execute corrective actions and responses without giving up verification or traceability, which is Pandora FMS’s strong point. Since we are diving in headfirst without prior testing, at least we can closely observe every step we take to correct issues.
Our tool detects, acts, checks and records what it did and on what, which is what turns blind automation into controlled automation.
And all of this under centralised visibility from the Metaconsole, that single pane of glass from which we observe the effects of change in real time and across all environments at once. The dream of being Marvel’s Heimdall, able to hear even the smallest whisper in the nine realms.
Added to this are consolidated alerts and events, with correlation and filtering so we do not drown in noise when something strange starts moving.
And if security is comprehensive, Pandora SIEM adds visibility of this type of event to the same picture. If an automated change opens a door it should not to malicious actors, it will not go unnoticed by Pandora.
In the end, not having a perfect validation environment is not the end of automation in an MSP, because otherwise almost nobody would automate.
The key is to stop thinking of automation as a switch we turn on all at once to see what happens. If we approach it as a gradual, observable and reversible process (deploying in phases, segmenting by criticality, validating before and after, and using rollbacks), we mitigate the risks of not having the perfect laboratory (which will probably never arrive).
Will we make mistakes? Of course, that is real life in IT, but the laboratory does not vaccinate us against that either and, by following the best practices we have seen, those inevitable failures will be small and drama-free.