This is a case study showcasing an engagement from our NOC Operations Consulting service. Get the full PDF of the case study here.

A software company that builds digital banking platforms for financial institutions was growing fast. As its customer base expanded, so did the demands on its Integrated Operations Center (IOC), and the extra load exposed problems in how the IOC handled incidents, staffed its shifts, and managed its processes.

The strain was showing up in the team’s everyday work:

• Incidents were handled differently depending on who picked them up.

• Alerts had to be copied by hand before anyone could work them.

• Escalations moved over email.

For a company whose customers are financial institutions, that’s an uncomfortable place to be. Those problems put the IOC’s ability to keep service steady for them at risk exactly when the company expected to keep growing.

We were brought in to assess the IOC and recommend what it would take to turn it into an operation that could scale with the business. This was an assessment engagement, so the outcomes described near the end of this piece are projections tied to the recommendations.

The operation we walked into

The IOC’s incident work was spread across SLAB, PagerDuty, Salesforce, and several homegrown tools. PagerDuty raised alerts while SLAB tracked incidents. The two weren’t integrated, and nothing really tied the rest of the stack together either.

On top of the tool sprawl, the team covering all of it was small relative to the workload. A few key people carried 24/7 coverage between them, and turnover was unsurprisingly high. There wasn’t a structured training program or defined career path, and new hires were often put straight into their roles without the onboarding the work demanded.

The IOC also ran remote-first. Combined with the turnover, that left people working largely on their own. Staff felt isolated, and the team lacked the more intangible (albeit critical) level of cohesion that knowledge sharing and consistent handoffs depend on.

How we ran the assessment

Over five weeks we looked at four areas: people, processes, technology, and business alignment. We interviewed stakeholders and staff, and analyzed the IOC’s tooling and processes alongside what they told us.

In the interviews, employees said they felt underprepared for complex incidents. Managers said they had trouble keeping incident handling consistent from one shift to the next.

Both groups were essentially describing the same problem from opposite ends of it.

The findings

Here’s a quick breakdown of the specific things we documented from our assessment.

How the findings fed into each other

These problems compounded.

Manual tooling made every incident slower, skill gaps pushed more incidents up to senior staff, and without continuous skills development, the IOC couldn’t build the capable, resilient team it needed, so the gaps kept reopening as people left.

Mark Biegler, who led the assessment, put it this way:

“When we first looked at the IOC, it was clear the team was putting in tremendous effort, but they were trapped in a cycle of manual processes and disconnected systems that slowed them down.”

Effort wasn’t the constraint. The structure around the team was.

What we recommended

The recommendations we handed to the company addressed the immediate problems while laying out what it needed to build give the IOC a base it could scale on.

Automate event management and bring the tools together

We recommended integrating SLAB and PagerDuty with a centralized ITSM platform so a ticket is created automatically for every monitored event. When PagerDuty raises a server alert, the corresponding ticket opens in SLAB on its own, and incident management starts without anyone retyping anything. Every alert gets tracked from detection to resolution.

Alongside that, we developed a plan to automate event correlation. When a server outage throws alerts from several parts of the infrastructure, the system groups them into a single incident. That cuts the noise from duplicate alerts and removes a source of human error during the incidents where errors cost the most.

The larger goal was consolidation. We recommended bringing all monitoring and ticketing into a single platform that handles both infrastructure and application events. The team would work from one dashboard with real-time tracking across every system, and they’d prioritize incidents from a complete picture instead of switching between tools to build one.

Formalize incident and change management

We designed standardized workflows for incident and change management so every incident, whatever its severity, follows a defined process. That included SLAs for response and resolution times, plus automated escalation paths that put high-priority incidents in front of senior staff immediately.

We also proposed defining the company’s NOC services in a service catalog. Incident Management, Change Management, and Major Incident Management would each have defined SLAs and OLAs, so performance and customer impact could be tracked service by service.

Build a CMDB the team can rely on

We recommended a CMDB mapping all infrastructure and application components across the company’s services. With that map in place, the system can automatically identify which services or components an alert affects, and the team can see the dependencies behind an incident without having to work them out mid-outage.

Report on the numbers automatically

We advised automated reporting on Mean Time to Repair (MTTR) and First-Level Resolution (FLR). The point was to let leadership see where the incident process was getting stuck and take proactive steps to fix it.

A few examples of the kinds of reports we build:

Invest in the team

The people side of the plan started with knowing where each person stood.

We recommended a competency matrix to assess every team member’s current skill level against the technical and operational needs of the IOC. That shows where each person needs to develop and keeps training pointed at the skills incident management requires.

New hires would go through a structured onboarding process of six to seven weeks, tracked in a learning management system. Established staff would get access to ongoing certifications, including AWS and Azure, along with ITIL training for service management.

Regular technical workshops and incident simulations would keep development continuous. The workshops cover root cause analysis, incident triage, and system diagnostics. The simulations recreate real scenarios, so people have worked through a high-pressure incident before they face one live.

Career progression paths go after the turnover directly. Clear advancement gives people a reason to stay, which cuts how often the company has to recruit and train from scratch. It also means less experienced team members grow into handling more on their own, escalate less often, and leave senior staff free for the complex work.

For a remote-first team that felt isolated, we suggested regular team meetings, virtual check-ins, and a mentorship program pairing new hires with experienced staff. The aim is a team that shares what it knows and handles incidents the same way regardless of who’s on shift.

Growth exposed the problems here

The IOC had been getting by on effort, and effort stopped being enough once the company’s growth outran its tools and its team.

The tooling findings were the most visible part of the assessment, and integrating the tools is what makes the team faster. Mark is clear about where the larger change sits, though. “The real transformation came through investing in the team itself,” he said.

If your own operation looks like this one, our NOC Operations Consulting practice starts the same way: an outside assessment across people, processes, technology, and business alignment, followed by recommendations and help putting them in place without pulling your internal team off its day jobs. Contact us and we’ll follow up within one business day to set up a time to talk.

About INOC, a service of Xerox IT Solutions

INOC is an ISO 27001:2022 certified 24×7 NOC and an award-winning global provider of NOC Lifecycle Solutions®, including NOC support, optimization, design, and build services for enterprises, communications service providers, and OEMs. INOC solutions significantly improve the support provided to partners’ and clients’ customers and end users.

INOC assesses internal NOC operations to improve efficiency and shorten response times, and provides best practices consulting to optimize, design, and build NOC operations, frameworks, and procedures. Proactive 24×7 NOC support is provided with several options, including North America, EU, or APAC only or global integrated NOCs. INOC’s 24×7 staff provides a hands-on approach to incident resolution for technology infrastructure support.

Learn more about our NOC support and NOC operations consulting services. Get in touch to start the conversation. We’d love to talk NOC.

Get in touch with us

Staffing is one of the biggest factors that drives organizations to strategically outsource their network operations center. If you’re running it 24x7x365, that means three shifts a day, every day, forever.

We’re often asked how many people are actually needed to cover a NOC around the clock. That question is obviously important, but it’s also in some ways the smallest part of what’s actually being asked. The number that keeps the lights on and the number that keeps an operation working well are often different — and it’s almost always more heads and hands than people assume.

While there’s no standard handbook for this, there are some common steps that we’ve found have to go into the process of resourcing a modern support operation.

Many teams end up guessing at how those pieces fit together, and the business absorbs the cost of the guesses as outages while it gets sorted out.

Then there’s the part almost nobody plans for, which is that none of it ends. A NOC isn’t something you simply “stand up.” It’s something you manage continuously, and it can punish the business in both directions if it isn’t carefully managed.

• Understaff it and a small team takes on too much until it burns out.

• Overstaff it and you’ve spent money without touching whatever operational problem was generating the workload to begin with.

What’s changed more recently is mostly on the input side. The breadth of technical knowledge a NOC engineer needs keeps widening, the market for people who have it is tight, and AI is adding demands that even long-tenured engineers haven’t dealt with before. None of that has made the math of staffing any easier.

Here’s the quick rundown of what staffing a NOC takes: the coverage math, what that math leaves out, and the hiring and retention work that ends up costing more than either.

1. Basic headcount for a NOC

The staffing question we field more than any other is how many people it takes to cover a NOC 24/7.

The answer is 10 to 12 at an absolute minimum, and most NOCs need more. If you’re running five people and telling yourself you have coverage, it’s a fragile place to be. You have a rotation that will almost certainly break the first time two of them are unavailable in the same week.

But 10 to 12 is a coverage floor, not a staffing plan. Here’s what sits on top of it:

There’s also the measurement piece: how do you know if you’re getting staffing right? Most teams try to answer this with KPIs and SLAs, which measure how well the NOC is performing at its job, but say almost nothing about whether it’s staffed correctly. The metrics that answer the staffing question are utilization metrics, and they’re a blind spot nearly everywhere.

Here are three key utilization metrics we use to measure our own staffing efficiency and help teams measure theirs:

Labor content for each edit of a ticket

This tells you how much work a ticket’s lifecycle actually consumed. The central distinction here is that ticket duration is not the same as labor content. Teams almost always conflate them. Duration doesn’t capture how much human labor was spent each time the ticket was edited because of the time that exists between edits while the ticket sits open.

A real ticket lifecycle looks like: created, picked up, edited, assigned a status, returned for another edit, maybe another, then resolved. Our own average is roughly 4.3 edits per ticket given the environments we support. An example is four edits at 20 minutes each, which is 80 minutes of labor content on a ticket whose duration might read as days.

Once you have the average, you extrapolate. 20 minutes of average labor content across 1,000 tickets gives you a real labor requirement you can staff against.

We always tie this directly to OPEX and morale. Without it, you get stuck in a bad (but very common) cycle: leaders can’t investigate why everyone is busy, staff feel destined to burn out, low morale drives churn, churn adds hiring cost, and the understaffed NOC then asks more of whoever remains while they still have to hit their SLAs.

Staffing correctly lets you be one of the rare operations that can actually step out of that.

Number of edits processed per hour

The simplest of the three here. It measures how many edits in the ticket lifecycle one person can perform per hour.

Its value is almost entirely relational. If labor content for a given ticket type is 15 minutes, you can reasonably expect a maximum of four per hour at 100% efficiency. This metric isn’t useful examined alone, only as a component of the larger picture.

It’s also the metric that lets you benchmark individuals. It gives the data to say one engineer is doing 50% of what they should and another 150%, and making staffing and coaching decisions from there.

A heatmap of edits by time of day and day of week

This is arguably the most important metric for making staffing decisions, because it answers the question teams actually need answered: how many people do I need in the NOC at a given time?

Here’s an example of a similar report (created tickets) to give you an idea of the insights you can almost immediately glean from it:

A well-designed heat map tells you that at 8:00 a.m. on Mondays you should expect to work up to 60 tickets. Stack the other two on top and it becomes 60 tickets on Monday at 8:00 a.m., each with four edits taking 20 minutes, which converts directly into a shift level instead of a guess.

Capturing these shouldn't be particularly burdensome, since it's mostly a matter of configuring your tools. Reporting and visualizing them is the bigger lift for a team that's metrics-deficient generally. Talk to us if you want to set this up within your own operation.

More on metrics and reporting in the NOC:

2. Structure comes before headcount!

We address headcount here first because it’s what most teams think of first when they’re staffing. But the most common sequencing error we see is hiring first and building the structure around whoever end up on the team.

Operational structure comes first. It tells you how the work will move, which tells you who you need.

A tiered support structure is the usual starting point for a NOC or similar support-focused operation that is dealing with a significant volume and variety of events.

We wrote about our tired structure extensively here:

In short though, Tier 1 works the monitoring tools and sits at the center of the flow, connecting to end-user help desks, to Tier 2 and 3 engineers, and out to third parties. Working that way, we’ve seen NOCs, our own included, resolve 65 to 75% of incidents at Tier 1.

That number is the whole economic argument for the structure. It means most of your volume is handled by the tier it should be handled by, and your senior engineers aren’t spending their week on break-fix.

Our structure has distinct operational layers, each with a defined scope of responsibility, and clear routing between them.

• Alarms enter from the top.

• Support requests enter from the bottom left.

• Everything flows through defined paths to the right people at the right tier.

Workflows come next. Issues get prioritized into queues, each one handled by the group with the right skills, with SLAs and technology and technician skill level determining how the queues are cut. Without that, you get the wall of red and a team that treats everything as equally urgent.

Then, and only then, the org chart.

Once you have roles on paper, map them properly. A RACI chart is worth the afternoon it takes: for each task, who is responsible, who is accountable, who gets consulted, and who gets informed. Build it with the other teams in the room rather than alone, since the NOC touches most of them.

And be honest about the process list you’re signing up for alongside the people:

• Escalation

• QA and QC

• Disaster recovery and continuity

• Onboarding

• Training

• Continuous development

• Workflow management

• Incident, problem, and knowledge management

• Access and credential management

In many NOCs, every one of those needs an owner, and the owners are people you have to hire.

3. What skills you're actually hiring for

Technical skills are the easy part to specify and the easier part to teach.

The modern NOC engineer needs breadth across network technologies, cloud, server operating systems, virtualization, storage, and applications. That breadth is getting harder to find each year, and AI is adding demands that even experienced engineers haven’t faced. Plenty of seasoned people have never worked on a network that reasons about its own traffic.

But after hiring into hundreds of NOC and NOC-adjacent roles across more than twenty years, the pattern we’d point to is that the soft skills are the ones you can’t train your way out of like: Communication in writing and out loud, urgency, accountability, empathy, customer service instinct.

Hire for those and the technical gaps close. Hire against them and they don’t.

It also matters which tier you’re filling. Tier 1 candidates need to read and interpret documentation quickly and follow a written procedure under pressure. That’s a different aptitude from troubleshooting. As you move up the tiers, the weight shifts toward investigation and diagnosis.

4. The part that costs more than hiring

Getting people in the door and into the NOC is the visible cost. Keeping them is the larger one, and it’s mostly a design problem to solve.

Start from the premise that most people don’t want to be NOC engineers forever. That’s fine! The NOC is a strong first step into a lot of disciplines, and the operations that retain well are the ones that treat it that way. Lay out paths toward technical specialization and toward management. Promote people off the frontline visibly, so the rest of the team can see the path is real.

A few things that work well for career progression here include:

There’s also a company culture question underneath all of it. A team working ticket to ticket, optimizing for how fast they can close, ends up in a permanently anxious environment that managers have a hard time leading their way out of. Shifting from a ticketing culture to a customer-driven one is slow and needs constant reinforcement, but it’s the difference between a NOC people stay in and one they pass through.

One last thing that catches people: When a team outgrows its management, no warning light comes on. It happens quietly, and the first symptom is usually that nobody’s growing anymore because the manager no longer has the hours. Watch the ratio deliberately, because it won’t announce itself.

5. Where staffing usually lands

Put the pieces together and you can see why so many teams end up strategically outsourcing some or all of this.

A build takes 16 to 24 weeks minimum before the parts are even in place, and months or years beyond that to reach real operational maturity. Then the recruiting, training, scheduling, and quality work continues indefinitely. Many companies that move to us cut their total cost of ownership roughly in half, mostly because an economy of scale absorbs costs that a single in-house team has to carry alone.

One caution on the middle path: outsourcing only nights and weekends solves a scheduling problem and creates a management one, since you’re now maintaining quality and consistency across two teams instead of one. It can work. But it needs to be designed rather than improvised.

The clearest way to think about it is that the question isn’t whether a 24x7 NOC is worth having. It’s whether building and continuously managing the staffing machine behind it is the best use of your team’s attention, given everything else they could be doing.

For most companies we talk to, the answer turns out to be no.

We've also written two longer guides on this subject: Staffing a 24x7 NOC: Costs, Challenges, and Key Considerations and How to Build and Manage an Effective NOC Team.

About INOC, a service of Xerox IT Solutions

INOC is an ISO 27001:2022 certified 24×7 NOC and an award-winning global provider of NOC Lifecycle Solutions®, including NOC support, optimization, design, and build services for enterprises, communications service providers, and OEMs. INOC solutions significantly improve the support provided to partners’ and clients’ customers and end users.

INOC assesses internal NOC operations to improve efficiency and shorten response times, and provides best practices consulting to optimize, design, and build NOC operations, frameworks, and procedures. Proactive 24×7 NOC support is provided with several options, including North America, EU, or APAC only or global integrated NOCs. INOC’s 24×7 staff provides a hands-on approach to incident resolution for technology infrastructure support.

Learn more about our NOC support and NOC operations consulting services. Get in touch to start the conversation. We’d love to talk NOC.

Get in touch with us

Gartner (now infamously) reported back in 2019 that only about 25% of organizations investing in a CMDB felt they were getting meaningful value out of it. That number gets quoted a lot, usually as evidence that CMDBs are a lot of work for not much return.

We’d put it differently. Most of what gets called a CMDB is actually just an asset inventory with a more impressive name.

An inventory answers only one question: what do we own? That’s important to answer sometimes. It’s just not the question anyone is asking at 2 a.m. when they’re trying to see which services were affected by an outage.

When an alarm fires, the questions are different.

An asset list answers none of those, which is why teams who built one keep finding that it doesn’t do much for them during an incident.

Our own CMDB here at INOC has eight core kinds of data in it. Asset data is one of them.

Let’s run through each dimension, because this is what most NOCs should have.

What broke and where that device is located

Asset data is the foundation, and it’s the part most teams already have in some form. Make and model, serial numbers, device type, vendor, platform. The physical and virtual things that generate alarms and need monitoring.

Location data is where it sits. Site name and exact coordinates so it can actually be found.

This sounds trivial until you watch an engineer try to dispatch someone. A technician who can’t tell you which building, which floor, which rack, is a technician who is about to spend twenty minutes on the phone finding out. Every one of those minutes is inside your resolution time.

Circuit data is the layer most inventories skip entirely, and it’s where a lot of resolution time disappears in service provider environments. There are some specific things you need documented here:

Who it affects

Here’s where an inventory stops being useful and a CMDB “starts” from a functional standpoint.

Service data covers the things customers actually pay for. That’s usually email, internet access, and LS links. Any services that ride across circuits but aren’t the same thing as circuits.

This distinction has actually become more important over time, not less. Services used to get modeled as circuits. Modeling them as application services instead draws a clear line between physical connectivity and the customer-facing offering, which matters a lot in cloud environments.

Customer data closes the loop here. It’s contact information so you can reach the affected parties and they can reach you about things that never triggered an alarm. It’s also any service details, so engineers know what each customer depends on.

One that gets overlooked here is terminology alignment. Clients and customers describe their services in their own words, and those words often aren’t yours. Getting the vocabulary matched in advance prevents a category of confusion that shows up right when nobody has patience for it.

Who you call or otherwise contact

Third-party contact data is the least “glamorous” item on this list but it’s one of the biggest time savers. We attach the correct vendor support numbers, emails, and portal URLs right to the asset record so an engineer opens a case instead of searching for how to open a case. Carrier contacts with the account and circuit identifiers are already alongside them.

What makes this work is being precise about who is who. Field technicians and carriers have different responsibilities, and calling one when you needed the other wastes a cycle. Customers and venues aren’t the same either. A customer may own the service while a venue is accountable for the physical space and hardware. Vendor technicians are a third category again.

Get those distinctions into the data and engineers stop having conversations that were never going to help. Simply putting this into an actual database for reference pays massive dividends in stress and time reduction.

What to do about it

Our knowledge articles live in right inside the CMDB for a reason. They’re what turns a diagnosis into an action, and they’re how a practice that lives in one experienced person’s head becomes something the whole team can execute.

Two rules keep a knowledge base useful rather than overwhelming.

Whatever you add here, it has to be accurate, reachable during an incident without opening four other systems, and reviewed on a schedule. An article that no longer matches the network is worse than no article, because it costs the whole library its credibility.

The piece that makes the rest of it work

Alarm data is the eighth ingredient in the recipe to an effective support-focused CMDB, and it behaves a little differently from the others. We don’t store it as CMDB data. But it has to match the CMDB data, and when it doesn’t, everything downstream breaks in a specific and predictable way.

Here’s the chain within our NOC operation to show how it works:

If the alarm doesn’t match the CI, none of that happens.

The ticket gets misprioritized, so a Priority 1 incident sits in the queue looking like a Priority 4. The customer notification never fires. The customer eventually notices the problem themselves and calls to ask about it. The NOC, having no record connecting that call to the alarm, opens a second ticket to investigate.

One naming mismatch, and you’ve produced a delayed high-priority incident, a customer who found out on their own, and duplicate tickets.

This is why CMDB data quality is a major onboarding focus for us, and why we hold clients to change processes that keep it current through the service lifecycle. The service doesn’t work without it, but when it does, it works incredibly well.

Two design decisions that make a huge difference in CMDB design

1. Federate rather than hoard. A CMDB should give you a centralized view of asset data. That isn’t the same as storing all of it in one place. Pull data from the systems where it already lives and reflect it, rather than creating a second copy that drifts. Ours isn’t a single database at all. It’s an integrated set of tools and data sources presenting one view.

2. Discover and audit, don’t type. Manual entry, no automation rules, and discovery tools that get run once and forgotten are the three habits that produce an outdated and misleading CMDB. The best ones combine traditional bottom-up discovery, mapping infrastructure and extending out to customer-facing CIs, with event-driven discovery that maps related CIs when a system event triggers it.

There’s a third thing to mention here, too: A lot of CMDBs have no defined objective, no designated owner, and no mechanism for updating when things change. Name the owner. A database nobody is accountable for reverts to fiction on a predictable schedule!

If you'd rather have someone look at your environment directly, our NOC Operations Consulting practice assesses support requirements, analyzes gaps, and builds the workflows and operational visibility to close them. Contact us and we'll set up a discovery call.

About INOC, a service of Xerox IT Solutions

INOC is an ISO 27001:2022 certified 24×7 NOC and an award-winning global provider of NOC Lifecycle Solutions®, including NOC support, optimization, design, and build services for enterprises, communications service providers, and OEMs. INOC solutions significantly improve the support provided to partners’ and clients’ customers and end users.

INOC assesses internal NOC operations to improve efficiency and shorten response times, and provides best practices consulting to optimize, design, and build NOC operations, frameworks, and procedures. Proactive 24×7 NOC support is provided with several options, including North America, EU, or APAC only or global integrated NOCs. INOC’s 24×7 staff provides a hands-on approach to incident resolution for technology infrastructure support.

Learn more about our NOC support and NOC operations consulting services. Get in touch to start the conversation. We’d love to talk NOC.

Get in touch with us

Many conversations we have about improving incident management naturally turn into conversations about systems and tools (which ITSM platform, what correlation engine, how much automation).

Those are real questions and they matter, but they also tend to take time, money, and configuration work.

Meanwhile the same three gaps show up in team after team, and none of them require anything you don’t already own. They’re process items. A team that isn’t ready for a platform migration can still close all three inside a week, and the effect shows up in the numbers you already report.

Here they are, in the order an incident encounters them.

1. Establish priority levels with definitions

If you’re dealing with incident flow in any volume where everything can’t be handled immediately, you need a priority field. Many teams use some kind of system here, but far fewer have definitions behind it that make it work in practice.

Without them, priority gets assigned by “feel,” which means it varies by engineer, by shift, and by how the last few hours have gone.

The same event gets logged as a P2 on Tuesday and a P3 on Thursday (we use P1s for the highest priority incidents, with P2s, P3s, and P4s following that). Once that’s happening, everything downstream becomes unreliable. Your SLA reporting measures how people felt more than what occurred, and your escalation rules fire against a field nobody really trusts.

Behind those definitions are two questions worth making explicit for your team, because they’re what the definitions are really testing.

1. Impact: how many people or systems does this affect?

2. Urgency: what does it cost per minute that it continues?

An outage hitting one workstation and an outage hitting a trading floor circuit are not the same event, and a priority scheme that can’t tell them apart will send your engineers running at the same speed for both.

The other thing definitions fix is the “3 a.m. problem” that exists in organization where systems and infrastructure needs to be up 24x7. When a piece of personal equipment gets powered off at the end of the day and generates a critical alarm, somebody’s phone rings for nothing. Do that often enough and the cost stops being wasted attention and starts being people quitting. Backhaul going down deserves a phone call. (A laptop going to sleep doesn’t.)

2. Build the notification template, and specify what has to be known before it goes out

The complaint we hear more than any other from clients and from teams assessing their own performance, isn’t about resolution speed. It’s about not knowing what’s happening in full.

Teams tend to pour their improvement effort into detecting and characterizing incidents faster, and then leave the communication that follows entirely to whoever happens to be on shift. So, the first notification goes out as a sentence that says something is wrong somewhere, and the next forty minutes get spent answering questions that a better first message would have preempted.

A documented notification process has four pieces, and you can draft all of them pretty quickly:

The third one is the piece is the one we find missing the most in support operations, and it’s where most of the value is a lot of the time.

What is affected, what isn’t, when did it start, what’s being done right now, who’s doing it, and when will the next update come.

An engineer who has to answer those six questions before hitting send produces a message that’s worth receiving. Without them, the notification is a fire alarm with no address on it.

Calibration is worth a moment too, because more communication isn’t automatically better.

• Notify too often and the important message gets buried in the routine ones.

• Notify too rarely and stakeholders operate blind and start calling.

The answer varies by client and by priority. A P3 usually needs occasional updates. Some teams want every notification because they sort and route internally. And a simple time threshold cuts a lot of noise on its own: if the incident resolves inside a few minutes, don’t send anything. Nobody needs a notification about a port that bounced and came back.

One more thing that lands well and costs basically nothing: Tell people why you’re not doing something. If a carrier is repairing a fiber cut and can’t move faster, and you’re checking in hourly because hourly is the actual cadence of useful information, say that. Stakeholders escalate when they think nothing is happening. Explaining the decision usually stops the escalation.

3. Put a clock on escalations

The third gap is the one that produces some of the worst individual outcomes because it’s invisible while it’s happening.

Here’s what it usually looks like:

The fix is a time limit paired with judgment rather than replacing it. Set a maximum for how long a ticket can sit at a given tier before it has to escalate, varying by priority. When the clock runs out, it moves, and moving isn’t a failure or a mark against anyone. Then have leads watch for patterns, because someone hitting the limit repeatedly on the same category of incident is telling you exactly where your training gap is.

We find that two things make this work rather than just shifting the pile upward.

1. Define what an escalation has to contain. What’s been tried, what was ruled out, what the current hypothesis is, and what data has already been gathered. Most escalation friction is a content problem, not a routing problem, and a receiving engineer who has to reconstruct the last ninety minutes from scratch loses most of what the escalation was supposed to save.

2. Make sure the information the engineer needs arrives with the alarm in the first place. A technician who has to go hunting for the device location, the customer, and the service it supports before they can start is spending their clock on lookup rather than diagnosis.

We have a few guide that go quite a bit deeper into this, including the parts that do need tooling:

• A Complete Guide to NOC Incident Management

• 5 ITIL Incident Management Best Practices

• How to Elevate Your ITIL Incident Management Process

If you'd rather have someone look at your operation directly and tell you which fixes will pay off fastest in your environment, that's what our NOC Operations Consulting practice does. Every engagement opens with an assessment built on interviews, data collection, and technical analysis, and closes with recommendations we help you put in place. Contact us and we'll set up a discovery call.

About INOC, a service of Xerox IT Solutions

INOC is an ISO 27001:2022 certified 24×7 NOC and an award-winning global provider of NOC Lifecycle Solutions®, including NOC support, optimization, design, and build services for enterprises, communications service providers, and OEMs. INOC solutions significantly improve the support provided to partners’ and clients’ customers and end users.

INOC assesses internal NOC operations to improve efficiency and shorten response times, and provides best practices consulting to optimize, design, and build NOC operations, frameworks, and procedures. Proactive 24×7 NOC support is provided with several options, including North America, EU, or APAC only or global integrated NOCs. INOC’s 24×7 staff provides a hands-on approach to incident resolution for technology infrastructure support.

Learn more about our NOC support and NOC operations consulting services. Get in touch to start the conversation. We’d love to talk NOC.

Get in touch with us

Part 1 of our mini-series on common NOC problems covered the five issues we find on the path an alarm takes from firing to fixed: fragmented monitoring, missing correlation, scattered incident management, runbooks nobody trusts, and manual work where automation belongs.

Those are the “visible” kinds of fixes we advise on. They’re also the ones that come back if you don’t also fix the elements we’re covering today.

We’ve watched teams do good work on all five of those first problems and then lose most of it inside a year because the conditions that produced the problems were never addressed.

The five below are those conditions. They might be less “satisfying” to fix, but they’re central to a support operation working well, and they’re what separates one that improves from one that improves temporarily before sliding back into stress.

If anything here feels like a familiar operational problem you’re ready to solve, learn more about how we can help and get in touch.

6. A CMDB that doesn't match the network

The configuration management database is a key ingredient of any NOC that’s looking after many devices and services.

It brings order to what’s otherwise chaos: a central repository designed to consolidate and streamline information about IT assets and services, offering a unified and comprehensive view of the entire IT landscape.

At a global tech provider we worked wit, the configuration management database they had in place rarely matched the actual infrastructure as the environment evolved faster than the CMDB was getting updated. During incidents, engineers burned time cross-referencing systems and manually validating configurations, which routinely pushed resolution out by hours on exactly the incidents where that hurt most.

The fiber-optic provider we brought up in Part 1 had it worse. Staff regularly couldn’t find current documentation for their own network configurations, so engineers troubleshot in the dark with very little reliable data about the equipment that had just failed. One fiber hut failure that should have closed inside an hour ran multi-hour because nobody could locate updated configuration information for the affected gear.

In both cases the root cause was the same pair: no regular updates, and no automated tooling keeping the record in sync with reality.

Read our CMDB guide for a deeper dive on developing a NOC-focused system here.

7. Changes the NOC can’t see

This one produces some of the most frustrating outages we’ve assessed because the team is troubleshooting hard and looking in entirely the wrong place!

The banking platform’s operations center attended its Change Advisory Board meetings but never recorded the resulting changes in their ITSM system. That gap led to unauthorized changes and preventable outages, and we watched NOC staff spend time diagnosing problems that were obviously caused by work somebody had done that morning.

The tech services provider held CAB meetings too, but hadn’t connected their monitoring systems to their change management tooling. Changes went in without impact analysis. In one particularly expensive outage, the team spent hours on a diagnosis that an undocumented network change would have answered immediately.

The fiber provider we worked with was the riskiest of the three. Infrastructure changes were made ad hoc with no review at all, and we documented several outages caused by unauthorized reconfigurations that basic change control would have caught.

We did quite a bit of tool- and process-based solutioning for all three:

• For the banking platform, we automated ServiceNow workflows to capture and track every change, including emergency modifications made outside the normal CAB path.

• At the technology services provider we integrated MicroFocus OBM with their ITSM platform for automatic change tracking, which removed the manual documentation lag and made real impact analysis possible before changes went live.

• For the fiber provider we built the process from scratch inside ServiceNow: defined roles, approval chains, change windows, and maintenance schedules.

8. No NOC performance data (so no argument)

A shortage of real performance measurement shows up in every NOC we’ve assessed and in nearly every organization that comes to us about outsourcing part or all of their monitoring and service deck.

The fiber provider didn’t have formal KPIs for incident response times, staff utilization, or change effectiveness. So leadership had no visibility into how fast incidents were being resolved or how the team was actually spending its hours. It’s cliche, but you really can’t improve what you can’t measure.

The tech provider tracked basic availability, but the more granular numbers like first-level resolution and mean time to resolution and utilization, were collected by hand or not at all. During post-incident reviews we watched them struggle to identify anything to improve, because they had no detail to find patterns in.

We generally introduce automated reporting through something like Tableau or PowerBI integrated with the ITSM platform, which lets a NOC track MTTA, MTTR, FLR, and utilization without anyone assembling a spreadsheet.

At the global technology provider, one of the more useful findings fell straight out of that: FLR rates varied considerably across teams inside the same NOC. Team-level reporting told leadership exactly where to direct training and resources, which an aggregate number would have hidden completely.

9. Skill concentrated in people you can’t afford to lose

Staffing a NOC for continuous coverage, three shifts a day and 365 days a year, is extremely difficult to plan and expensive to sustain for everyone but the biggest operations (and it’s not easy then, either). If you under-invest in too small of a team, it will absorb too much until it burns out. Over-invest and you drain the budget without touching the operational problems that created the workload.

At the fiber provider, only four people covered 24/7, with no cross-training program. When their primary network troubleshooter wasn’t available, resolution times doubled or tripled. That’s not a staffing shortage, exactly. It’s a single point of failure that’s bound to cause problems.

The tech provider showed us the other end of it: 90% staff turnover in the previous year. Training consisted mostly of shadowing colleagues who were already overloaded, so new hires struggled with routine incidents, and the ones who did develop expertise left for better opportunities. Knowledge came in, matured, and walked out continuously. This is characteristic of any stressful support operation.

Read our full guide to staffing a 24/7/365 NOC. There’s a lot that goes into it.

10. A NOC cut off from the teams it depends on

The last one is structural. When the NOC operates separately from development, security, and second-level support, the cost surfaces exactly when cross-functional work matters most.

At the banking platform, the operations center worked in isolation from development and infrastructure. So during application outages, NOC staff tracked down developers through email chains and phone calls while the service stayed down.

The technology services provider had poor handoff between the NOC and second-level support. Escalations went up without adequate context, which produced long back-and-forth exchanges while the outage continued.

At a large digital services provider, the split was between the NOC and security. DDoS attempts and attempted breaches were detected by the security team, and the NOC often didn’t hear about them until the incident had already progressed.

The underlying cause is the same in all three: different teams, different tools, separate communication channels, and priorities that don’t line up. The NOC can’t function as the hub of incident management when it’s structurally a spoke.

A few questions worth answering

If you want a quick read on where your own operation sits, these are among the more revealing questions from our playbook’s self-assessment:

• Can you state the exact percentage of the alerts your NOC receives that are noise?

• Can your team immediately determine which customers are affected by any infrastructure incident?

• Do you have accurate numbers for your first-level resolution rate?

• Do you have documented evidence that your escalation procedures actually work 24x7?

• Can you accurately trace service dependencies across your infrastructure?

Several “no” or “not sure” answers is the normal result, including at organizations that believe they’re performing well!

Our NOC Operations Consulting practice is here to answer those questions with data rather than impressions. We open with an assessment built around interviews, data collection, and technical analysis of your environment, and end it with recommendations we help you implement rather than hand over.

Contact us to schedule a discovery call, and we’ll tell you what we’re seeing. If you missed part 1, read it here.

About INOC, a service of Xerox IT Solutions

INOC is an ISO 27001:2022 certified 24×7 NOC and an award-winning global provider of NOC Lifecycle Solutions®, including NOC support, optimization, design, and build services for enterprises, communications service providers, and OEMs. INOC solutions significantly improve the support provided to partners’ and clients’ customers and end users.

INOC assesses internal NOC operations to improve efficiency and shorten response times, and provides best practices consulting to optimize, design, and build NOC operations, frameworks, and procedures. Proactive 24×7 NOC support is provided with several options, including North America, EU, or APAC only or global integrated NOCs. INOC’s 24×7 staff provides a hands-on approach to incident resolution for technology infrastructure support.

Learn more about our NOC support and NOC operations consulting services. Get in touch to start the conversation. We’d love to talk NOC.

Whether we’re looking at an IT support operation formally through our NOC operations consulting practice or informally as part of a new service engagement, we spend a lot of time inside others’ operations.

If we’re consulting on improvements, that means interviews with the staff, a look at the tooling, whatever data they can pull for us, and then a written assessment of what we found along with a roadmap.

The report looks different every time, but the finding mostly aren’t. After enough engagements you can guess a good portion of what you’ll write before the first interview, and the part that surprises people is that none of it is shockingly novel. The problems that hold NOCs back are mostly what you’d expect, but they’re also compounding, which is why an operation can go from workable to underwater without anyone making a bad decision you can point back to.

Ten of them come up often enough that we wrote them into a white paper. In part 1 of 2 here, we covers the five that sit on the path an alarm takes from the moment it fires to the moment somebody fixes it. Part 2 will cover the five underneath, which are the ones that determine whether any of this stays fixed after we leave.

1. Monitoring split across tools that can’t see each other

Throughout each of these, we’ll pull in examples from actual operations consulting projects we’ve done.

Disconnected monitoring tools is a systems problem we see all the time.

At one fiber-optic infrastructure provider we worked with, monitoring lived in Ubiquiti UISP and Calix SMx, but notifications went out over plain email. There was no unified view of anything anywhere. We watched engineers work through alert floods by hand, and we documented preventable power failures at critical fiber huts where the signal was sitting in the queue behind a hundred things nobody needed to see.

A digital banking platform had a cleaner-looking version of the same problem. Their operations center split monitoring between SLAB for applications and PagerDuty for infrastructure. Two good tools, both configured competently, but analysts still had to track and acknowledge across both, and nothing correlated between them, so an application symptom and its infrastructure cause showed up as two unrelated alerts in two places.

The cost people notice is the extra clicks. The cost that actually hurts the business is that no correlation is possible across a boundary the tooling doesn’t cross.

2. Alerts that never get correlated

This is close enough to the first problem that it often gets bundled in with it, but it’s really worth splitting out on its own.

Consolidation gets your events into one place. Correlation is what makes them usable once they’re there.

At a global tech company running cloud services we went into, the NOC was handling more than 50,000 events a quarter with no automated correlation. Every single one required manual acknowledgment. During major incidents we watched engineers try to piece together alerts from different systems that were all pointing at a single infrastructure failure, and critical issues sometimes weren’t escalated until after service was already affected.

The fiber provider we mentioned in the previous section had the “priority version” of this problem. With everything arriving as email, a minor event and a genuine outage looked identical on arrival. Staff couldn’t triage what they couldn’t distinguish.

At the tech company, filtering the noise and grouping related alerts into single incidents gave engineers a queue they could actually work. At the fiber provider, we used LogicMonitor's correlation capabilities to group related events automatically, which gave their staff their first clear view of what needed attention right now.

3. Incident management scattered across three systems (or missing entirely!)

The banking platform ran incident management across Salesforce for case management, SLAB for application alerts, and PagerDuty for infrastructure notifications. During a critical incident, people lost minutes deciding which system to use. Some incidents opened a Salesforce case automatically, others needed manual logging in PagerDuty, and handoffs were a constant source of confusion about what had already been done.

What bothered us most in that assessment wasn’t the tooling. It was that they had no way to measure first-level resolution, so they couldn’t tell how many incidents their frontline staff were closing on their own. They knew the process was broken, but they had no way to locate the break.

The fiber provider was further back than that:

• No formal incident management system existed.

• Email chains and verbal handoffs, with delays and missed escalations visible throughout their incident history

• No ability to measure mean time to acknowledge or mean time to resolution because there was nothing to measure from.

4. Runbooks nobody trusts

Runbook problems are tricky and insidious because they often look like documentation problems and behave like culture problems.

At the banking platform, the runbooks hadn’t been updated in months despite meaningful infrastructure changes. Staff had stopped opening them, which is the rational response to documentation that no longer matches the network. We watched different engineers handle identical incidents in completely different ways. During one major incident, that inconsistency added roughly two hours because the engineer on duty didn’t know about a workaround a colleague had used successfully the week before. Different things in different peoples’ heads, and best practices trapped person-to-person.

The fiber provider had exactly one procedural runbook for handling outages and most of the team didn’t know it existed.

Three things were true in both places and we see this all the time with documentation. The runbooks weren’t being updated as the environment changed, they weren’t easy to reach in the middle of an incident, and nothing in the process required anyone to use them.

Here’s the general template for our runbooks/knowledgebase articles. Read our full guide to NOC runbacks for more depth.

5. Manual work sitting where automation belongs

At that tech services provider, event acknowledgment and ticket creation were entirely manual. That works until it doesn’t! During outages, when volume spikes and the work matters most, incidents piled up faster than the team could process them, and we found critical incidents sitting unattended for periods nobody would defend.

The fiber provider’s analysts were manually acknowledging and triaging across PagerDuty and SLAB, spending hours on redundant and low-priority alerts while real problems waited in the queue.

Both cases responded well to well-tuned automations. Automating acknowledgment, ticket creation, and initial triage at the technology company cut average alert resolution time by 70%. Moving the fiber provider onto LogicMonitor’s automated platform reduced initial response times by as much as 60%.

Be warned though — automating a broken process only gives you the broken process at higher speed. If your triage logic sends the wrong things to the wrong people, automation makes that happen faster and more consistently. Fix the flow first, then automate it.

What’s coming next

Everything above lives on the incident path. You can fix all five and still watch the operation slide backward within a year, because what holds them in place is underneath.

Part two will cover those five: a CMDB that doesn’t match the network, changes the NOC can’t see, the absence of performance data, skill concentrated in people you can’t afford to lose, and a NOC that’s cut off from the teams it depends on. Those are the ones that decide whether the work in part one survives.

If any of this sounds like your operation, know that all of these issues are extremely common. Our NOC operations consulting practice uncovers everything holding your operations back and then gives you the steps we suggest taking to unlock better performance. Every engagement opens with a focused assessment of your specific environment and ends with a detailed report and recommendations, with additional implementation support if you need it.

Contact us and we’ll set up a discovery call to learn more about where you’re at.

About INOC, a service of Xerox IT Solutions

INOC is an ISO 27001:2022 certified 24×7 NOC and an award-winning global provider of NOC Lifecycle Solutions®, including NOC support, optimization, design, and build services for enterprises, communications service providers, and OEMs. INOC solutions significantly improve the support provided to partners’ and clients’ customers and end users.

INOC assesses internal NOC operations to improve efficiency and shorten response times, and provides best practices consulting to optimize, design, and build NOC operations, frameworks, and procedures. Proactive 24×7 NOC support is provided with several options, including North America, EU, or APAC only or global integrated NOCs. INOC’s 24×7 staff provides a hands-on approach to incident resolution for technology infrastructure support.

Learn more about our NOC support and NOC operations consulting services. Get in touch to start the conversation. We’d love to talk NOC.

Get in touch with INOC

From Peter Prosen, VP of Managed Services Operations, INOC/Xerox IT Solutions

Almost every IT support team that comes to us to either improve how they work or have us take on their workload has some reporting. That often looks like ticket counts, some measurement of average response time, and an availability percentage somebody drops into a slide deck once in a while.

Then we ask what actually caused last month’s downtime or where the bulk of incidents actually comes from across the infrastructure and the room gets quiet.

That gap is the problem. A count tells you volume, but it doesn’t tell you cause, it doesn’t tell you timing, and it won’t tell you that the same router in the same closet has been failing for three months straight.

We published the full breakdown of our reporting program over on our blog — every standard, optional, and custom report we run, with screenshots of each one. This is the short version: the reports I would want if I were the one buying NOC service, and what each is good for.

One rule before any of it: If you can’t click a number and see the tickets behind it, that isn’t a report. It’s more of a claim. Drill-down is what separates reporting you can act on from reporting you have to take on faith.

Start with the trend line

The first screen most clients open shows tickets opened and closed over the past several months, the current open count, and a breakdown by type: monitoring, change, problem, and so on.

The useful number here is direction. Four hundred tickets in a month means nothing by itself. Four hundred this month against 250 in March means something, and it’s worth an hour of somebody’s time to find out what.

Same with the gap between opened and closed. If closed keeps trailing opened, a backlog is forming, and you want to catch that in the report rather than in a staffing argument six months later.

Why the ticket happened

Every closed incident in our system gets a resolution code.

Aggregate those over a few months and chronic problems stop being anecdotes. One carrier sitting behind a third of your P2 tickets over two quarters should be a vendor conversation with evidence attached (and vendors negotiate differently when you bring the data).

Watch the “no trouble found” bucket too. When it climbs, the network usually isn’t the issue — your monitoring thresholds are.

Our monthly ticket report gives a detailed breakdown by ticket type and open/closed status. This report includes full drill-down capability, so you can click any data point to see the actual tickets that make up that number.

This transparency is critical! You’re never wondering what’s behind a statistic, which is one of the prime complaints we hear about prior NOC service providers: the numbers look good, but service feels poor.

When tickets happen

This heat map report breaks incident creation down by priority, day of week, and hour of day. It surfaces patterns that a monthly total buries completely.

If your highest priority incidents (what we call P1s) spike on Tuesday mornings after a weekend maintenance window, look at your change process before you go looking at hardware. If P3s cluster at 9 a.m., that’s people arriving and finding things, which tells you something about what you're monitoring missed overnight.

Of all the reports we walk clients through, this is the one that generates the most follow-up questions. Staffing decisions and maintenance scheduling both come out of it.

The frequency analysis shows you not just when tickets are created, but how their priority levels distribute across time periods. Now you can understand your true incident profile and finally answer questions that (for most teams) were previously unanswerable.

For example, are most of your issues critical, or are they lower-priority items that can be batched and addressed during business hours?

Every one of these reports includes drill-down capability. Simply click a specific time period or priority level to see the individual tickets that contributed to that data point.

Compliance is pass/fail —  the average tells you the margin

We track Time to Notification, meaning how fast we told you, and Time to Action, meaning how fast an engineer actually started working the incident. Those are two different things, and providers who report only one are usually reporting the easier one.

Here’s the part people miss a lot: A compliance percentage is binary. If your P1 target is five minutes, 100% compliance reads exactly the same whether your average is four minutes fifty seconds or forty seconds. One of those has no room left in it. The next bad month puts you under.

So always ask for the compliance percentage and the average, by priority, month over month. Then ask about Time to Impact Assessment: how long before someone identified probable cause, what services were affected, and what the plan was. TTIA is the metric that tells you whether the NOC is thinking or just acknowledging.

The two reports nobody asks for (but everyone needs)

Time to Close covers the entire life of a ticket. Our troubleshooting, vendor escalation, whatever your team had to do, and administrative closure at the end.

Fast Time to Action next to a slow Time to Close is a specific and fixable finding. It usually means tickets are sitting open for paperwork reasons after the technical work is done. That’s a process problem, and process problems are cheaper to fix than engineering ones.

The exclusions report is the one I’d push hardest for. It lists every ticket pulled out of the SLA calculation and why: maintenance windows, client-requested testing, a regional carrier event, duplicates.

Legitimate exclusions exist and we use them. But if a provider can’t produce that list on request, or the list quietly gets longer each quarter, you’re reading a number that has been groomed.

Which devices are actually costing you

This pulls from the configuration management database, or CMDB: every monitored asset, where it sits, what it is, and how many incidents are tied to it. We create one for every client.

You don’t have 500 devices with a problem. You have about ten devices generating most of your tickets. Once that list exists on paper, replacement schedules stop being guesswork, and budget requests get much easier to defend.

Pay attention to the P3 through P5 alarms while you’re in there. Low-priority alerts get ignored by design, but a device throwing them every week is frequently a device on its way to becoming a P1 at two in the morning.

Call quality and abandonment

You can hit an answer-time SLA and still be failing callers. We track abandons past two minutes separately from everything else, because someone who waited and hung up experienced a service failure regardless of what the contract says. Those spikes point at staffing or at a volume surge nobody flagged.

Read our full reporting guide for many more critical dashboards and readouts.

Put these on a calendar!

Having the data changes nothing on its own. I’ve watched teams log into a portal, nod at a dashboard, and close the tab.

The clients who get real return out of this run a standing review, monthly or quarterly, with their own team and their engagement manager on our side. Same agenda every time. Where did SLO performance move, and which direction. What showed up in resolution categories that also showed up last quarter. Which assets are collecting incidents. What in the ticket-timing data should change how we schedule maintenance or where we sit staff.

An hour a month, run consistently, is worth more than a beautiful reporting suite nobody opens. The point of the review is to leave it with two or three things somebody owns and a date attached. Replace that switch. Escalate with the carrier. Move the threshold. Next month you find out whether it worked, which is the entire reason the data exists.

You should be getting this level of insight

If you take one thing from this: the reports worth having are the ones that can embarrass the provider. Exclusions, averages next to compliance, time to close. Any provider can build a dashboard that says the NOC is busy.

The full guide covers all of it, including the standard, optional, and custom tiers we run, screenshots of every report, and how our clients use them in monthly and quarterly reviews.

And if your current reporting can’t answer the questions above, get in touch. We’ll run a discovery workshop, map where your monitoring has gaps, flag what’s an immediate risk, and give you a right-sized NOC plan with pricing you can actually read.

About INOC, a service of Xerox IT Solutions

INOC is an ISO 27001:2022 certified 24×7 NOC and an award-winning global provider of NOC Lifecycle Solutions®, including NOC support, optimization, design, and build services for enterprises, communications service providers, and OEMs. INOC solutions significantly improve the support provided to partners’ and clients’ customers and end users.

INOC assesses internal NOC operations to improve efficiency and shorten response times, and provides best practices consulting to optimize, design, and build NOC operations, frameworks, and procedures. Proactive 24×7 NOC support is provided with several options, including North America, EU, or APAC only or global integrated NOCs. INOC’s 24×7 staff provides a hands-on approach to incident resolution for technology infrastructure support.

Learn more about our NOC support and NOC operations consulting services. Get in touch to start the conversation. We’d love to talk NOC.

We ran a survey last summer to get a read on where IT and network operations teams actually stand right now. Not where vendors say they stand, but where the people doing the work say they stand.

We collected 152 responses. Most were manager level or above, most worked at organizations under 500 people, and most came from technology, telecommunications, or education.

One caveat first, because it changes how you should read the rest. The people we asked were mostly already in our orbit: customers and contacts who think about operations for a living. It’s a picture of teams that already care about this work. Keep that in mind as you go.

Here are a few things that stood out. Read the full report below.

Read the full report

Most teams rate themselves “good enough,” and that’s the risk

Across all five capability areas we asked about, the most common answer was the same: a 3 out of 5. Good. Not poor, not excellent. Very few teams put themselves at either end. Almost everyone landed in the comfortable middle.

Rate your organization’s capabilities in the following areas (1 = Poor, 5 = Excellent).

That middle is a stable place to sit, right up until it isn’t. A team that rates itself “good” has little reason to change anything until something forces the issue: a breach, or a senior engineer walking out the door. The plateau holds until it breaks.

I see where it comes from. The “good enough” we run into is usually from teams running services for an internal group. If the experience isn’t great, that’s often fine. They just need to get by. On the service provider side, where the way you run the service lands on a paying client, teams care a lot more. When your users are down the hall, “good enough” survives. When your users are customers, it doesn’t.

Security is the problem today, AI is the problem teams see next

We asked teams to rank their current challenges. Security and compliance sit at the top, followed by rising operational costs and gaps in visibility.

What are your top current challenges in IT/Network operations? (Rank top 5)

Then we asked what they expect over the next 12 to 24 months. AI adoption climbs up to tie security threats for first. Cost containment slips from second to third.

What do you anticipate as your biggest operational challenges over the next 12–24 months? (Rank top 3)

That last part is worth sitting with. Cost is the second-biggest problem today and the third-biggest problem teams expect tomorrow. Either they think efficiency gains are coming, or they have made peace with paying more. The survey doesn’t say which, but my read is that it’s some of both.

Everyone is talking about AI, fewer are using it

AI showed up as the top expected challenge. It also showed up as a source of hesitation. 43% of teams said they are integrating AI in some form. Only 6% called that integration extensive.

Are you currently integrating AIOps/GenAI/Agentic AI into your IT operations (ITOps, ITSM, Network, NOC)?

Meanwhile 21% have no plans to adopt it and another 13% aren’t sure. So the thing teams named as their number one concern for the next two years is something a third of them either aren’t pursuing or haven’t decided on. For a lot of teams, AI is a worry more than a project.

The hesitation is not hard to understand. When we lean AI-heavy on the work we do, some teams get nervous. Everybody knows it is here. A lot of people are just wary that trusting it completely would cause bigger problems, and they don’t want to lose the human judgment and the human eyes on things.

There is a generational split in it, too. On some calls we join, the room is young engineers who want all of it, and they’re surprised we still use people for certain tasks. On others, the team has been in this industry a long time, and they treat AI as an assistant worth having, not a replacement for the person doing the work. Both views are right about something. That tension is what teams are working through.

Among the teams that are adopting, incident detection and classification leads by a wide margin. Capacity and resource management trails well behind, which suggests AI isn’t yet trusted for allocation decisions, or that the tools for it aren’t mature enough.

In which operational areas are you using or planning to use AIOps / GenAI / Agentic AI? (Select all that apply)

Three findings that surprised us

We ran the data to see which relationships held up once you control for the rest. Three of them pushed back on the conventional wisdom.

1. AI adopters trade cost worries for integration worries

You might assume the teams adopting AI are further along, with their tooling already sorted. The data says the opposite! Teams adopting AI were 21 points more likely to name tool and platform integration a top challenge. Adoption doesn’t clear the tooling problem; it adds to it. The more AI you bring in, the more systems you have to connect, and the more the seams show.

They were also 27 points less likely to name rising costs a top priority. The read there is a little simpler: teams that have already spent to adopt AI have made their peace with spending to get a result.

One more difference in the same direction. AI adopters were 25 points less likely to prioritize ITIL and formal best practices. It may be that they have already built those foundations, or that they now ask a tool to solve a problem instead of reaching for a framework.

Worth watching either way, because skipping the foundation in a rush to adopt AI is a real risk.

2. The teams that need the most help ask for the least

Teams that rated their own incident management lower were less likely to invest in fixing it, not more. For every point lower a team rated itself, it was 17.5 points less likely to invest in incident-management tools. Teams with higher overall maturity were 24 points more likely to invest.

Here is what I see behind that number: A lot of these companies have an internal pool of engineers who spot issues and fix them on the fly, so nothing is structured. There is hesitancy, a sense that dropping into a structured workflow would slow them down instead of just knocking the issue out.

That hesitation is the paradox in one sentence. The teams that would gain the most from structure are the ones most convinced structure would cost them. It usually flips after a major outage, when the thing they skipped becomes the thing that would have caught it.

3. Strong teams outsource more, not less, in the right places

The last surprise. Teams with stronger event and alarm correlation were 36 points more likely to outsource security operations. Internal strength didn’t reduce their appetite for outside help. It actually sharpened it!

A team that runs a tight operation can draw a clean line around the piece it wants someone else to own. But overall maturity cut the other way, making teams 11 points less likely to outsource security. The effect depends on which kind of strength you measure.

Where the money is going, and what teams let someone else run

When we asked where teams are investing, two answers led: monitoring tools and staff training. Teams are putting money into the systems and the people at the same time. Specific AI purchases, AIOps and the rest, ranked lower, even with AI topping the list of future concerns.

What IT/Network capabilities/tools are you actively investing in? (Select all that apply)

On outsourcing, 51% hand off some part of operations, though most call it minimal.

The most common thing to outsource is 24×7 monitoring and the NOC. AI and automation sit at the bottom of the outsourcing list at 6%, a strange place for the thing everyone says they’re most worried about.

Which areas of IT/Network operations are you currently outsourcing or considering outsourcing? (Select all that apply)

Part of that gap is timing. Part of it is that “outsourcing AI” no longer means what it used to. By that measure the 6% undercounts reality. Most teams using AI are already renting it from someone else. They just don’t file it under outsourcing.

About INOC, a service of Xerox IT Solutions

INOC is an ISO 27001:2022 certified 24×7 NOC and an award-winning global provider of NOC Lifecycle Solutions®, including NOC support, optimization, design, and build services for enterprises, communications service providers, and OEMs. INOC solutions significantly improve the support provided to partners’ and clients’ customers and end users.

INOC assesses internal NOC operations to improve efficiency and shorten response times, and provides best practices consulting to optimize, design, and build NOC operations, frameworks, and procedures. Proactive 24×7 NOC support is provided with several options, including North America, EU, or APAC only or global integrated NOCs. INOC’s 24×7 staff provides a hands-on approach to incident resolution for technology infrastructure support.

Learn more about our NOC support and NOC operations consulting services. Get in touch to start the conversation. We’d love to talk NOC.

Most NOCs we’ve walked into over the past 20 years share the same basic problem. It’s not the tooling or the talent. It’s the structure.

The typical NOC is a room full of engineers who all do roughly the same thing. An alarm fires, whoever is available picks it up, works it until they either solve it or get stuck, and then escalates it to someone more senior. The senior person drops what they were doing, takes over, and works it from there.

Rinse, repeat, all shift, every shift.

This works when you’re monitoring a few hundred devices. It stops working when you’re monitoring thousands, across multiple clients, with different SLAs, different severity tiers, and different escalation requirements. At that scale, the “everyone does everything” model means your most experienced engineers spend most of their time on routine issues that a less senior person could have handled. Your P1 response is inconsistent because whoever happens to be free gets the critical incident. And your Tier 1 engineers never develop because they escalate anything that doesn’t match a pattern they’ve seen before.

The fix isn’t better tools. It’s a better operating structure.

What a well-structured NOC actually looks like

At INOC and Xerox IT Solutions, we've spent years developing what we call the Structured NOC. It's the operational framework we use to deliver 24x7 support across enterprise, service provider, and OEM environments.

The diagram below shows how it's organized.

The structure has distinct operational layers, each with a defined scope of responsibility, and clear routing between them.

• Alarms enter from the top.

• Support requests enter from the bottom left.

• Everything flows through defined paths to the right people at the right tier.

Here’s how each layer works, and more importantly, what the client gets from each one.

The front door: alarm analysis and auto-correlation

Everything starts here. Alarms and events from client infrastructure flow into the platform through secure connectivity. Before a human touches anything, the AIOps engine processes the incoming data.

At this stage, three things happen:

This layer manages time-to-ticket SLOs. The time between an alarm firing and a ticket being created in ServiceNow, enriched with CMDB data and ready for an engineer, is measured in seconds. In our environment, the target is under 30 seconds from alarm to a fully contextualized ticket.

The outcome of this is engineers not being buried in noise. The correlation engine has already done the work of separating signal from noise, grouping related events, and attaching the context an engineer needs to act. The ticket volume the NOC team actually works reflects the real incident volume, not the raw alert volume. That distinction is the difference between a NOC that’s drowning and one that’s focused on the incidents that are of the highest priority.

The triage layer: Advanced Incident Management

This is where the Structured NOC model differs most from traditional operations, and where I'd argue the biggest value lives.

In a traditional NOC, incidents enter at the bottom of the org chart. Tier 1 picks up the ticket, tries to resolve it, and escalates when they can’t. The senior engineers sit at the top and wait for escalations. By the time a complex incident reaches someone with deep diagnostic experience, it may have been misdiagnosed once or twice, bounced between tiers, and accumulated hours of elapsed time.

We inverted that. Our Advanced Incident Management (AIM) team sits at the beginning of the workflow, not at the end. These are senior troubleshooting personnel whose job is to look at every incoming incident (particularly P1s and P2s) and do four things:

The AIM team also handles problem management and capacity management. They’re the people who look across incidents over time and identify patterns: recurring failures on the same circuit, capacity trends that predict future problems, chronic issues that need root cause analysis rather than repeated break-fix.

Our NOC clients get their incidents correctly diagnosed and routed from the start. The most common failure mode in NOC operations is misdiagnosis at initial triage, which causes the incident to bounce between teams, accumulate elapsed time, and frustrate everyone involved. The AIM model eliminates that by putting experienced analysts at the front.

Time to Impact Assessment (TTIA) drops because the person doing the assessment actually knows what they’re looking at.

We track TTIA as a formal metric here. It measures how quickly we can tell the client: here’s what’s happening, here’s what’s affected, and here’s what we’re doing about it. In a P1 situation, that communication is often more valuable to the client than the resolution itself, because it eliminates the uncertainty that causes organizational anxiety during an outage.

Service Desk and Tier 1: the workhorse

The bottom of the structure diagram is where most of the daily volume lives. The Service Desk and Tier 1 team handles the incoming support requests from client engineering and sysadmin teams (phone, email, chat), as well as the incidents that flow down from the alarm analysis layer after the AIM team has assessed and prioritized them.

Tier 1 work includes:

• Executing runbook procedures

• Searching known error and knowledge databases

• Providing call and email support to end users and client engineers

• Incident updates and follow-up

• Access control management

• Coordinating RMA and dispatch when hardware needs to be replaced or a technician needs to go on-site

The key operational metric at this tier is first-level resolution rate: what percentage of incidents get resolved without escalation. In our environment, that number is 85% or higher. Most teams we talk to who run their own NOCs see 40 to 60%, sometimes even less.

The difference comes from two things: the enriched ticket (which gives the Tier 1 engineer full context from the moment they open it) and the runbook library (which gives them a defined procedure instead of requiring them to improvise).

Our clients get fast, consistent resolution for the majority of incidents. When 85% of tickets resolve at Tier 1, the client’s experience is that most problems get fixed quickly by the first person who touches them. They’re not being bounced between teams or repeating the same information to three different engineers. The structure ensures that Tier 1 has the tools and context to resolve what they can, and a clean handoff path for what they can’t.

Tier 2 and Tier 3: the specialists

When Tier 1 can’t resolve an issue (the remaining ~15%), it escalates to Tier 2 or Tier 3. These are advanced technical specialists with deep domain expertise in specific technology areas like routing and switching, optical, wireless, cloud infrastructure, security, and voice.

The critical thing about this tier is utilization. In an unstructured NOC, your most skilled engineers spend 60 to 80% of their time on work that a Tier 1 engineer could handle. That’s expensive, and it means they’re unavailable when a genuinely complex problem comes in.

In the Structured NOC model, Tier 2/3 engineers spend their time on the incidents that actually require their expertise. Our framework typically reduces high-tier support activities by 60% or more, sometimes as much as 90%. We measure this by tracking how activities migrate between tiers over the first few months of a new client engagement.

When our clients have a complex problem that requires deep expertise, that expertise is available. The specialists aren’t buried in password resets and routine maintenance tickets. They’re actually working on the problems that need them.

The external connections

The right side of the diagram shows the operational integrations that extend beyond the core NOC team.

To run through them quickly:

It’s one point of contact for everything. When a circuit goes down and the carrier needs to dispatch a technician, the NOC handles that coordination.

When a switch fails under warranty and the OEM needs to ship a replacement, the NOC handles the RMA. The client’s internal team isn’t spending time on vendor management and ticket tracking across multiple providers. That work is absorbed into the NOC operation.

The migration effect

One of the things that validates this model is what happens during the first few months of a new client engagement. When we onboard a team, their incident distribution usually reflects the unstructured model they came from.

Too many incidents are escalating to high tiers. The Tier 1 resolution rate is low. Senior engineers are handling routine work. Time to resolution is inconsistent.

Over the first 90 to 180 days on the Structured NOC, activities steadily migrate to their appropriate tiers.

• Tier 1 starts resolving more because the runbooks are built, the CMDB is populated, and the enriched tickets give them what they need.

• AIM starts catching misdiagnoses earlier.

• The auto-resolution rate climbs as the platform learns the client’s environment and more patterns qualify for automated remediation.

By the six-month mark, the distribution looks fundamentally different. High-tier activities have dropped by 60 to 90%. Tier 1 resolution is at or above 85%. Auto-resolution is handling nearly half of all incidents without human intervention. And the metrics (TTN, TTA, TTIA, MTTR) are tracking at levels the client has never seen from an internal operation.

At more or less full maturity, the numbers usually look like:

• 85%+ Tier 1 resolution rate.

• 48% auto-closed without engineer intervention.

• 45% reduction in time-to-action year over year.

• Year-over-year cost reduction commitments of 5% in years 2 and 3 through AIOps maturation.

Why structure (in some ways) matters more than tools

We’ve been in this industry long enough to have watched multiple generations of NOC tooling come and go. SNMP managers, NMS platforms, ITSM systems, AIOps engines. The tools keep improving, but the NOCs that perform well have one thing in common: the operational structure is right.

A well-structured NOC with average tooling will outperform a poorly structured NOC with best-in-class tooling every time. The structure determines who handles what, how incidents flow, where decisions get made, and whether your most experienced people spend their time on the problems that need them or on the problems that just happen to land in their queue first.

The Structured NOC model we run is the result of two decades of building, measuring, and refining this structure across hundreds of client environments. It’s not the only way to organize a NOC. But the principles behind it, senior triage at the front, tiered resolution with clear escalation paths, automated correlation and enrichment feeding the human workflow, dedicated critical incident response, and continuous measurement driving improvement, are the principles that produce consistent, measurable results.

If you’re running a NOC internally and your Tier 1 resolution rate is below 60%, your senior engineers are spending more than half their time on routine work, or your P1 response quality depends on who’s on shift, the structure is the thing to fix. Not the dashboard and not the alerting rules. The structure.

If you want to see how this model would apply to your environment, get in touch. We’ll walk you through the operational framework and show you what the tier migration looks like for organizations similar to yours.

About INOC, a service of Xerox IT Solutions

INOC is an ISO 27001:2022 certified 24×7 NOC and an award-winning global provider of NOC Lifecycle Solutions®, including NOC support, optimization, design, and build services for enterprises, communications service providers, and OEMs. INOC solutions significantly improve the support provided to partners’ and clients’ customers and end users.

INOC assesses internal NOC operations to improve efficiency and shorten response times, and provides best practices consulting to optimize, design, and build NOC operations, frameworks, and procedures. Proactive 24×7 NOC support is provided with several options, including North America, EU, or APAC only or global integrated NOCs. INOC’s 24×7 staff provides a hands-on approach to incident resolution for technology infrastructure support.

Learn more about our NOC support and NOC operations consulting services. Get in touch to start the conversation. We’d love to talk NOC.

The term “managed workstation” has always undersold what’s actually happening around helping end-users. That term conjures an image of someone keeping laptops patched and answering the phone when Outlook stops working. And ten years ago, that was basically it.

Today though, the scope of what it means to support a workforce is different. Employees are in offices, at home, in hotel rooms, at client sites, and just about everywhere else. Their devices are laptops, tablets, phones and conference room hardware. The software stack is mostly cloud-based now (but not entirely).

Some companies still have servers sitting next to the water heater while others are fully on Azure. Most are somewhere in between, running a hybrid setup they didn’t entirely plan, with tools they’ve accumulated over time rather than chosen.

I’m responsible for service delivery across Xerox IT Solutions, including everything from our NOC operations through our managed services and support functions. My teams handle onboarding, tools configuration, and getting clients plugged into our platforms and services.

I want to quickly walk through what modern workforce support actually involves, because most of the descriptions I see out there, including our own in some cases, don’t go deep enough to be useful.

What the job of workforce support actually is now

We call it Modern Workforce Solutions, and that name is better because it captures the real scope:

That’s a big surface area, and most MSPs cover a slice of it. Some cover a few slices. What we’re building is a model where one partner handles all of it, from procurement to retirement, with one platform underneath.

If you’re running IT for a growing business, the practical question isn’t “who manages my laptops.” It’s “who manages the environment my people work in.” Those are different questions with different answers.

The tower model for endpoint security

At Xerox, we’ve restructured how our support teams are organized. We’re moving to a tower model where support is divided into three verticals: end-user compute (laptops, desktops, mobile devices), data center and server infrastructure (DCO tower), and networking.

Each tower has tiered support within it. When a user calls in with a laptop issue, the help/service desk handles the initial assessment. If the problem goes deeper than a 15-minute resolution, it routes to the end-user compute tower. If the problem turns out to be a broader system issue, it routes to the DCO or networking tower as an incident.

The reason this matters from the client’s perspective is routing accuracy.

• In a traditional help desk, every call goes to the same queue. The person answering doesn’t always have the context to distinguish between a user’s laptop problem and a system-wide application failure.

• In our new model, the system will classify and route, so the right specialist gets the right problem.

Pete Prosen, who runs our enterprise service desk operations, put it well: when a user calls in and says “I can’t use this application,” and it turns out nobody on the site can use it, that’s not a help desk ticket. That’s an incident. The tower model handles the reclassification without the user having to know the difference.

What “proactive” actually means for workforce support

The word “proactive” gets used a lot in managed services and really deserves to be defined and challenged. Here’s what it means in ours:

We just finished migrating to Datto RMM as our remote monitoring and management platform. What we’re doing with it draws on the same operational thinking behind our Xerox INOC Platform 3.0 on the NOC side: use the data the tooling generates to identify and resolve problems before the user has to call in.

That same approach applies to CPU utilization, memory, backup status, software update failures, certificate expirations. The RMM gives us the data points. The question we’re working through is which of those data points can trigger an automated remediation versus which ones need a person.

This is the same logic that drives our NOC’s high auto-resolution rate on infrastructure incidents. We’re applying it to the endpoint layer.

End-user onboarding never stops!

On the NOC side, onboarding is an actual event. A new client comes in, we integrate their environment, and the concept of onboarding fades into the background. On the managed services side, onboarding is continuous.

A small business with 50 users adds 10 new hires over the course of a year. Each one needs a laptop, configured to spec, with the RMM agent deployed, the EDR solution installed, Microsoft 365 provisioned, security baselines applied, and everything enrolled in monitoring. If that process is painful for the client (calling us, filling out forms, waiting days), they’ll leave. The friction alone isn’t worth the headache.

What we’re building is automatic detection of new machines entering the environment. A device shows up on the network, we detect it, deploy the RMM agent, install the EDR, apply the security baseline, and enroll it in monitoring. The client told us they have a new user. We take care of the rest behind the scenes. Nobody had to touch anything.

For clients who want more control, we’re working on a self-service model through ServiceNow where they can log into a portal and say “monitor this device,” “push this software,” or “take this out of monitoring.” It puts them in control without requiring them to call us or wait for a ticket to be processed.

That second model works well for companies that have some internal IT capability and want to co-manage the environment. The first model (full automation) works for companies where we are their entire IT department. Both are legitimate, and the service needs to accommodate both.

Workforce security

Human factors are still the biggest risk at the workstation level. You can deploy every tool in the world, and someone will still click the wrong link in an email.

Our approach to security now is layered rather than a single tool. Here’s what that looks like:

What makes this work as a managed service is that the client doesn’t have to source, configure, and maintain five separate security tools anymore. The whole stack comes as part of the service, deployed consistently across every device, updated automatically, and monitored centrally.

For compliance-heavy clients (healthcare, financial services, education), the monitoring and reporting data feeds directly into audit documentation. That alone saves significant time during compliance cycles.

Microsoft 365 and the cloud layer

This is a bigger part of modern workforce support than most people realize. Xerox is a Tier 1 Microsoft CSP (Cloud Solution Provider) partner, which means we handle Microsoft 365 licensing, Azure management, and tenant administration directly.

In practice, that means a client’s Microsoft licensing, mailbox management, Teams configuration, SharePoint administration, OneDrive policies, Intune (mobile device management), and Azure Active Directory all get managed through us. When something breaks in their M365 environment, they call the same number they call for a laptop issue. One team, one ticket, one platform.

The alternative, which is how most companies do it today, is managing Microsoft licensing through one partner, getting help desk support from a second, and running security through a third. Then spending internal IT hours integrating the three.

Where AI fits on the endpoint side

On the NOC side, there’s commonality in workflows, but each incident has individual character. A circuit goes down, and it might be the same carrier, but it’s not the same phone number or the same root cause every time.

On the help desk side, the tickets are more repetitive. If you read 1,500 help desk tickets and 1,000 of them are the same password reset problem, AI can identify that pattern and generate the knowledge article to prevent it from being a ticket next time.

The gap right now is ticket data quality. Closing a ticket without documentation into what was done makes it hard for any system (AI or human) to learn from the resolution. We’re addressing this in the new platform by automating time tracking (you click “work ticket,” work the ticket, click save, and the system calculates your time) and by building in categorization guidance (the system suggests the correct category based on the ticket content, so engineers aren’t filing password resets under “server” because they happened to log into the AD server to do the reset).

That data quality work is boring but foundational. AI can’t learn from bad data.

What “one provider” actually gets you

I’ve been in and around managed services long enough to know that most clients aren’t looking for magic. They’re looking for less coordination.

When you buy workstation support from one vendor, security from another, cloud management from a third, procurement from a fourth, and telecom from a fifth, you end up spending a meaningful chunk of your IT budget on integrating those providers. Not in a formal sense. In the sense that your internal people become the glue between five different companies that don’t talk to each other.

I’ve started using a different label internally for what Xerox is becoming. We’re really a technology services provider. Managed services is one part of that. But when you can procure hardware across every category (traditional, non-traditional, network, server, endpoint), manage Microsoft licensing as a Tier 1 CSP, provide help desk and NOC support through one platform, do provisioning and deployment through our integration centers, and offer optimization and vCIO advisory on top of all of it, you’ve moved past the MSP model.

A small business shouldn’t need to find five vendors and Frankenstein them together with Zapier and hope it works. They should be able to start with one partner and grow into whatever they need as their business grows. That’s the model. And I realize we’re not fully there yet for every service area. But the structure is in place, and the direction is clear.

What to look for if you’re evaluating this

If you’re shopping for modern workforce support, either because your MSP isn’t keeping up or because you’re building this function for the first time, a few things are worth asking:

The companies that build IT support on a platform rather than a collection of point tools will spend less time coordinating and more time working. That's the shift we're in the middle of. I'll keep writing about it as we get further along.

Learn more about our modern workforce solutions and get in touch to start the conversation.

About INOC, a service of Xerox IT Solutions

INOC is an ISO 27001:2022 certified 24×7 NOC and an award-winning global provider of NOC Lifecycle Solutions®, including NOC support, optimization, design, and build services for enterprises, communications service providers, and OEMs. INOC solutions significantly improve the support provided to partners’ and clients’ customers and end users.

INOC assesses internal NOC operations to improve efficiency and shorten response times, and provides best practices consulting to optimize, design, and build NOC operations, frameworks, and procedures. Proactive 24×7 NOC support is provided with several options, including North America, EU, or APAC only or global integrated NOCs. INOC’s 24×7 staff provides a hands-on approach to incident resolution for technology infrastructure support.

Learn more about our NOC support and NOC operations consulting services. Get in touch to start the conversation. We’d love to talk NOC.

Most service providers who monitor and manage network infrastructure say they’re “AI-enabled” one way or another now. In practice, though, that usually means they’ve bolted on a dashboard or reporting tool that uses the word “intelligence” somewhere in the interface. Maybe they’ve got a couple of correlation rules. Maybe they’ve integrated GenAI into their ticketing.

Our NOC operations service, INOC, has been applying machine learning and automation to operations infrastructure for several years now. We call the result the Xerox INOC Platform 3.0. I’ll walk through the platform and give you a tour of how AI and automation actually show up in our operations and what they do.

This is going to be detailed! If you run a NOC or you’re evaluating one, this is the piece where you can compare claims against actual mechanics.

The Xerox INOC platform in 30 seconds

Before getting into the AI specifics, it helps to understand what the platform is. The Xerox INOC Platform 3.0 is our operating system for NOC and service desk delivery.

It has three major flows: alarm, ticketing, and notification.

I’ll unpack this:

On the left side, client infrastructure and monitoring tools (LogicMonitor, SolarWinds, New Relic, Datto, Nagios, Dynatrace, whatever the client runs) feed alarm and event data into our AIOps engine.

That engine processes, correlates, and contextualizes that data. The output feeds into ServiceNow, our ITSM platform, where incidents are created, enriched from the CMDB, and worked by engineers. On the right side, integrations push data back into the client’s ITSM (ServiceNow, Jira, Freshworks, HaloITSM, Salesforce) and communications tools (Slack, Microsoft Teams, Twilio, email).

The whole process from alarm ingestion to an engineer holding a fully contextualized ticket can take under a minute. If you want the full deep-dive, stop here and read our platform explainer first.

This gives you an idea of the “shell.” Now here’s where AI, ML, and automation actually live inside it.

1. Event correlation

This is the core of our AIOps tooling, and it’s one of the platform’s biggest differentiators. A typical mid-sized environment can generate hundreds of events during peak hours. When a fiber line gets cut, 50 different devices might fire alarms in the same two-minute window. In a traditional NOC, that’s 50 tickets, 50 triage actions, and several engineers chasing what turns out to be one problem.

Our AIOps engine takes those 50 alarms, analyzes them against topology data and historical patterns, identifies that they’re all expressions of the same underlying incident, and generates one ticket. The engine is doing alarm correlation, pattern matching, and noise reduction in real time.

Rather than querying a static set of rules, we’re using extremely carefully trained AI to improve the correlation accuracy over time as we process more data from a given client’s environment. The more it sees, the better it gets at grouping related events and separating genuinely new issues from noise.

The operational impact is direct: engineers aren’t triaging 50 tickets. They’re triaging one! The cognitive load difference is enormous.

2. Automatic ticket creation and enrichment

Once the AIOps has correlated events into an incident, the system automatically creates a ticket in ServiceNow. This is where the CMDB does its work.

The incident ticket gets created with the following context already attached:

All of that is pulled from the CMDB and linked in a few seconds.

In a traditional NOC, an engineer would spend the first 10 to 15 minutes of an incident gathering this information manually. By the time they opened the ticket, they’d be starting from nothing. In our environment, the engineer opens a ticket that already has everything they need.

This is automation and data integration, not machine learning specifically. But it’s the foundation that makes the ML-driven pieces (like correlation and self-healing) useful. Without the enriched ticket, the automation upstream doesn’t translate into faster resolution downstream.

3. Automatic ticket creation and enrichment

Not all incidents are equal, and our platform knows this. Each incoming incident gets classified by business impact and severity using the data the system already has about the affected services.

Our priority system runs Priority 1 (P1) through Priority 4.

Here’s a visual breakdown of our structured NOC operations with tiered workflows:

The prioritization also drives our notification workflows. Client stakeholders get notified through IVR (PanTerra), email, Slack, Twilio, or whatever other platform they use based on the priority tier and the client's notification preferences. Those notifications fire without anyone manually composing them.

4. Self-healing and auto-resolution

This is where things get interesting. For a growing class of routine operational issues, the platform doesn’t wait for an engineer.

It executes the remediation itself!

Memory restarts, service restorations, certain types of configuration drift, predictable backup failures: these all have automated remediation paths. The platform detects the pattern, matches it against known resolution procedures, executes, validates, and closes.

We now typically auto-close 48% of incidents without an engineer touching them.

5. Recurring event handling

This is an automation function that sometimes gets overlooked but saves a lot of time. A feature called Recurring Event Handler separates events that have already been associated with an existing ticket from events that are genuinely new.

When a device flaps, or an issue recurs while a ticket is still open, the platform auto-associates the new event with the existing ticket, acknowledges the alarm, and logs the correlation (engineers don’t get a second ticket for the same problem). This directly reduces mean time to resolution because it stops the NOC from treating a recurring symptom as a new incident. This is one of the most common issues in NOCs and other support functions that drive people crazy while really dragging down service levels.

6. Automated runbook execution

Runbooks are step-by-step troubleshooting procedures.

Here’s the basic “anatomy” of our typical NOC runbook:

Traditionally, an engineer would open a runbook, follow the steps, and document what they found. We’ve automated the first diagnostic steps for many incident types.

When an incident is created, the platform can automatically locate affected infrastructure elements, retrieve relevant event data, run initial diagnostics, and update the ticket with the findings. By the time a human engineer opens the ticket, the first phase of investigation is already done.

This is rule-based automation, not ML. But the benefit compounds with the correlation and enrichment pieces. The ticket arrives with context, the diagnostics have already started, and the engineer can skip the research phase and go straight to resolution.

7. GenAI

We’ve been working generative AI into ServiceNow, which is our platform’s platform so-to-speak.

The most production-ready application we have in our NOC service today is really tight ticket summarization.

When a complex incident spans days, involves shift changes, and accumulates dozens or hundreds of updates, the ticket history becomes a document in itself. An engineer joining mid-incident has to read through all of it to understand what’s happened, what’s been tried, and where things stand. GenAI condenses that history into a concise summary: what happened, what was done, what’s still open. The engineer reads a few paragraphs instead of scrolling through 80 ticket updates.

This is a genuine time savings. Anyone who has worked a long-running P1 knows how much context gets lost at shift boundaries.

The broader GenAI roadmap is what we’re calling 4.0. It might include things like:

• Automatic knowledge article generation from resolved incidents (so the system learns from every ticket it closes).

• On-the-fly runbook creation when no runbook exists for a given pattern.

• Automated action plan generation.

• Agentic workflow execution where the AI doesn’t just suggest remediation but executes it.

Platform 4.0 is in development. The GenAI applications beyond ticket summarization are still maturing. We're testing, validating, and refining.

8. Change management automation

Maintenance windows are common hurdles in NOC operations. Every time a client schedules maintenance on a device or circuit, the NOC needs to know about it to avoid creating false incidents from expected alarms. Unsurprisingly, the NOC isn’t notified as much as it would like to be.

The platform handles this automatically. When a maintenance event is recorded, alarm suppression activates for the affected configuration items. The system only creates tickets if something fires after the maintenance window closes, or if an unexpected alarm occurs during the window. The system also ties maintenance records back to configuration items, so future incidents can be correlated with recent changes.

This is one of those capabilities that’s invisible when it works and painful when it doesn’t. Maintenance-related false alarms are a significant source of noise in any NOC. Automating the suppression removes that category of waste.

9. Problem management and predictive analytics

AIOps can be used for much more than just attacking incidents. It also drives problem management by spotting patterns across incidents over time.

When the same type of incident recurs on the same circuit, device, or vendor, the platform surfaces that trend. Our reporting provides visibility into incident volumes, resolution times, categorization patterns, and infrastructure hot spots.

Clients can see which carrier circuits experience the most outages, which hardware components fail most frequently, and which categories of incidents consume the most engineering time.

The predictive element is still developing alongside the tooling available to us. We can identify patterns that precede failures (CPU trends, capacity curves, degradation trajectories) and flag them before they cause an outage. The accuracy depends on having enough historical data from a client’s environment. For clients who have been on the platform for a year or more, the pattern library is substantial.

10. Help desk applications

The help desk side is where GenAI has the most untapped potential. The help desk is (arguably) more ripe for AI than the NOC was, because the tickets are more repetitive and the workflows are more standardized.

The actual logistics of help desks are different than the NOC in ways that complicate integration, but here’s the direction we’re heading:

We're also working on smarter routing, where AI classifies incoming requests and sends them directly to the right tier or specialist, skipping the generic intake process. Executives get faster escalation. Engineers skip Tier 1 and go directly to Tier 2. Apple-specific issues go to a dedicated Apple support team, for example.

What the numbers look like

A few operational metrics that are downstream of these capabilities give some hard insight into how this all translates into actual value:

• 85% or more of incidents resolved at Tier 1.

• 48% of incidents auto-closed without an engineer touching them.

• 45% reduction in time-to-action year over year.

• Year-over-year cost reduction contractually committed: 5% in year 2, 5% in year 3, driven by AIOps maturation.

These numbers came from the AIOps and automation investments described above. Each improvement connects to a specific capability: correlation reduces noise, enrichment reduces research time, auto-resolution reduces ticket volume, runbook automation reduces diagnostic time.

What “AI-enabled” actually means in the NOCs of today

If you’ve read this far, you have a concrete picture of where AI, ML, and automation live in a modern NOC platform.

Here’s how I’d summarize it:

AI-enabled doesn’t mean the NOC runs itself. It means the platform handles the parts of operations that don’t need a person: event correlation, ticket creation, data enrichment, routine diagnostics, known-pattern remediation, stakeholder notification, and report generation.

It means the parts that do need a person (complex troubleshooting, vendor coordination, architectural decisions, client communication during outages) get better because the person starts with full context instead of a blank screen.

The humans aren’t going away! They’re going to spend less time doing things machines are better at and more time doing things that actually require judgment and expertise. It’s operational, and it’s already running.

If you want to see what this looks like on real infrastructure, get in touch. We’ll show you the platform and walk you through the data.

About INOC, a service of Xerox IT Solutions

INOC is an ISO 27001:2022 certified 24×7 NOC and an award-winning global provider of NOC Lifecycle Solutions®, including NOC support, optimization, design, and build services for enterprises, communications service providers, and OEMs. INOC solutions significantly improve the support provided to partners’ and clients’ customers and end users.

INOC assesses internal NOC operations to improve efficiency and shorten response times, and provides best practices consulting to optimize, design, and build NOC operations, frameworks, and procedures. Proactive 24×7 NOC support is provided with several options, including North America, EU, or APAC only or global integrated NOCs. INOC’s 24×7 staff provides a hands-on approach to incident resolution for technology infrastructure support.

Learn more about our NOC support and NOC operations consulting services. Get in touch to start the conversation. We’d love to talk NOC.

From Austin Kelly, VP of Service Delivery at INOC & Xerox IT Solutions

I’ve spent years running operations and advanced technical services across enterprise network environments. Before that, I worked on the service provider side doing NOC operations, quality programs, performance management, and network project delivery.

The work has given me a pretty clear picture of what separates the networks that run well from the ones that fight their operators every day.

If you run or support enterprise networks, I want to give you eight things that I’ve seen make a measurable difference. Some of these are operational practices while some are technology decisions. A few are just organizational habits that save more time than anyone gives them credit for.

1. Design for the network you're going to have

Most enterprise networks were built to solve a problem that existed three or four years ago. The business has grown, the workforce went hybrid, the application stack moved to the cloud (partially), and the network design hasn’t caught up.

Too often, we encounter networks built to meet a set of immediate needs with little consideration for what comes next.

Then when the company needs to scale, add locations, support remote workers at volume, or integrate an acquisition, the network can’t absorb the change without a significant rearchitecture project.

A few things to keep in mind at the design stage:

If adding a new office location or onboarding 200 remote workers would require a major redesign, your network probably isn't ready for how the business is growing. Design with expansion and ease of upgrade built in from the start when you can.

2. Rethink what monitoring means now

“Network monitoring” has sort of outgrown its title. In 2026, a monitoring strategy that only covers network devices is incomplete. Cloud compute, applications, infrastructure, databases, and security all have to be part of the picture now.

The challenge we see teams struggle with is understanding exactly where their infrastructure lives and how to build a monitoring program around it. Until recently, the trend was to move compute to someone else’s data center. That made management easier but made monitoring harder.

What exactly do you need to monitor when half your infrastructure is in AWS, a quarter is SaaS, and the rest is on-prem?

Jim Martin, our VP of Technology, puts it well: monitoring is more than just the raw alerts off of devices, because that’s only giving you a piece of the puzzle. Understanding the business impact of a malfunctioning device or port is what actually lets you prioritize resolution. It’s about understanding why you should care and how much you should care, not just that something is red.

Before building out or revisiting a monitoring strategy, I urge teams to ask these questions:

3. Master your protocols

This is a more technical point, but it matters operationally, too. Monitoring is not just SNMP anymore, especially at the enterprise and service provider level.

Depending on what you’re monitoring, you may need to work with APIs for cloud services, gNMI (gRPC Network Management Interface) for modern network telemetry, WMI for Windows infrastructure, synthetic transactions to validate end-user experience on web applications, and database queries to confirm things like SQL transactions are completing correctly.

Each protocol gives you a different type of visibility.

• SNMP tells you device-level health.

• API polling tells you application-level status.

• Synthetic transactions tell you whether the actual user experience matches what the device metrics say it should be.

• gNMI gives you streaming telemetry at a granularity that SNMP can’t match.

If your team is monitoring a complex environment with SNMP alone, you’re seeing one layer of a multi-layer system.

4. Start correlating events to stop drowning in noise

This one makes the list because it’s the single most impactful operational change we’ve made in our own NOC, and I’ve seen it change outcomes for every client we’ve deployed it for.

Here’s the problem anyone in network operations knows well:

• A typical enterprise environment generates hundreds of events during peak hours.

• When one thing goes wrong, ten or twenty related devices might fire alerts in the same two-minute window. In a traditional monitoring setup, each of those alerts becomes its own ticket. Now you’ve got an engineer looking at 20 tickets when there’s actually one problem.

• Some platforms cut a separate ticket for each event instead of correlating events into one ticket, and it becomes a self-inflicted operational problem.

• You end up spending your time on a ticketing platform trying to combine and make sense of it instead of being able to clearly see the real incident flow.

Our AIOps engine solves this by analyzing events against topology data and historical patterns, grouping related events into a single incident, and generating one enriched ticket with context from the CMDB attached. The engineer opens one ticket, sees the full picture, and starts resolving.

The operational impact comes out in the numbers: 48% of incidents in our environment auto-close without an engineer touching them. 85% of the rest resolve at Tier 1. Those numbers are directly downstream of correlation. If you’re running a NOC or evaluating one, ask whether their platform correlates events or just forwards alerts. The difference in engineer productivity is dramatic.

5. Fix your default thresholds

This is so common that it deserves its own section, even though it’s technically a subset of monitoring tuning.

Out-of-the-box thresholds in monitoring tools almost never match the actual needs of the environment they’re monitoring. We regularly find environments where CPU and memory thresholds are set at 80% because that’s the default. The result is a torrent of alerts and tickets that consume engineering time without corresponding to actual problems.

Subscribe now

When you question those thresholds, you often realize they can be set to 90% or 95% for most devices without losing any real visibility into genuine issues. The ticket volume drops just about immediately.

We've seen environments where adjusting CPU alert thresholds from 80% to 95% cut ticket volume by more than half, with zero impact on actual incident detection. The 80% threshold was generating alerts for normal operating conditions. Nobody had questioned it because it was the default!

6. Turn monitoring data into actual business intelligence

Once you’ve got the right monitoring in place and the noise is under control, the data itself becomes valuable for more than just incident response.

I’ve always believed that the more data you have, the more apt you are to identify where issues are and where they’re occurring. That sounds obvious, but in practice most teams stop at “detect the alert, resolve the ticket.” They never circle back to ask what the data is telling them about longer-term patterns.

Two examples from our own operations:

The practical application is building reports that do more than confirm you met your SLAs last month. Good reporting should answer questions like:

• Which circuits experience the most outages?

• Which hardware categories fail most frequently?

• Are there time-based patterns to recurring incidents?

• What should we address through problem management versus change management?

7. Sharpen your focus on change management and communication

This one isn’t glamorous. Nobody writes conference talks about change control procedures. But it saves more time than almost any technology investment you can make!

The amount of time wasted chasing false alarms because a customer moved equipment without telling anyone is staggering. I’ve seen it happen where a client relocates a rack and doesn’t notify the NOC. That triggers a cascade of false alarms. An engineer spends an hour and a half investigating. A senior resource on the client side spends another hour and a half. All of it could have been prevented by a single email.

The financial cost of this kind of thing is hard to measure directly, but the opportunity cost is real. Every hour an engineer spends chasing a ghost is an hour they’re not spending on actual problems or improvement work.

The sectors that we see get this right the most are instructive.

• Federal agencies tend to have strong change control: changes get reviewed, notifications go to all relevant parties, and documentation is maintained throughout.

• Financial institutions, especially high-frequency trading firms and large banks, have similarly strict procedures. In both cases, everyone knows what’s happening and when, and changes are handled in a structured, predictable way.

Most enterprises can’t replicate federal-grade change control overnight. But a few basics go a long way:

8. Know when to build vs. partner

This is the decision that sits underneath all the others. Some organizations can build and run an effective NOC internally. Most can’t, or at least can’t do it at the maturity level their environment requires.

The honest calculus: maintaining 24x7 coverage requires a minimum of 10 to 12 staff. Finding people with NOC expertise is its own challenge because the skill set is specialized. Building the operational framework (runbooks, escalation procedures, SLA management, quality programs, reporting) is a multi-year project that requires experience most companies don’t have in-house. And the tooling investment, a proper AIOps-enabled monitoring platform with CMDB integration and automated workflows, is substantial.

When a company partners with a provider that already has all of this running, they get “instant operational maturity.” Years of process development, tooling refinement, and staffing optimization are available from day one. The internal team can focus on strategic work instead of break-fix.

The decision factors I’d consider:

If the answer to any of those is “not really,” the partnership model is probably the better path.

Getting started

If you’ve read this far and some of these points hit close to home, the first step is usually an honest assessment of where your monitoring and operations actually stand. Not where you think they are. Where the data says they are.

We do this regularly with clients through alarm analysis, operational assessments, and monitoring program reviews. The output is specific: here’s what your alarm data is telling you, here’s where the gaps are, here’s what to prioritize. No commitments required. Just a clear-eyed look at the current state and a practical path forward.

Get in touch if that’s useful.

About INOC, a service of Xerox IT Solutions

INOC is an ISO 27001:2022 certified 24×7 NOC and an award-winning global provider of NOC Lifecycle Solutions®, including NOC support, optimization, design, and build services for enterprises, communications service providers, and OEMs. INOC solutions significantly improve the support provided to partners’ and clients’ customers and end users.

INOC assesses internal NOC operations to improve efficiency and shorten response times, and provides best practices consulting to optimize, design, and build NOC operations, frameworks, and procedures. Proactive 24×7 NOC support is provided with several options, including North America, EU, or APAC only or global integrated NOCs. INOC’s 24×7 staff provides a hands-on approach to incident resolution for technology infrastructure support.

Learn more about our NOC support and NOC operations consulting services. Get in touch to start the conversation. We’d love to talk NOC.

If we’re not already working together, we’re INOC, a service of Xerox IT Solutions. We’re an ISO 27001:2022 certified 24×7 NOC providing end-to-end monitoring and management solutions including NOC support, optimization, design, and build services for enterprises, communications service providers, and OEMs.

We improve the support provided to partners’ and clients’ customers and end users.

Since we’ve become part of Xerox IT Solutions, we’re now part of an organization supporting more than 1.2 million endpoints and 250,000 infrastructure devices. We’ve resolved over a million help desk tickets. We have 600-plus engineers and network architects spread across operations, help desk and desktop support, managed workstation, managed server, managed network, managed security, data protection, field services, cloud hosting, and much more.

That work generates a lot of knowledge:

• How incidents move through a service desk.

• Where endpoint patching breaks down.

• What actually changes when you bring AIOps into a workflow.

• How backup and failover testing catches problems that scheduled patching creates.

In most teams, that knowledge usually stays internal. It lives in runbooks, in post-incident reviews, in quarterly business reviews, in the heads of the engineers who worked the problem.

The Operations Desk is where we start sharing those insights with you.

What is The Operations Desk?

This is a publication written by the people at INOC who build and run these services.

It’s not another marketing blog, nor is it a product announcement channel. We’re writing about the actual work of managing IT operations for organizations that can’t afford downtime, from the network layer to the end user’s workstation.

Here’s some of what we’re planning to publish here:

Who should subscribe?

If you work in or around IT operations, you’ll probably want to sign up.

• Maybe you lead an infrastructure team and you’re responsible for uptime, SLAs, and keeping costs under control.

• Maybe you run a help desk or service desk and you’re trying to scale without just adding headcount.

• Maybe you manage endpoints across a distributed workforce and patching alone is a full-time job.

• Maybe you’re at an MSP and you’re thinking about what operational maturity looks like at the next stage of growth.

• Maybe you’re a CIO or IT director at a mid-sized organization trying to figure out what to keep in-house and what to hand off.

You don’t need to be an INOC/Xerox IT Solutions customer to get something out of this. We’re writing for everyone in this space.

Why a newsletter?

The way people consume IT operations content has changed. Blog posts built for search engines reward generality when a lot of what we know (incident-level detail, cross-environment pattern recognition, opinionated takes from experienced engineers) doesn’t always fit that format well.

A subscriber publication does. We can write at a depth and specificity that wouldn’t make much sense elsewhere. So we’ll write what we know, put names and experience behind it, and keep it useful.

What’s coming?

Our first few posts will cover ground we already know well: topics our teams have been working on across network operations, managed services, and the help desk. After that, we’ll build out recurring formats: incident breakdowns on a regular cadence, field pattern reports, and conversations with our engineers and architects on specific service areas.

If there’s a topic you want us to cover, anything from endpoint management to data center architecture to how we run a 24x7 service desk, reply to any email. We read them.

Welcome to The Operations Desk.

About INOC, a service of Xerox IT Solutions

INOC is an ISO 27001:2022 certified 24×7 NOC and an award-winning global provider of NOC Lifecycle Solutions®, including NOC support, optimization, design, and build services for enterprises, communications service providers, and OEMs. INOC solutions significantly improve the support provided to partners’ and clients’ customers and end users.

INOC assesses internal NOC operations to improve efficiency and shorten response times, and provides best practices consulting to optimize, design, and build NOC operations, frameworks, and procedures. Proactive 24×7 NOC support is provided with several options, including North America, EU, or APAC only or global integrated NOCs. INOC’s 24×7 staff provides a hands-on approach to incident resolution for technology infrastructure support.

Learn more about our NOC support and NOC operations consulting services. Get in touch to start the conversation. We’d love to talk NOC.

Share The Operations Desk