4294

Get a Live Demo

You need to see DPS gear in action. Get a live demo with our engineers.

White Paper Series

Check out our White Paper Series!

A complete library of helpful advice and survival guides for every aspect of system monitoring and control.

DPS is here to help.

1-800-693-0351

Have a specific question? Ask our team of expert engineers and get a specific answer!

Learn the Easy Way

Sign up for the next DPS Factory Training!

DPS Factory Training

Whether you're new to our equipment or you've used it for years, DPS factory training is the best way to get more from your monitoring.

Reserve Your Seat Today

T/Mon in an IT to OT Network Cutover - Client Visit Report

By Andrew Erickson

July 10, 2023

Share: 

My day began early when I arrived at the client's HQ at 9:30 am. Our goal was a security-driven network cutover, shifting their T/Mon remote monitoring servers from the current IT network to a more secure OT network.

The benefits of such a move were clear: reducing potential attack vectors, isolating sensitive operational environments, and creating a robust barrier against cyber threats.

At 9:41 am, we set off for the cutover site, the destination of our critical operation. Upon our arrival at 10:00 am, I could feel a buzz of anticipation in the air. I always prepare for cutovers like this one, but things can always go wrong.

The cutover begins...

At 10:20 am, we signaled the start of the cutover operation by notifying affected teams within the organization. I found myself on a conference call with various client stakeholders.

I immediately began verifying the new on-site workstation's access to the Primary and Secondary T/Mons, noting both were showing recent Digitize alarm activity. (Digitize is a DPS Telecom company as of 2022, so now I represent both T/Mon and the Digitize Prism LX system).

By 10:30 am, IT reinforcements had arrived to facilitate network cabling changes. Everything was falling into place.

As the NetGuardian IP address was changed to the new OT address, instructions went out to HQ users to monitor alarms using the Primary T/Mon. I noticed an issue: a firewall effect that prevented web interface access from the new workstation. The staff on the conference call reassured me this was expected due to firewall rules.

The secondary T/Mon needed a push to take over active polling

I continued working through the morning, focusing on the Secondary T/Mon. Its job was to poll the local NetGuardian at its new OT IP address. However, it remained in its usual passive role, not taking over polling activity from the Primary T/Mon. With a bit of advice from Rich Howell from DPS Telecom's T/Mon Engineering team, I worked to force the polling takeover by the Secondary T/Mon.

Our work paid off by noon - the Secondary T/Mon successfully separated from the Primary T/Mon. This resulted in both T/Mons polling all RTUs, causing alarm from the fleet of NetGuardians to be split between them.

We needed a bit more time. My quick consultation with a client rep led to approval for a brief downtime for the Primary T/Mon to adjust its IP settings.

But then, there was another wrinkle: A preliminary configuration change led to an IP conflict. However, we quickly confirmed and resolved the issue by 1:10 pm.

This was progress. The Primary T/Mon was now at its new OT IP address and back online.

We reverted the process to avoid being offline during the day's busy period

As the clock ticked towards the 2 pm deadline, the client's leadership decided it was time to start the reversion process.

Under the pressure of the deadline, we managed to revert the Primary T/Mon, Secondary T/Mon, and the local NetGuardian to their original IP addresses and settings.

As the 2 pm deadline struck, we completed our short-term mission to avoid getting stuck in an in-between state during the daily busy period. The long-term goal of completing the cutover from IT to OT network would wait for another day.

Despite the day's challenges, it illustrates the intricacies of IT and OT network separation. When you're working on a network transition like this one, you can't be too careful.

Our next step was to plan for a smoother next attempt at this cutover.

We performed an "Initial T/Mon Failover, Synchronization, & Cutover Verification" at DPS HQ

To ensure there would be no problems with the next cutover attempt, we wanted to go "back to basics" and verify that nothing about this client's database would hold up the failover from Primary T/Mon to Secondary T/Mon.

Prior to the "Preliminary T/Mon Failover & Synchronization Verification" in the client's network, DPS Engineering would test the following at DPS HQ in California:

  1. T/Mon failover and restoration (Primary to Secondary and back to Primary) using client's database
  2. T/Mon cutover (change IP space according to cutover procedure)

Test results: Successful T/Mon Failover and Restoration

  • 2:40PM - Primary Webmin command to enable Maintenance in Login screen
  • 2:43PM - Primary VM ESC and R into Edit Master Menu
  • 2:43PM - Secondary countdown begins
  • 2:44PM - Verified Secondary (Active), wait for change-of-state
  • 2:47PM - Verified new change-of-state on secondary via F3 on All Alarms window 2:47PM - Verified Performance/Stats via F6 show Queue: 5 and incrementing
  • 2:48PM - Primary brought into Monitor Mode
  • 2:49PM - Verified Primary syncs new change-of-state events from secondary
  • 2:50PM - Verified Primary (Active)
  • 2:50PM - Verified Secondary (Passive) and Queue: 0
  • 2:51PM - Primary Webmin command to disable Maintenance in Login screen

This initial testing cannot be a complete simulation of the client's IT/OT networks, but it demonstrates the basic functions of T/Mon and the validity of the cutover procedure.

We found a small improvement to the cutover procedure

In the IP Cutover steps (1-26), you need to change step 13 to manually change IP address in the Primary Configuration since you won't be able to "Push these new database changes from Secondary" now that the IP of the primary is different.

I'll continue writing about this cutover

This "IT network to OT network" cutover is only partially completed. More remains to be done.

I'll continue writing anonymized details like the above in future articles. This way, you can learn more about cutovers and T/Mon in a real-world context.

Call DPS to learn more about IT-to-OT network cutovers and the T/Mon alarm management platform

If you're considering or actively planning a network cutover from IT to OT, call us at DPS Telecom. With our expertise in network monitoring (NetGuardian RTUs and the T/Mon alarm management platform), we can help you achieve successful cutovers without the usual headaches and delays.

We not a consultant for general cutovers, but we're happy to speak with you for a few minutes to share some "tips and tricks" to help you succeed.

Give us a call at 1-800-693-0351 or email info@digitize-inc.com

Share: 
Andrew Erickson

Andrew Erickson

Andrew Erickson is an Application Engineer at DPS Telecom, a manufacturer of semi-custom remote alarm monitoring systems based in Fresno, California. Andrew brings more than 17 years of experience building site monitoring solutions, developing intuitive user interfaces and documentation, and opt...