Telemetry Tutorial: What to monitor and how to design your system
A solid introduction to telemetry essentials:
- What equipment you must monitor.
- How to design an alarm system to meet your current and future needs.
- How to limit transition costs.
How This Telemetry Tutorial Will Help You
If you're just starting with telemetry monitoring, you probably have lots of questions. What equipment do you really need for a useful alarm system? How do you balance the need for network visibility against the need to cut costs?
This White Paper will help you answer these questions for yourself. It covers how to look at your network and determine your specific monitoring needs and create a system that fits your requirements and your budget.
>Section I: Telemetry - Where Do You Start?
You've just been put in charge of finding, a new telemetry system for your company. Where do you start? What telemetry equipment do you need? What monitoring features are key, and which can you live without? How can you make sure your network is fully protected, without spending too much on equipment you won't use?
This White Paper is a quick guide to how you can answer these questions for yourself. This paper will NOT tell you, "Just buy this system and everything will be fine." Every network is different. A one-size-fits-all system won't provide the single coverage you need and may cost more money than you really need to spend.
Instead, this White Paper will show you the right questions to ask. Before you can decide what telemetry system to buy, you need to analyze your network and determine its single monitoring needs. Figuring out what you really need your telemetry system to do is your first step to designing a "perfect fit" system - one that's custom-designed for your network equipment, your available data transport - and your budget, too.
The 3 Step Plan for Creating a Perfect Fit Telemetry System.If you call DPS Telecom and ask what kind of telemetry system you need, the DPS sales engineer won't make a quick recommendation of "Buy this! Everyone has it. You should have it too!" Instead, your phone call starts a consultation in which your sales engineer will help you identify the network elements you need to monitor and the most effective way to monitor them.
This White Paper will take our through the same 3 steps as the DPS Telecom consultation process:
1. Survey where you are now: What telemetry do you currently use, if any? What equipment do you need to monitor? What data transport is available in your network?
2. Define your monitoring goals: what would your ideal telemetry system - the telemetry system that does everything you need and want - look like? Do you need 24/7 pager and email notification? Do you want to integrate several different telemetry systems onto one user interface?
3. Plan your telemetry system upgrade: How do you get from where you are now to where you want to be? Is upgrading at once feasible and within your budget, or should you phase your upgrade over several budget cycles? What telemetry capabilities do you need right now, and which can wait?
Learn Monitoring the Easy Way: Attend DPS Telecom Factory Training
- Bill Speck, 3 Rivers Telephone
Learn telemetry in-depth in a totally practical hands-on class. The DPS Telecom Factory Training Event will show you how to make your telemetry easier and more effective. You'll learn SNMP telemetry, ASCII telemetry processing, Derived Alarms and Controls, and how to configure automatic email and pager notifications. DPS training is the easiest way to learn telemetry, taught by technicians who have installed hundreds of successful telemetry deployments.
For dates and sign up information, call 1-800-693-3314 today or go to www.dpstele.com/training.
Start Here: Network and Remote Site Survey
Your first step to get your telemetry upgrade rolling is a complete survey of your current network and remote sites. This survey will document your existing telemetry situation, in order to build a road map for your upgrade.
In your site survey, you're looking for three things:
1. The equipment you need to monitor and the number of alarm points you'll need to monitor it.
2. The currently available data transport between your remote sites and your Network Operations Center (NOC) - the office where your telemetry presentation master is located.
3. Any existing alarm collection and presentation equipment you already have. You may be able to save money by incorporating your existing telemetry equipment into your new, upgraded telemetry system.
(DPS Telecom offers a five-page Remote Site Survey template that will help you organize your network and remote site survey. See box: "DPS Telecom Remote Site Survey.")
Now let's look at what kind of network equipment you should be monitoring.
What Do You Need to Monitor?
It takes a lot of equipment working together correctly to keep your network running, and you need accurate information about every object involved.
That means monitoring not only your base telecom equipment, but also all the equipment that supports it and the environmental conditions that all your equipment requires to operate correctly.
The things you need to monitor fall into four categories:
Don't settle for monitoring your revenue-generating equipment with simple summary telemetry that just tells you whether the equipment is up or down. Ideally, you want a broad series of alarms that identify problems down to the card level.2. Power supplies: commercial AC power, battery plants, rectifiers, backup generators, UPS systems, etc.
Monitor your power supplies as wholly as possible - power outages are the most common cause of remote site failures. Just as your power supply has multiple fail-safes and backup systems, each of those backups should be monitored.
At the basic level, you must monitor commercial power and battery level. Getting more advanced, it's also a good idea to monitor rectifiers and generators, including whether the generators perform their regular self-start tests. If you want the earliest possible warning of any problem that might stop your power supply, monitor every link in the power supply chain, right down to the fuel levels in generator diesel tanks.3. Building and facility telemetry: intrusion, entry, open door, fire, smoke, flooding, etc.
It's vital to monitor the safety of the buildings that house your key equipment. Since remote sites are usually unmanned and often in isolated locations, they're more at risk to vandals and intruders. Accidents like short circuits and small electrical fires, even if they're small, can become disasters if you don't have any way to detect them and react in time.
Your site monitoring should begin with at least monitoring open doors and fire alarms. For more security, you may want to consider adding an electronic building access control system and video surveillance to your alarm system.4. Environmental conditions: temperature, humidity, etc.
Most electronic equipment operates best within a defined range of temperature and humidity - monitoring these factors will give you early warning of potential problems.
You'll probably want to monitor different environmental conditions, depending on the location of the remote site. If the remote site is in a desert, humidity might not be a concern to you, but temperature probably will be. On the other hand, if your remote site has to function through an Iowa summer, humidity may be a major concern to you.
Another factor is the staying power of your equipment. If it's rated to operate under extreme ranges of temperature and humidity, you won't have to monitor environmental factors quite so closely, but you'll still want to make sure the site stays within the range specified for your equipment.
If your remote site is an environmentally controlled facility, you have a different set of factors to worry about. You need to monitor the continued operation of the heating, ventilation and air conditioning (HVAC) equipment that maintains the facility environment, plus you must be sure to monitor the power supply to the HVAC system. On top of that, you should still monitor temperature and humidity, as another safety check to make sure the HVAC is doing its job.
DPS Telecom Remote Site Survey
RTU Capacity and Function
- How many remote sites do you need to monitor?
- Do you want video surveillance at those sites?
- Do you want a building access control system to manage entry to those sites?
- How many alarm points do you need to monitor at each site?
- How much growth, in sites and telemetry at each site, do you anticipate over the next 5 years?
- Do you need any analog inputs (e.g., voltage, temperature, humidity, signal strength)?
- How many ASCII device (e.g., switches, routers, etc.) will you monitor at your remote sites?
- How do you currently connect to your remote sites? (LAN, overhead, digital or analog circuit, terminal server, microwave?)
- Do any of your sites support an alternate path communications link?
- What type of power do you have at the master and remote sites? (-48 VDC, 110 VAC, other?)
- How do you want to mount your RTUs? (23" rack, 19" rack, wall, tabletop?)
- Who will install your RTUs?
This is just a small sample of the DPS Telecom Remote Site Survey. The full Remote Site Survey is a complete 5-page guide to evaluating your telemetry needs. For your copy of the Remote Site Survey, call DPS Telecom at 1800-693-0351.
General Principles for Selecting What to Monitor
In the perfect alarm system of your dreams, you'll have an alarm for every single factor that can affect network actions, but you'll never spend extra money on alarm capacity you don't need. In the real world, time and budget constraints usually mean you have to set priorities and carefully select which telemetry you're going to monitor.
When you're choosing network elements to monitor, keep these three principles in mind:
1. Paranoia is your friend. Think about everything that can possibly go wrong, because - guaranteed - someday it will.
2. The more detailed your monitoring, the smaller your windshield time and repair costs. Precise diagnostics help you send the right tech with the right tools on the first site visit.
3. It's OK to start small and scale up. If you get an alarm system that can be upgraded, you can start monitoring your most critical network elements now, and gradually add more monitoring over several budget cycles.
4. Plan for your needs for the next five years. Your network and your monitoring needs will grow, and an alarm system that can't grow with them will be obsolete as soon as it's installed
What RTU Features Do You Need?
How do you find the right RTU? Here's 5 key features to look for:
- Discrete alarms: Monitor device failures, intrusion alarms, beacons, and flood and fire detectors.
- Analog alarm inputs: Monitor voltage, temperature, humidity and pressure.
- Ping alarms: Detect IP device failures and offlines.
- Control relays: Operate remote site equipment directly from your NOC.
- Terminal server functions: Control switches and other gear remotely via Telnet over LAN.
DPS Telecom offers RTUs that meet all these requirements - and offer local visibility via Web browser, email and pager notification, and more.
For more information about DPS RTUs, see us on the Web at www.dpstelecom.com/rtus..
Section II: How Do You Monitor It?
Now that you have an idea of what you should be monitoring, your next consideration is the nuts and bolts of how you are going to monitor it.
There are three phases to telemetry: acquisition, transport and presentation. Let's look at each phase in order.Acquisition: Getting Telemetry Out of Your Equipment
There are three kinds of alarm inputs: contact closures, analog inputs and protocol inputs.
Contact closures are also called discrete alarms or digital inputs. A contact closure is a simple on/off switch that produces an electrical impulse when it's activated or deactivated. Contact closures are the simplest kind of alarm input, so they're often used as a kind of lowest-common-denominator means of getting some kind of alarm from any kind of equipment.
Analog inputs accept current or voltage level inputs over a continuous range. They're the ideal kind of alarm for monitoring things like temperature and battery charge, where it's important to get an actual, physical measurement of the condition in real time.
Here's where having a quality alarm system really counts. Some alarm systems simulate analog telemetry with "threshold" alarms. For example, you might get a low-battery alarm if the battery voltage drops to -48 volts. But that information by itself is meaningless. After the voltage crosses the -48-volt threshold, does it stay there (indicating that the battery is merely low) or does it continue to drop (indicating that the battery is being rapidly drained)? With threshold alarms, you have no way to tell.
DPS Telecom telemetry equipment features analog alarms that report live, real-time analog values, giving you true visibility of these kinds of alarm conditions. Additionally, DPS analog alarms support four user-configurable thresholds (Major Under, Minor Under, Minor Over and Major Over), to provide best-quality notification of changing events.
Protocol inputs are electrical signals formatted into a formal code that can denote much more complex information than contact closures or analogs. There's a wide variety of protocols for transmitting telecom telemetry data. The most common telemetry protocols are open standards like SNMP, TL1, ASCII and TBOS, but there are also manufacturer-specific proprietary protocols. SNMP, TL1 and ASCII are simply ways of encoding ordinary written text for electronic transmission; these protocols are human-readable, if you know the code's terminology and operators.
Getting Telemetry Data from Telecom and Transport Equipment.
Unfortunately, there's no standard telemetry output for switches, routers, SONET equipment and other telecom and transport gear. You'll have to check each type of transport equipment in your network to see what kind of telemetry it supports.
The best way to find out what kind of telemetry your equipment can do is to check the documentation supplied by the manufacturer. The documentation should have at least a short section describing the equipment's telemetry outputs.
Ideally, your equipment will support some kind of protocol interface, giving you detailed visibility of its internal actions. But your equipment may only support contact closure outputs, which - depending on how many contact closures it has - may only give you a handful of summary alarms.
However, if your equipment doesn't have a documented protocol output, check it for a printer port, a report-only printer (ROP) port or a craft port. This port is designed to output a detailed log of equipment activity in the form of an ASCII text stream.
Historically, this ASCII output port was originally intended to connect to a printer for producing activity log printouts. A printout is a great way to keep a detailed record of what has happened in the past, but it's not a good way to monitor what's happening right now.
However, T/Mon provides a way to turn that ASCII stream into actionable, real-time telemetry data. T/Mon's optional ASCII Processor Software Module can automatically capture ASCII text, extract important information from the text stream, and convert the text to a standard T/Mon telemetry notification.
RTU (Remote Telemetry Unit) Choice: NetGuardian 832A.
The NetGuardian 832A is a full-featured remote telemetry unit. The NetGuardian supports 32 discrete alarms, 32 ping alarms, 8 analog inputs, 8 controls, and 8 serial reach-through ports. The optional NetGuardian Expansion Unit can expand the NetGuardian's discrete alarm capacity to 80, 128 or 176 discrete alarms. The NetGuardian reports to T/Mon NOC or to multiple SNMP managers - or you can use the NetGuardian's built-in Web Browser Interface and email alarm notification to monitor your remote site without a master.
For more information, check out the NetGuardian on the Web at www.dpstelecom.com/netguardian.
If It Prints, You Can Monitor It.
Why is ASCII telemetry processing so great? First, it's a simple way to get useful information from nearly any transport gear. If it prints, you can monitor it. Second, how many times have you been woken up by an alarm page at 3 A.M.? Wouldn't you like to know if you really have to go to the remote site - or if you can safely go back to bed?
ASCII alarms give you detailed reports on the condition of your equipment, isolating problems right down to the level of what shelf and what card need repairs.
Once an ASCII alarm is acquired, you can use it with any of T/Mon's advanced features: automatic pager and email notification, automatic alarm correction responses and more.
Getting Power, Facility and Environmental Telemetry.
Power, facility and environmental alarms are collected by groups of single sensors joined to site equipment like battery plants, generators, doors, temperature sensors and so on.
Outputs from these sensors are in turn joined to a remote telemetry unit (RTU) that converts contact closure and analog inputs into a protocol output, which is forwarded to your telemetry presentation master.
Every model of RTU (Remote Telemetry Unit) has a defined capacity of how many contact closure and analog inputs it supports. The alarm capacity of your RTU is the limiting factor for how much telemetry information you can acquire from your remote site. You don't want an RTU that has too little alarm capacity, because that will give you only vague and incomplete information about the state of the remote site. On the other hand, you don't want to pay for unneeded alarm capacity, either.
Your remote site survey will help you determine the correct alarm capacity for each type of remote site in your network. It's also a good idea to look for an RTU (Remote Telemetry Unit) whose alarm capacity can be easily upgraded. An RTU with expansion capability will grow with your remote site without your having to buy entirely new equipment.
Transport: Getting Your Telemetry from the Site to Your Screen
Once telemetry data is collected at your remote sites, it needs to be transmitted over a data network to your telemetry presentation master at your NOC. Telemetry data can be sent over nearly any kind of data transport: Ethernet LAN/WAN, dial-up modem, dedicated circuit, overhead channel, etc.
There are two things you should keep in mind about telemetry data transport:
1. As much as possible, you want to work with transports that are already available in your network. You don't want to create added expenses by committing yourself to installing new network infrastructure. It's best to choose a telemetry system that is compatible with the transports you already have.
2. It's a good idea to have a secondary backup path for your telemetry data in case your primary path fails. No transport is 100% reliable, and you don't want to lose telemetry visibility of your revenue-generating network under any circumstances.
Presentation: Displaying Your Telemetry in an Usable Format.
The final phase in telemetry is presenting the telemetry data in a useful way so that a human being can read the information and use it to direct repairs. This is done through a specialized computer called a telemetry presentation master. The master collects the telemetry reports from RTUs at the remote site and then format, sorts and displays the information for a human operator.
The master is really the most important part of the entire telemetry system. For the NOC technicians who monitor telemetry and dispatch repairs, the master IS the telemetry system - it's the only window they have to see what's going on in the network. The features and capabilities of your telemetry master directly control how much useful information your NOC techs can see. A high-quality, full-featured telemetry master gives you the tools to substantially lowers your network maintenance costs.
7 Critical Features for Telemetry Masters
Here's a list of 7 critical features that your telemetry master should have:
1. Protocol mediation and multiprotocol support: You probably have several different types of transport equipment to monitor, and you may have several generations of legacy telemetry equipment as well. All these different types of equipment report alarms using different incompatible protocols.
You definitely want to have one telemetry master that can support all the monitoring protocols your equipment uses and display all your telemetry on one screen. Trying to monitor by watching two or more screens is hard work that confuses even the best system staff, and sooner or later someone will miss a major alarm.
2. 24/7 unmanned monitoring via pager and email notification: Some companies can afford to pay staff to watch a monitoring screen 24 hours a day, seven days a week, including weekends and holidays. Your company probably isn't one of them. But you don't need a 24/7 staff if your telemetry master can automatically send alarm notifications to on-call technicians via pager and email.
3. Detailed alarm descriptions: Some telemetry masters display telemetry as cryptic numeric codes. You want a system that displays telemetry in plain English, with a complete description of what the problem is and what action you should take to correct it.
4. Alarm sorting and categorizing: If your telemetry system just shows you one long list of alarms from your entire network, it's easy to lose track of critical information. A quality telemetry system can sort and categorize your telemetry several different ways, by severity, remote site, equipment type or other criteria you define.
5. Separate Standing Alarm and Change of State (COS) Alarm lists: A Standing Alarm list displays all alarms that are currently uncorrected. A Change of State (COS) Alarm list displays all new events that happen in your network, including alarm points that go into an alarm state and alarm points that are cleared. If your telemetry master supports both kinds of view, you have the quickest and most accurate picture of your network's current status.
6. Nuisance alarm filtering: Your equipment might generate a lot of telemetry that consists merely of status reports that require no corrective action. These are nuisance alarms, and they're more dangerous than you might think. Nuisance alarms desensitize your monitoring staff to telemetry reports, and they start to believe that all telemetry is nonessential alarms. Eventually they stop responding even to critical telemetry. Look for a telemetry system with tools to filter out nuisance telemetry.
7. Expansion capability: A telemetry system is a long-term investment that will last for as long as 10 to 15 years. So you need an telemetry system that will support your future growth for up to 15 years. In that time your network is going to grow in size, you're going to add new kinds of equipment, and you're going to need new telemetry monitoring capabilities. Make sure your telemetry master can grow and change with your network.
Why You Need a Real Telemetry Master - NOT Switch Scan Points or an SNMP Manager.
It's tempting to try to build a home-grown telemetry master out of equipment you already have, like your switch equipment or an SNMP manager. But these won't give you the best visibility of your network. Here's why.Switch scan points have limited capacity and flexibility.
A telecom switch remote access node only supports five or six alarm points. You'll quickly outgrow those five or six alarm points. You'll quickly to tie multiple sensors to one point. At that point, an alarm can mean anything - maybe the building is on fire, or perhaps the battery is just low.SNMP managers don't support the functions you need.
Off-the-shelf SNMP managers don't support the critical telemetry presentation functions. Here are some of the features you can't find on a standard SNMP manager:
1. Detailed alarm descriptions, including date/time stamping, location and severity.
2. Immediate notification of change of state (COS) alarms.
3. Continuously updated list of current standing alarms.
4. Multi-user security.
5. Alarm sorting and nuisance alarm filtering
Section III: How to Plan Your Telemetry Upgrade.
n the previous sections, you've seen what equipment you should monitor and what features a good telemetry system should have. So you should have some sense of what would be the ideal telemetry system that will give you the best possible visibility of your network.
The question is, how do you get from where you are to where you want to be? It's very rare for a company to be able to suddenly leap from their current telemetry to their ideal system. Budget restrictions and the cost of installing equipment mean you can't usually get everything you want in one budget year.
Here are some strategies that will help you find a smooth, gradual upgrade path that will let you transition to a new telemetry system over several budget cycles:
- Define your immediate monitoring needs: What are the essential telemetry capabilities that you must have today? What critical equipment do you absolutely have to monitor right now?
Keep in mind, your definition of an immediate, key need might be different that someone else's. For example, if you have the staff to keep an eye on a telemetry screen 24/7, you might not need pager notification. But if you need to manage critical network assets during unmanned after-hours and weekend times, paging is an essential capability.
- Start slow, then expand: Once you've taken care of your bare minimum needs, you can add more alarm capacity and more monitoring capabilities over several budget cycles. You don't have to spend more than you can afford in one budget year, but you'll gradually move toward your ideal system.
- Use protocol mediation to incorporate existing equipment: The first stages of your upgrade can be easier and more cost-effective if you can install a new telemetry master first and then gradually replace RTUs at your remote sites. An telemetry master with multiprotocol support can support your existing remotes, so you can immediately add new presentation capabilities without replacing all your remote site equipment.
- Keep your future goals in mind: While you're planning your expansion, think about what your monitoring needs are likely to be 5, 10, 15 years down the road. It's easier and more cost-effective to add alarm capacity in a controlled way in the immediate future than to rush a new deployment through when you've exceeded your alarm capacity.
Because of its multiprotocol capability, T/Mon NOC is the perfect system to:
- Integrate diverse equipment to your SNMP or TL1 manager.
- Save your older equipment instead of replacing it - at huge cost savings to you.
- Manage large, complex networks from one T/Mon station, dramatically reducing staff and training costs
- Never miss an alarm - if there's a problem anywhere in your network, T/Mon will see it. And T/Mon's advanced notification features will make sure you know about it.
- More T/Mon advantages:
- Easy-to-maintain system, so any company can monitor in-house.
- T/Mon's ASCII Alarm Processor extracts detailed information from switches, routers, SONET gear,, email, Web and FTP servers - and just about any other network device
- Monitor 24/7/365 - even when no one's in the office. Companies around the world safely rely on T/Mon's pager and email notification for after-hours monitoring. It's a 24/7 NOC without the hassle or expense.
-Brian Krest, Senior Telecom Engineer
- Pinpoint the exact location and description of alarms.
Monitor proactively, not reactively. T/Mon tells you everything you need to know to fix problems on the very first site visit - which site, which device, alarm severity and a plain English description of the alarm. You'll eliminate unnecessary and overtime truck rolls, for a dramatic reduction in windshield time costs.
- Tell system staff exactly what to do when an alarm happens.
T/Mon's customizable text messages enable you to database detailed explanations and instructions for handling every alarm. Everyone on your staff, no matter what their skill or training, will know exactly what to do when an alarm happens.
- Control nuisance telemetry.
T/Mon gives you three ways to filter nuisance telemetry: alarm tagging (ignore alarms until user un-tags them), alarm silencing (temporarily ignore alarms for specified time) and alarm qualification times (ignore momentary and self-correcting alarms).
- Create custom telemetry from multiple alarm inputs.
T/Mon's Derived Alarms help you track complex events by combining alarm inputs and date/time statements. If you need to know when a site's generator and battery have both failed, or you want to know if a generator doesn't run its weekly self-test- or any other combo of events; Derived Alarms will tell you.
- Use these and all other T/Mon features on all telemetry.
All your alarms from all your devices - no matter what protocol - can access all of T/Mon's advanced features. Even your oldest devices can use pager and email alerts, Derived Alarms and Controls and nuisance alarm filtering. T/Mon NOC is a complete upgrade of your telemetry monitoring in just one unit.