3795

Get a Live Demo

You need to see DPS gear in action. Get a live demo with our engineers.

White Paper Series

Check out our White Paper Series!

A complete library of helpful advice and survival guides for every aspect of system monitoring and control.

DPS is here to help.

1-800-693-0351

Have a specific question? Ask our team of expert engineers and get a specific answer!

Learn the Easy Way

Sign up for the next DPS Factory Training!

DPS Factory Training

Whether you're new to our equipment or you've used it for years, DPS factory training is the best way to get more from your monitoring.

Reserve Your Seat Today

Developing the ADP: Modern Analog Comparator & Distribution Panel

By Andrew Erickson

April 21, 2020

Share: 

A few years ago, we were approached by a major subway system operator to replace their legacy audio selection/distribution panels. These are conceptually very similar to analog comparators (also known as "voter comparators"), except the channels are selected by priority instead of receiver voting. The decision isn't simply based on a voting system choosing the receiver with the best signal quality and outputting the voted audio.

The old distribution panel wasn't getting the job done

There were many issues with the old audio panels that the subway system was looking to solve:

  • The power supply was single, nonremovable, had screw-down terminals, and ran at a ridiculous temperature of 155°F at idle.
  • Although the input voltage was -48 VDC, there was no support for a wide voltage range that also included -24 VDC.
  • Channel priority was limited to either "Low" or "High".
  • The Channel Bridging feature was limited to only Channels 7 through 10.
  • Input adjustment potentiometers had to be adjusted manually with a screwdriver by a technician. That tech had to travel across the city and descend deep into a subway station.
  • There was no web or software interface of any kind to either monitor or control the device remotely.
  • There were no output adjustment pots.
  • There were no adjustable detect levels.
  • The detection of 2175 Hz guard tones was permanent. It could not be disabled for the occasional device type that should trigger purely based on audio dB level.
  • Inputs could not be individually deactivated in software. The only way to stop a channel from future activation was to physically remove the wire.
  • There was no temperature and humidity sensing. This isn't a typical feature of an analog comparator. It just makes sense to remotely monitor these important factors in any remote site that's going to require one.
  • The noise level, total harmonic distortion (THD), inter-modulation distortion (IMD), and electromagnetic interference (EMI) were higher than desired.

Despite all these limitations, the system essentially worked at some level. If you've worked with telecom infrastructure for any period of time, you know that it's difficult to justify replacing something that is functioning. Sometimes, it takes a specific incident to give the organization a good push.

A police officer accidentally jams the radio network for hours

For this particular transit authority, that push came in 2017. A police officer started his duty shift as he always did: by inserting his handheld PTT radio into its holster.

Unfortunately, on this idle weekday, he inserted the radio at the wrong angle. This pinned down the push-to-talk button, causing a constant and transmission with an unknown source. The officer didn't know this was happening, and nobody else could easily locate the signal source.

Since the existing audio selection/distribution panels had only rigid, predefined built-in logic, those devices handled this in the only way they could. Police, as you might imagine, are given the highest possible priority (no "receiver voting" logic is used) within the radio system. They're capable of overriding transmissions from even the central rail controllers that manage the flow of subway trains throughout the network.

At this point, the entire subway system was suppressed by what was effectively an unintentional radio jamming attack. This persisted for hours and hours until the constantly transmitting police officer's radio drained its battery. In the meantime, the entire train network had to be managed with makeshift communications, including consumer-grade cell phones.

The subway system approached DPS for a solution

At DPS, I'd been supplying this particular subway system with remote monitoring devices for two decades. They're among the most prolific users of NetGuardian RTU's and T/Mon master stations in the world.

With that built-in familiarity, the senior management at this transit agency asked me a question. Could I offer a solution that would prevent any future radio jamming incidents?

The goal was to build a new box that would replicate the functionality of the old audio selection panel. It would then take things up several notches. It would have tremendously more built-in intelligence and remote management capability.

Two DPS engineers working on initial development of the NetGuardian ADP.
When we were contacted by our long-time client with a new challenge, we started developing a solution right away.

First and foremost, it would prevent any future "stuck carrier" problem with a simple algorithm.

The new box would prevent a stuck carrier from shutting down radio communications

The inciting incident for this new product development was a handheld radio that jammed systemwide communication. This was the first thing we needed to solve.

Working together with subway representatives, we all quickly rallied around the idea that some kind of timeout would be required. After a user-definable period had elapsed with a channel constantly broadcasting, that channel will be locked out.

What took longer to hash out was how a channel should be restored after being locked out. Eventually, we agreed to provide two options that can be selected in the web interface.

Option 1: Manual Service Restoration

The first option was for a manual restoration to service. If the same incident were to happen again, the police radio channel would be suppressed after 60 seconds of continuous carrier signal. The central administrative team for subway telecommunications would then need to log into the web interface. The team could manually restore service to that radio channel after the incident had ended.

That first option is obviously the most protective of radio availability, but it comes at a cost. It could potentially suppress police communications that are, in fact, important. That's why we would build a second option.

Option 2: Automatic Service Restoration

The second option would be for a communications channel to automatically reactivate after a period of silence. If that channel was blocked after 60 seconds of continuous transmission, it could be unblocked after another period of silenc. For example, five continuous seconds without a 2175 Hz guard tone.

We'd take the opportunity to improve many of the other gripes that the subway had about the existing box

Once I'm approached to fix a legacy box, I always try to find opportunities to add more value. In this case, the many undesirable functions that I listed for you earlier gave me tons of inspiration.

We would tackle overheating and unreliable power supplies, inadequate prioritization granularity, screwdriver-based potentiometer adjustments, and more.

We would avoid digital converters, instead using modern technology to manage the analog audio needed by the subway's various systems. This kept audio quality lossless.

We would also, during our 3 field trials, discover plenty of other opportunities that we could never have anticipated without visiting in person.

Testing in real-world conditions: The first field trial in September 2018

Having built an initial prototype of a new "NetGuardian ADP", two other DPS engineers and I hopped on a plane. We traveled from our headquarters in California to visit a team of subway engineers.

Front panel of the NetGuardian Audio Distribution Panel (ADP)
The front panel of our first-revision NetGuardian ADP hardware included status LEDs and an LCD. These reflect the input state and output state of the box at all times.

Back panel of the NetGuardian Audio Distribution Panel (ADP)
The back panel of our new NetGuardian ADP was designed specifically to mimic the precise placement of ports on the old box. This made replacement a breeze for the install teams. Notice the screw-down connectors for power and the cylindrical fuse, which were changed in the next revision to match our client's preference.

Within their simulation lab, we started a week of initial testing. As is usually the case, our initial enthusiasm was quickly dashed in the harsh light of reality. Obviously, this kind of experiential learning is exactly why you do a field trial, but it's always fairly disheartening initially.

Right out of the gate, we realized that we hadn't adequately noted the desire for dual hot-swappable power supplies. In our minds, we know that NetGuardian power supplies are incredibly reliable and proven themselves around the world.

Nonetheless, we knew that this particular client had been traumatized by the old box and is unreliable power supply. A dual hot-swappable design would provide redundancy and also the ability to quickly swap in a spare when necessary.

The field trial "roller coaster" begins - and drives massive learning

And so, we had launched our first typical "field trial roller coaster" of a week. Inevitably, the first half of any field trial is filled with various disappointments. We learn the various ways that our best-guess design just doesn't cut it. This always stings a little, but that's exactly why you do a field trial in the first place.

Throughout the rest of the week, we continued to test our firmware in the subway's simulation lab. We discovered several unplanned realities about the subway's network, including much smaller signal-to-noise ratios on some of the longer lines.

That led us to take important notes about the next revision of the hardware. This would include the ability to tune comparator outputs at levels that were appropriate for the amount of wire being driven.

With five full days of testing under our belts, we returned to California. We had enough pages of design notes to keep us busy for at least a few weeks.

Returning with big improvements: The second field trial in November 2018

After two months and another circuitboard revision, we flew back to the client site for another field trial. Not only had the hardware improved, but we had made the single greatest leap forward in our firmware design that the project would see.

The new hardware design had dual, hot-swappable power input cards. We also changed the power connectors themselves to be pluggable instead of screw-down. This further reduces the labor required to change out a spare if it becomes necessary.

Front panel of the (revision 2) NetGuardian Audio Distribution Panel (ADP)
Our second hardware revision included dual, hot-swappable power supply cards. Notice also that the LEDs evolved into very clear RGB lights to support more than just red and green.

Front panel of the (revision 2) NetGuardian Audio Distribution Panel (ADP)
The key elements of the back panel remained the same in revision 2, but there were differences. The power connectors and fuses were changed to match the design our client already liked from the NetGuardian 832A RTU.

We had switched to full RGB color LEDs throughout the design. This was a very important improvement to give us more than two colors to reflect system status. Traditionally, a bicolor red-green LED is sufficient for showing transmit and receive. With this NetGuardian ADP, we needed at least one additional color (we settled on blue) to represent when a channel is blocked after excessive transmit duration.

The new firmware delivered major improvements, too

The firmware now had some truly cool features that proved their worth in this field trial.

We now had support for adjustable detect levels. Users would have complete freedom to select the decibel level. The could also choose whether a signal needed to include trigger conditions. Options here include a +0 dB high-level guard tone (HLGT) and/or a -26 dB low-level guard tone (LLGT).

We built temperature monitoring into the unit. That's not directly related to this device's audio function, but it's an easy thing to add, and it's a very handy level to know.

After we encountered some difficulty during the first field trial related to audio troubleshooting, it made sense for us to add test-tone generation into our product. This was immediately useful. We were able to send a test tone from one NetGuardian ADP. We'd then see if that signal would properly trigger another on the far end.

We even added a web-interface oscilloscope

Finally, for even deeper analysis of the audio levels, we built in a true oscilloscope, which we call the "Analysis Scope". This function also proved its worth as we hacked our way through unexpected operation on noisy audio lines. We also managed to stamp out a bug related to a rare-but-possible event that would corrupt measured audio values and disrupt related functions.

This field trial went so much better than the first. We added a ton of new functions, and the client really seemed to be enjoying the process and the product.

Still, we had a few more notes that required a bit of additional development. We returned home after promising to come back for one final week of testing.

The final tweaks and a big new feature: The third field trial in February 2019

Although our biggest product design upgrades came between field trials one and two, there was more to do. We returned back to our office in Fresno, California with several new things to add for the final design.

We did some additional fine tuning in the web interface, ensuring that all of the functions and default settings were precisely what our client intended. Obviously, we were designing the NetGuardian ADP with an eye toward applicability at many different entities. Still, we first needed to satisfy and delight our very first client for this new custom product.

We built the user manual into the box itself

At this stage, it was time for me to embrace my documentarian role. I had previously written preliminary documentation into a PDF user manual from my hotel room. I was updating this on-the-fly during the field trials to support testing and evaluation.

To make this new product far easier to use, I inserted helpful "how to use the screen" notes on each page of the web interface. That's how modern devices should function. It eliminates the very real risk that, years from now, somebody will struggle to make adjustments without a manual.

Auto-Tune took out all the guesswork

Finally, we were incredibly excited to have developed an automatic tuning function. The idea here was that manual adjustment of the tuning level would no longer be required.

At our offices, this feature went through several stages. The team started out with an "cruise control" adaptive algorithm. The idea here is that the system moves from its current location to the ideal location in a smooth fashion. This sacrifices speed in the interest of smoothness.

Ultimately, however, this algorithm proved to be overly complicated and unnecessarily slow. Although it only took a few seconds, using this method would require installers to key the radio several times to achieve automatic tuning. During this phase, it's possible that messages would be too loud or too quiet to be understood properly.

I suggested to the team the choice that we eventually made: "Why not just make a 0-255 reference voltage lookup table?"

In this incredibly simple algorithm, the input signal strength (dB) would be translated to a step level on the digital potentiometer controlling signal gain. Since there were no other factors in the signal chain of our device, this was the one and only consideration that needed to be made.

What's more, the simple algorithm would auto-tune at high speed within about 5ms on each and every radio broadcast. It could theoretically be left on to automatically adapt to changing line conditions. I still chose to give the users an option to disable auto-tune when more manual control was desired.

We headed back to the client site with final hardware

We flew back out to the client site with this final version of hardware and firmware in hand. We made just a few firmware tweaks during the week. The "roller coaster" experience of the first field trial a few months prior seemed like a distant memory. Things were going incredibly well.

Full Comparison: Existing Box vs. New NetGuardian ADP

Table comparing existing solution against the new NetGuardian ADP device

I prepared this complete table for our end user during development. It describes every significant difference (and a few useful similarities) between the old box and our new custom solution.

Long-term pre-deployment testing through March 2020

As excited as everyone was to start rolling out the new NetGuardian ADP, it's always important to do a reasonable amount of endurance testing. We had 11 units deployed (with a handful in labs but the rest seeing real use in the comms network).

Since we were looking at a full deployment of well over 100 units, we chose to let those units run in place for a full year. They had their own performance logging, which was occasionally exported for review.

Because any embedded remote device has relatively limited storage, we were also using a deployed T/Mon master station. It was set to poll each ADP roughly every 30 seconds. Any alarm conditions were logged indefinitely on T/Mon's redundant hard disks.

Endurance testing proves the design in the real world

After all of our efforts during the 3 field trial visits, the year of endurance testing went incredibly well. We had no hardware failures, and only a handful of tiny firmware changes to address:

  • When two channels activated at the same time with a shared priority level, multiple "bump" events were logged into history instead of just one. This is a simple firmware bug to correct in the next firmware release.
  • Performance monitoring graphs needed slight improvements.
  • The interface for level calibration of output voltages needed an adjustment. This will make it easier to correct situations where the output is high or low.

The complete 2020 rollout

We're now very excited to roll out the NetGuardian ADP to all remote receiver sites for this massive DPS client. Although the evolving Coronavirus reactions have hampered large segments of society, we press on. Virtually every DPS user is an "essential business" or government agency that must continue to serve the public. Therefore, so are we.

We're tidying up the final firmware adjustments. We're helping our client secure funding internally for the project.

As you might expect, our long history with this user is helpful. It makes us a known quantity for key decision makers who must always manage risk. 20 years of proven NetGuardian performance mitigate a ton of that risk.

What Device do YOU need DPS to build?

This is just one example of custom remote monitoring development from DPS Telecom. It took us beyond our core of RTUs and master stations in many ways. We incorporated brand-new audio circuitry that will be useful in many different applications.

What development story do YOU want to be part of? What can DPS make for you?

Tell me now

Give me a call at 1-800-693-0351 or send me a quick message to tell me what problem you're facing. I promise, it's a fun (and sometimes wild) ride that always ends smoothly - and makes your job easier.

Share: 
Andrew Erickson

Andrew Erickson

Andrew Erickson is an Application Engineer at DPS Telecom, a manufacturer of semi-custom remote alarm monitoring systems based in Fresno, California. Andrew brings more than 17 years of experience building site monitoring solutions, developing intuitive user interfaces and documentation, and opt...