Create Notification Triggers With PrtgAPI

 The Use Case

Here's a hypothetical situation: You are an MSP using PRTG in combination with PagerDuty for after hours alerting. However, your your engineers are being woken up all night by alerts that they don't really need to know about. Your configuration might be as follows:

• Each client has their own probe
• Each probe has a State Notification Trigger assigned to it to send an email to a client-specific PagerDuty email
• Notification Triggers will be activated when a sensor has been down for 15 minutes
• PagerDuty will ignore any emails that are sent within business hours

As a result of your Notification Trigger being defined at the root of the client's probe, all sensors under that probe will trigger alerts outside of business hours. Do you really care that the Terminal Server's CPU usage is high at three o'clock in the morning? Multiplied across thousands of sensors, a setting like this means your team is getting woken up all night for absolutely no reason. You need to find a way to somehow prevent the sensors you don't care about from triggering their alerts.

Libraries can let you assign triggers to specific sensor types, but since each client has their own unique email, you would need to maintain a separate library for each individual client. 40 clients equals 40 libraries. Good luck maintaining that! You could always use schedules to disable the sensors you don't care about, but then you'll lose valuable data that could have helped solve an issue.

Ideally, you would like Notification Triggers to only apply only to specific types of sensors for specific types of devices. Complicating matters further is the fact that each client has their own naming scheme. All of the clients in your Hosted Environment follow a similar naming scheme, but everyone else's is different.

A Potential Solution

One way to solve this problem is to perform the following:

1. Define a set of naming schemes used by your clients, specifying how each client identifies certain types of servers (Exchange, SQL, Firewall, etc)
2. Specify the types of sensors each server type cares about. You can potentially extend this to require additional sensors on specific server types for clients with special requirements.
3. Construct a client map, outlining the probes, notifications and naming scheme to use for each client.

For each client, we will

a) retrieve their probe,

b) retrieve the devices on the probe that match this client's naming scheme,

c) retrieve the sensors we need to apply notification triggers to for each device type, and finally

d) apply notification triggers to the sensors that (still) require them.

As a large part of this project involves defining configurations, we will store our configuration in a set of separate JSON files. Our script will then parse these files to perform the required actions. This will then allow other engineers to update the configuration without having to wade through all the code.

Our config files are as follows:

File	Description
Clients.json	Provides the high level identifying information for each client: [      ["contoso (host*", 1001, "hosted"],      ["fabrikam", 2002, "fabrikam"] ]   Each client record contains three pieces of information: a wildcard describing the probe that this record should apply to, the ID of a sensor to copy the notification trigger from, and the naming template to use for the client.  In this scenario, we have two clients, Contoso and Fabrikam. Contoso has two probes (Contoso (Hosted) and Contoso (Office)). We clone our notification triggers from the Ping sensor of each client's firewall, which probably won't be deleted any time soon.
	Describes the naming scheme used by each client {      "hosted": {           "probe": "Probe Device",           "domain-controller": "*-dc-*",           "exchange": "*-exch-*"      },           "fabrikam": {           "probe": "Probe Device",           "domain-controller": "FABSRV01",           "exchange": "FABSRV01"      } }   Hosted clients follow a fairly consistent naming scheme, utilizing one server per role. By contrast, Fabrikam has a single SBS server functioning as both the Domain Controller and the Exchange Server.
SensorNames.json	Defines the sensors that require notification triggers on each server type {      "probe": ["Probe Health"],      "domain-controller": [           "DNS", "Service: Active Directory Domain Services"      ],      "exchange": [           "HTTPS", "SMTP", "Exchange Database:*"      ] }

We start by converting our JSON config files into PowerShell objects by using the ConvertFrom-Json cmdlet. For each client row in Clients.json, we then attempt to retrieve the probe of the client (using the Get-Probe cmdlet), the sensor containing our “master trigger” (using the Get-Sensor cmdlet) and then locate our naming template from the templates defined in NamingSchemes.json.

We then proceed to iterate over all of the devices under each client (retrieved via the Get-Device cmdlet). Based on the naming scheme(s) our device matches in NamingSchemes.json, we can determine the roles this device performs. Once we have the role, we can identify all of the sensors this device should be alerting on by cross referencing with the sensor names defined in SensorNames.json. As a secondary benefit, we can also verify that our devices have these essential sensors.

When we are ready to create our new notification triggers, we do so by using Notification Trigger we retrieved from our “master sensor” (via the Get-Trigger cmdlet) in combination with the the New-TriggerParameters and Add-Trigger cmdlets.

 

You can download examples of the JSON files and the PowerShell script here: 

Sensors types that are considered mandatory for all clients (such as Ping and Disk Usage) can potentially be hard coded within the script. Further enhancements that can be made include tracking which devices weren't processed, implementing naming scheme "hierarchies", specifying device ignore lists and customizing scanning interval settings.

Once everything has been configured, the script can potentially be set up as a scheduled task. An EXE/Script Advanced sensor can be setup to validate if new clients have been added to the script, but that's another project!

 

And that concludes our series of posts covering the useful PrtgAPI. We highly recommend that you follow lordmilko on GitHub. If you have questions or comments about the tool, then definitely leave us comments below!