So at the weekly “Blame & Shame” meeting, it was mentioned that the SMS notifications of Server problems were possibly a bit excessive. The monitoring system sends out a notification of a problem every hour. If you happen to have your phone switched off over night you are flooded with messages as soon as you turn it on again. There is also a “Costings” Issue due to the number of messages being sent, and the number of people receiving them.
No problem, Our Intrepid Systems Team will solve the problem.
So we have A Systems Administrator “Linux none of your iffy Microshite stuff here“, and a Systems Developer/Programmer.
The Design process went something along these lines. :-
Plan A: Procmail:
The server was set up for local delivery of messages, a fancy set of procmail scripts were considered to pick up the incoming emails, analyse for importance, repetition, and then handle accordingly. A Trial system was put in place, and a service taken down to test the error reporting. Then a better idea was suggested.
Plan B: Scripting:
Re-Write, loose the procmail! Why not have a custom monitoring command that calls a specifically designed set of scripts to do what was required? Have a database backend, the whole works. All that is required would be the choice of programing language. The decision went like this: –
- SA “We need a language the Boss doesn’t know So he cant play and break!”
- SD “what do you fancy? Python? “
- SA “C+?“
- SD “lisp?“
- SA “Fortran?“
- SD “Assembler?“
- …..Several Minutes of Listing Obscure Languages…..
- SD “Ok PHP it is, lowest common delimitator and everyone knowing it…“
So the script was designed. Databases created, testing put in place, with a script that was becoming more and more fancy in its plan and functionality. Almost to the point of the script not just notifying of any errors, But actually connecting to Server with a problem and fixing it.
What was the final outcome?
The little tickbox labeled “Only send out 1 notification per problem” was ticked.