The Evolution of Automation

After 5 lessons (well, 10 really), I thought I'd pause and take a look, not at specific automation code, but how automation evolves out of current processes. One of my first posts on this site was about the reasons to automate (I'M HAVING AN INDIGNATION MEETING). It gave four reasons why to automate and I've spent the last five weeks showing how to automate. In this article, I want to discuss how current processes evolve into an automated process in an office environment (can you use anymore Darwin buzz words without talking about 'Natural Selection'?!). There were a couple of surprisingly useful developments in a recent project that I hadn't actually planned out but just organically evolved when the project came to critical junctures.

The other day a project came across my desk that needed some updating. For some time, I've been retrieving some text files via a secured web server and adding some html to the contents to deliver to end users via a different secured web connection. Due to the security setup at my organisation, there had to be a lot of 'tweaking' done to make the automation server (let's call it 'Athens') connect with the data server (let's call it 'Carthage') as well as the machine that was to be the final resting place of the data where the end users will access the report (let's call that one 'Hellespont'). The process was executed by a 7 year old VB exe that frankly was a pain to deal with. Removing reports were a pain; adding reports were a pain; not fubar-ing the entire process doing either of the above was a pain...Not knowing what you were doing with VB was a pain. The whole thing was enough to make you ignore it till you could get your hands on an Impala outfitted with a military class JATO rocket, put the entire source-code and exe in the front seat, flip the switch, sit back and watch the glory. Well, as it turns out, the final security piece fell into place before I could get my hands on that JATO and I knew it was time to grab a large-caliber handgun, a tac-light, and crawl into that Damnbeast den (better make it machine pistol...um...actually two (John Woo style you know)....with extra ammo..all armour piercing of course...(the bullets not the pistols)...although....armour piercing pistols....hmmmmm...anyway). I then started to map out how I was going to make this thing work.

My first decision was to make it easy to add and subtract reports from the process (see above). I didn't want to code each report into the function separeatly because that would take a 10 line process and make it 'the-number-of-reports' times 10 lines. In case you weren't sure of what to call that, it's called 'Ugly and Kludgey'. I decided to create a text file that would contain the address of each report needed on the server Carthage, as well as, the final name of the report when I place it on the server Hellespont for distribution. These two pieces of info would be separated by a '|' character. It is just a very simple text document that I can step through one line at a time using 'foreach' to parse the file and and 'gett' to parse the line. The file is sort of a 'definitions' file. This method has several benefits:

1. to add a report, I just add one line to the end of a text document rather than opening up QM to add like 10 lines of code to the 180 that are already there (if there were say 18 reports). When I say 'I', I'm really saying 'any of my co-workers' who need to add a report later on because I got hit by a bus or won the lottery.
2. to remove a report, all I have to do is delete one line rather than opening up QM and deleting like 10 lines of code. When I say 'I', I really....well you get the point....It's called Agile baby!!
3. If reports are added or deleted, there's no need to make 'versions' of the program (like we do now) because I'm not actually changing the code.

So, now that I have my list of reports and can get info on each of them individually, I need to build the routine for Athens to get them from Carthage..... And here we have all this simple and wonderful code using cURL to retrieve the files from Carthage and load them each individually into a string variable . I'm not going to go into... (wow, that didn't work well at all)...I'm not going to go into what I'm doing to get them or how I'm manipulating them (maybe I'll go over that another time); I'll just say that for https retrieval, cURL is really, really, nice!

So, to sum up, I've retrieved the files from Carthage and modified them so they look better for the end user and then placed them out on Hellespont for the end user. When I ran it the 'first' time it worked great till I realized one of the reports is no longer used. No problem, all I have to do is delete that line and....wait....that is never a good idea when dealing with capricious users (or any for that matter). I'll tell you a little principle I've developed in my years of report design. The report that has not been looked at for over a year by a single living soul will never even be thought of until you delete it from your report delivery system. Then an admin assistant is on the phone saying they want it....oh, and they want it for a meeting....today....at 9:30.......um, it's 9:40 now....yep. And here it is in business speak:

The value of a report is not based on its actual use or even its usefulness but rather directly on its difficulty in recreating its process after it has been discontinued....for six months....by someone other than yourself.....who quit because they won the lottery.......you just wish they would get hit by a bus....while you were driving it.....which is what you'll be doing for a career if that admin assistant doesn't get that report printed within the next 10 minutes.

I don't have a catchy name for it yet but now that it's published I have a copy-write on it 80) ....BTW: I'm taking suggestions for the name.

So, as you can see, I'm a little reluctant to just delete the report entry. [Enter Stage Right: Comment Delimiter] ....Ooooh, excellent! Comment delimiter! I can actually use almost any character since all the report addresses have to start with the '/' character. I chose to use the '#' symbol because I'm used to it when I'm doing Korn shell work in Unix. My next step was to use 'find' in a simple 'if' statement to skip the entire process if the first character is a '#'. Done. Now I can keep that report ready in case someone wants it again.....wait a minute....I can also use a 'Comment Delimiter' to do things like 'make comments'...just in case I'm found guilty of a crime for hitting that guy with a bus and have to go to prison or something...yeah, I know, long shot; no jury is going to convict me of 'painting a new center-line' with that guy after deleting that report process without documentation. So without any extra code, I've also added the ability to place as many comments as I want anywhere inside the file as well as deactivating reports.

The 'definitions' file and the commenting capability were things that weren't specifically planned on, they just 'grew' out of the process as it went along. It solved problems I hadn't realized and became more and more useful as the project progressed. The comment capability really makes the whole project much more Agile since anyone who comes in behind me can easily find out what I'm doing from the comments in this definitions file. Which leads me to the next step...documentation.....uhgh.....talk about a Damnbeast.

Lesson 4: The Extra Mile

For me string variables are the sonic screwdriver of coding. I use them to build not only the functions I need but the logs that monitor them, so being able to manipulate them is crucial for doing the things I want.

Let's take a look at the whole log file concept. I have a text file that is filled with lines of text something like this:

Lacedaemonians|60|Menelaus
Boeotians|50|Thersander
Minyans|30|Ascalaphus
Arcadians|60|Agapenor

The first thing you need to do is get the file data into a string so you can start modifying it.

str a b c
int z
a.getfile("C:\qm\ships.txt")

Now lets get just the last line of that text file by using two different functions.

b.getl(a (numlines(a)-1) 2)
out b

The 'getl' function first asks you for the string from which you want to get the line (variable 'a'). The second part is a little trickier; the 'numlines' function tells you how many lines are in the string 'a', but the '-1' needs some explaining. The 'getl' function is what is called 'zero-based' which means (it's terribly annoying and prone to 'one-off' mistakes) it starts counting at '0' not '1' just like a computer....um...ok yeah, that does make sense now but it's still annoying. So, it has 4 lines but the last line is 3 hence the '-1'. So now 'b' contains the last line of 'a'.

Now, let's remove the line that we loaded into 'b' and put it back into 'a' but in the 2nd position.

a.RemoveLineN(numlines(a)-1);;this removes the line at the end (that we just copied) so it won't be duped.
a.InsertLineN(b 1);;again zero-based function.

out a

For the next step, let's add a new line. This new line can come from anywhere i.e. a file, the clipboard, or text selection, but here I've just set 'b' to it and then dropped it onto the end of 'a'.

b="Salamis|12|Telamonian Ajax"
a.addline(b 1)
out a

A note about the flag used on line two (not zero-based 80) ). The '1' tells 'addline' to not add a 'new line' character at the end of 'a'.
A note about flags: flags are just parameter codes for functions that allow you to modify the basic behavior of a function.

Now let's say you want to pull out the last part of each string and take a look at it (e.g. Menelaus, Thersander, Ascalaphus, etc). Here's how to set up the loop and grab the last section.

rep numlines(a)
,b.getl(a)
,c.gett(b 2 "|[]")
,out "c=%s" c

Note: the commas are another way of indenting the code within the QM Editor. However, when you copy/paste the code into another program, the tabs are replaced with commas. If you copy the code above to your QM Editor, you'll get the commas unless you use the menu Edit -> Other formats -> Paste escaped option.

Now, about the code, I'm using the 'numlines(a)' instead of checking for the error code output from the 'getl' just as a matter of preference (I'll explain more about that in a minute). The 'getl' is missing the line number to pull but when that value is missing, it just pulls the next line; you could use an incrementing integer but this is simpler. If, however, there is no next line, the 'getl' function returns -1; it would be that -1 that you would check for if you didn't know exactly how many lines where in the variable. The real money maker is the 'gett' or 'get token'. Think of a token as a 'chunk' of the string. To break up each line, I've used the pipe character '|' as my delimiter because it was very unlikely to show up in the data. But I also have the '[]' within the 'gett' command in its delimiter section. This tells QM that the last chunk of data in a line will end at the new-line character. Otherwise, it will grab the next line's first section (you can try it out to see how it affects the data pull). And finally, you'll notice that through out this lesson I've been using the 'out' command to monitor my changes in the data but I've used a modified version of this at the end to include some prefix text to identify the variable. This technique really comes in handy when I'm debugging and vetting several variables at once and it can contain many variables on the same line as well as identifying them.

So there you have it, some of the really useful string functions. And if you really want to turn it up, you can start getting into using 'replacerx' and Regular Expression coding...not that I would know anything about that. ;o)

So have at it and I'll see you next time cause the hook always brings you back.


BTW: Here's the entire code block to make it easy to import into the Editor (don't forget to use the Paste-Escaped option).

ClearOutput
str a b c
b.getl(a (numlines(a)-1) 2)
a.RemoveLineN(numlines(a)-1)
a.InsertLineN(b 1)
b="Salamis|12|Telamonian Ajax"
a.addline(b 1)
out a
rep numlines(a)
,b.getl(a)
,c.gett(b 2 "|[]")
,out "c=%s" c