Wednesday, December 28, 2011

Part 1 - Automating watch.slingbox.com

The first step towards creating our "Slingcatcher alternative" is to write a program that will launch a web browser, navigate to the correct page, connect to a Slingbox of our choice from the directory (if there is more than one to choose from) and then switch into full screen mode.

The idea is that we will then execute this program when the user chooses the appropriate option using their remote control, but in part 1 we will just look at constructing the program itself and not worry about how we are going to trigger it. That comes later!

One of the nice things about using OS X is that so much can be achieved using AppleScript. In a nutshell, AppleScript allows us to write simple English-style statements to control applications. These can be basic commands like launching or quitting applications, to more complex operations such as opening and manipulating documents inside an application. It turns out that everything we need to do for this project can be done with AppleScript.

For example, here is what we need to write to tell the Safari web browser to launch or if already launched to become the application with the focus:






If you load up the AppleScript Editor (you can find this inside Applications/Utilities) and type these lines into the main area and then press the 'Run' button then you can see this for yourself.

Now the precise possibilities achievable using AppleScript to control a program depends on what support has been built into the program, but even in applications where little support has been built in there is something known as "GUI Scripting" which basically means you write AppleScript to 'click' on certain buttons inside the application in order to control it. The application itself need have no knowledge of who or what is doing the 'clicking' - it just reacts in the usual way.

Anyway, returning to the subject in hand, having often used the watch.slingbox.com page to watch my Slingbox I was familiar with the process involved, assuming I had previously signed into my Sling account and told the browser to keep me signed in:

  • by navigating to the page, it would automatically start connecting to the last Slingbox viewed
  • I could hit Cancel at this point if I didn't want to view this Slingbox, or I could let it connect
  • If I hit 'Cancel' I would then be presented with a list of Slingboxes to choose from
  • I could then click on a Slingbox and it would then attempt to connect to this one
  • If I happened to be viewing the same Slingbox on another device I'd need to click 'Yes' when prompted, to override the other device
  • Once connected, I would need to click on the 'Full Screen' button to switch into full screen
  • If I became disconnected, it would switch out of 'Full Screen' and give me the option to re-connect
  • Quitting the browser or closing the window would be enough to disconnect from the Slingbox

So my aim was to write some AppleScript which would be able to handle the above process automatically without needing keyboard or mouse input and thus would be suitable for triggering via a hardware remote control.

In order to do this it was clear that the script would need to have some sort of knowledge of the state of the web page at any given time - for example, in order to automatically 'click' on the 'Full Screen' button it would need to know that this button was visible and depending on the method used to 'click' it might need to know its position on the screen.

The two browsers I looked at were the default Apple Safari browser shipped with OS X and also Google Chrome. It turned out that both would be useful. Safari provided the best AppleScript support and would thus be the basis for this project, but I discovered that Chrome could be very a useful testing/inspection tool.

In Chrome it is possible to choose the menu 'View' -> 'Developer' -> 'Developer Tools' and then by right-clicking on any item on the web page and choosing 'Inspect Element' you can see exactly how that item has been constructed in HTML-speak. Even better, by choosing the 'JavaScript Console' option you can type in JavaScript commands directly to further query the different HTML elements and generally try things out that can later be called from the AppleScript program.

Google Chrome Developer Tools

Using this method I was able to identify the key elements on the page that would indicate which 'state' the page was in. It was then a case of writing the corresponding AppleScript to tell Safari to execute Javascript on the page to retrieve back this information.

For example, here is some AppleScript that registers a JavaScript function to determine if the 'Full Screen' button is visible on the page:













To call this function from AppleScript we just execute some more JavaScript and store the return value in a variable:







In a similar manner, we can then write other functions to identify the presence of other key elements on the page. So using this method, now that we were able to identify the state of the page, we needed a way for the script 'click' on the buttons themselves depending on where in the overall process we are. 

This proved a bit trickier. Initially I took the approach of directly issuing mouse events onto the elements on the web page. A bit of Googling revealed how to do this, but I found this to be a little unreliable when applied to this web page. Sometimes the events just didn't seem to fire and (for example) the script would sometimes not click on the 'Full Screen' button.

So I thought a more reliable approach would be to address it at a different level and actually have the script move and click the mouse pointer at the correct co-ordinates on the screen and then let the web page react in the same way it would had this been the user doing the clicking. It turns out that AppleScript provides some options for doing this, but unfortunately I couldn't get these to work reliably either! A bit more Googling revealed that others had experienced problems too.

In the end, I was finally able to achieve what I wanted via a useful utility called 'cliclick' which you can download from here:


Once downloaded, you will need to copy the file 'cliclick' into your 'usr/local/bin' directory. If the 'bin' directory doesn't exist, then you will need to create it. You may need to enter your password to do this.

So what is this program? Well, this little gem is a small shell program which when invoked with the correct arguments will move the mouse pointer to a certain location and issue a click event. Since it is possible to call shell script from AppleScript, I was able to write a small function that would use this little program to click anywhere on the screen.

So piecing it all together, I was able to write an AppleScript program that could do everything I needed it to. In order to provide maximum flexibility I have allowed the script to take in an optional argument - the name of a Slingbox. If this argument is not supplied, then the script will just allow the web page to auto connect to the last Slingbox viewed. However, if a valid name is passed in (this must match the name shown on the Slingbox directory page) then it will instead automatically view this Slingbox instead.

Now rather than go into great detail about how the script works I've instead commented each section - if there are any expert scripters out there feel free to suggest improvements. Until a couple of weeks ago I hadn't written a line of AppleScript so I'm sure there are ways it can be improved!

You can download the script from here:


Once downloaded, you can double click on it to load it up into AppleScript editor and then click the 'Run' button to execute it.

Note that the script relies on the 'cliclick' program mentioned above having been installed in 'usr/local/bin'. If you installed it elsewhere then you will need to modify the AppleScript function 'click_mouse' to refer to its correct location. You can find this function near the bottom of the AppleScript.

As mentioned earlier, this system also relies on you being logged into your Sling account in Safari - the script will simply wait and do nothing if the page throws up the login page.

One other thing you may need to do before you run the script is to tick the 'Enable access for assistive devices' box inside 'System Preferences' -> 'Universal Access' -> 'Mouse & Trackpad'. If you don't do this, the 'cliclick' program may not work and thus neither will the script.

Finally, I have only tested the script on OS X 10.7.2 Lion.

So that's part 1 - In part 2 we will take a slight but necessary divergence to tackle the issue of the on-screen virtual remote missing some keyboard shortcuts. To build a fully operational system we need every key on the remote to have an equivalent keyboard shortcut.

3 comments:

  1. A** work Oviano. I've been searching for about a year on how to work around my now broken slingcatcher, and your walkthrough is the best I have found so far!

    I'm looking forward to reading part two. I will have to look in to purchasing a mini, as there is no way im apaying for another slingcatcher replacement via slingmedia.

    Keep up the good work!

    Si

    ReplyDelete
  2. Glad you like it Si - I really must get around to completing the final parts for setting up Remote Buddy to actually control the thing!

    ReplyDelete
    Replies
    1. This looks great, unfortunately it is not possible to download the script anymore. Do you still have the script available?

      Delete