Reverse engineering Think With Google's My Edit

I came across this lovely project of an analogue way of interacting with the "Think With Google, My Edit" web app.

You can read more about the project on Multi Adapter and Roland Ellis' websites.

There's a physical paper interface printed using conductive inks. Crossing a box on the paper form completes a conductive ink circuit and the built-in Arduino then sends a signal to the browser connected via a USB cable.

I was very interested in how the communication actually happened.

Raw access to the USB port is available in Google Chrome for Apps only and not normal websites so another approach must have been needed.

Digging into the code

I found the minified source code for the site and passed it through Unminify. It's written in Angular and after a while I'd thought I'd found the code responsible for connecting to the notepad device.

This block seems to match the dropdowns on the site:

var b = 36,
  c = 38;
var d = {
    3: "brand",
    4: "choice",
    5: "drive",
    6: "picks",
    7: "latest",
    8: "popular"
  },
  e = {
    12: "mobile",
    13: "search",
    14: "display",
    15: "video"
  };

And there were 2 functions that were assigned to $(document).on("keypress", l); listeners. So, when plugged in, the notepad must emulate a USB keyboard and start sending keypresses. When the browser is focussed, it hears these key presses and can change the interface.

The first editPadHandshake must be the initialisation of the pad to browser communication.

function g(e) {
  e.keyCode == b &&
    ((h = !0),
    a(function() {
      h = !1;
    }, 2e3)),
    h &&
      e.keyCode == c &&
      d.$apply(function() {
        d.$eval(f.onHandshake);
      });
}

keyCodes b and c above are ASCII characters $ and &.

To verify this, go to [https://myedit.withgoogle.com]() and click "Yes, I Have a Notepad", then type the letters "$&". The UI should say "Notepad connected". Success!

The next block was a lot more complicated:

function(a) {
  return a.keyCode == b ? void(i = 0) : a.keyCode == c ? void(i = -1) : void(0 > i || (a.keyCode >= 65 && a.keyCode <= 90 && (d[i] && f.$apply(function() {
      j.assign(f, d[i])
  }), j(f) && e[i] && f.$apply(function() {
      k.assign(f, e[i])
  })), i++))
}

It's almost impossible to read minified, so I manually tried to space it out to make sense of it, leading to:

return (a.keyCode == b
? void (i = 0)
: a.keyCode == c)
  ? void (i = -1)
  : void (
      0 > i ||
      (a.keyCode >= 65 &&
        a.keyCode <= 90 &&
        (d[i] &&
          f.$apply(function() {
            j.assign(f, d[i]);
          }),
        j(f) &&
          e[i] &&
          f.$apply(function() {
            k.assign(f, e[i]);
          })),
      i++)
    );

Not a lot better. void it turns out:

allows inserting expressions that produce side effects into places where an expression that evaluates to undefined is desired

—via MDN

Which I didn't know about. Armed with the new code layout and some (non-magic) pencil and paper meant I could draw a state diagram of the flow through that code for a given key press.

The sequence of keypresses starts with a $ and is then followed by ANY key between A and Z. Each of these presses increments the i counter. And when the value of i matches a array index in the list of categories (d above) then the UI is updated to select that category. So, typing the sequence $AAAAA will select the choice option in the UI. This happens so quickly that you don't see the UI flicking through every one.

The same counter is also used to select the second dropdown. But how? It took me a lot longer to realise that any non A-Z keypress also increments the counter, I chose to use @ but anything will work. So, you have 3 types of commands: $ starts the sequence, 'A' increments i but also triggers a UI update and '@' increments i but doesn't trigger a UI update.

So, to choose item 6 in the first dropdown and item 14 in the second dropdown you'd send the following keypresses:

$@@@@@@A@@@@@@@A

You can try all this with your keyboard or I wrote some functions to be dropped into the Developer Console that triggers the keypress events.

/*
 Return a sequence of keypresses to select a category and sector:
   https://myedit.withgoogle.com
*/
function cmd(cat, sec, start, active, pad) {
  var items = [start];
  for (var i = 0, len = sec; i <= len; i++) {
    i === cat || i === sec ? items.push(active) : items.push(pad);
  }
  return items;
}

/*
    Trigger a keypress on the document
*/
function dispatch(k) {
  var e = $.Event("keypress");
  e.keyCode = k;
  $(document).trigger(e);
}

/*
    Type a sequence of keys
*/
function type(keys) {
  keys.forEach(dispatch);
}

To use it to choose item 6 in the first dropdown and item 14 in the second dropdown you'd do:

var start = "$".charCodeAt(0);
var select = "A".charCodeAt(0);
var move = "@".charCodeAt(0);
type(cmd(6, 14, start, select, move));

Thoughts

This is quite a simple way of enabling interaction with web pages and is pretty cross-platform and cross-browser. I love the idea of being able to draw on paper to complete the circuit and presumably erase the pencil line to reset it.

However, having something randomly typing keypresses is probably a bit fragile when you switch to another application and it's writing random characters into your document. I don't have one of the actual notepads so I'm sure how often it sends the characters.