A Magic Trackpad (that is not simply oversized): Using a tablet as a customizable PC input device

I think productivity and usability enhancements could be afforded by replacing standard laptop trackpads with multi-touch displays. The use of multi-touch gestures and tracking for mouse input could be retained, but the flexibility offered by a touch screen could offer new interfaces dedicated to specific applications. If an external mouse is present, standard trackpads become redundant.

For instance, if the user is running Ableton Live, the touchscreen trackpad could display virtual sliders to control volume to mix different tracks. Ideally the custom interface would be integrated directly with the application, but it could also create input through simulating key chords or mouse movement.

To test the utility of this setup I built an Android application for my Android-ified Touchpad and a companion C#/WPF application for Windows. The Android application connects to a COM port on the laptop via the Bluetooth Serial Port Profile service. Both the Android application and the companion application can read and write to the COM port.

The Android application is capable of three types of interaction. Two rows of icons at the top of the tablet can launch Windows applications, open websites, and control global settings like volume. A panel on the right-hand side serves as a trackpad with left, middle and right click buttons. An object that listens for multi-touch gestures is attached to the panel for the trackpad; multi-touch gestures could be used to open, close, preview, and navigate through windows as well as scrolling. The left side of the application features a scrollable list widget that serves as an extended clipboard. The companion application creates a global hotkey (Windows key + “C”); when pressed it simulates the “Control” + “C” (copy) key chord, retrieves the text item from the Clipboard, and writes it to the COM port. The Android application receives the text and copies it to a new cell in the list widget. The user can then store multiple items in this extended clipboard and select which item to paste.

My use of Bluetooth SPP has been problematic. Both the Android app and the companion application need to constantly listen for incoming data. In Windows this is managed easily, but in Android this requires creating a new thread for the application to read the Bluetooth socket at all times; reading the socket is a blocking operation that prevents the UI from functioning while the app waits for the socket to receive data. Multi-threading avoids this issue, but I am still experiencing an issue in which the serial stream seems to become clogged, for lack of a better understanding of the problem. After extended use of the trackpad, the subsequent message from another part of the app is not immediately sent. Instead, it is necessary to queue seven or eight messages behind it before they are all pushed to the COM port. This results in a sometimes unresponsive UI. I suspect that the app’s trackpad is sending messaging too quickly.

 The Android project

 The C# project

 

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>