Turn! Turn! Turn!

I was just looking over the Styra Project blog this weekend and was shocked to see that it has been two months since our last update.  Where does the time go?  The holidays were an incredible amount of fun and we had the opportunity to visit many friends and family, but the time was also tinged with sadness as my Grandpa Jerry passed away in mid December.  Annie and I debated the best way to catch up, whether to have a series of small posts or just one large post.  I’ve opted for one large post, and as there has been a lot going on; I guess I’d better get started.

In September and October I had started working on a new prototype I call Mystique.  Mystique has the same 16 button browser navigation controller that we have been using all along, but in more of an arch shape.  The unique feature of Mystique is that it is the first prototype to incorporate two Styra Controller boards into the same prototype.  The second controller board allows additional buttons to be added.  For Mystique, the additional buttons are used to store username, email address, and a password.  

Mystique

Mystique also is the first prototype to utilize the i2c eeprom chip for storing keyboard configurations. Configurations can be created and modified using a simple text editor on the workstations and uploaded to the keyboard using a python-based command-line tool I wrote for just that purpose. What this allows is that each button can have a macro action of up to 64 bytes and each keyboard can have up to 512 macro actions defined.  These macros can be updated from the host workstation without the need to modify code on the keyboard.  

32KB i2c thumb drive

In November, I was able to take two weeks off from work to travel the Pacific Northwest visiting family.  Annie, Geddy, and I set off on our adventures loaded down with supplies for the trip AND four Styra keyboard prototypes.  The first stop on the trip was to see my cousin Andrew.  This was Andrew’s first opportunity to use the prototypes we’ve been developing and we spent two days working with him and my aunt Jonell on the physical constraints of workstation layout in addition to giving Andrew a chance to play with the prototypes.

Annie and I chose not to try taking any pictures of Andrew using the keyboards because this is private and he has been gracious in allowing us to share his story on this blog.  Working with Andrew in person was incredibly helpful in understanding the limitations that he faces every day.  No amount of wrapping my hand and using one arm can compare.  The following are just a few of the notes from seeing him use the prototypes:

• Trackball might work for left and right motion, but Andrew is not able to roll the ball forward and back.  The joystick controller worked, but will require a lot of practice before it will be a usable solution.
• Andrew tends to push the buttons on the edges rather than in the center.  The cheap buttons I originally purchased do not always make great contact when pressed in this manner and results in missed button presses.  I think that higher quality buttons might help.
• When Andrew presses a button, it is generally very slowly.  Sometimes his hand will jerk with the effort of controlled motion.  This generates bounces that are well outside the 25ms of the hardware debounce period.  I am able to reproduce this and I think that adding a new macro action type that only registers a button press once per second would help with the physical bouncing (as opposed to electrical bouncing).
• Navigation keyboard prototypes—all of them—are a bit too complicated for Andrew to use. For someone who is familiar with surfing the Internet, they are just a little inconvenient until you learn where the buttons are to do what you want to do.  For someone who has NEVER surfed the Internet on his own, the navigation keyboards are overwhelming.  Combine that with the fact that websites are not keyboard navigation friendly and it is a real problem.  Some websites have hundreds of fields that take forever to ‘tab’ through.  Others have popup adds that simply cannot be cleared with keyboard shortcuts.

For Andrew, position is everything.  To get the keyboard close to his wheelchair, a desk wouldn’t work.  He needs to have the keyboard come to him and have it adjusted for height, angle-positioned for his left hand.  To complicate matters, as he becomes tired, he needs to adjust his position and therefore the keyboard position needs to be adjusted on the fly.  For our testing I brought a music stand and strapped each keyboard to it.  We could then move it around his wheelchair, raise, lower and tilt the keyboard to best fit his needs.  One option I am going to play with is drilling a VESA mount pattern on the back of the keyboard and mount it to an adjustable monitor arm.  This could be a really good solution but will require some experimentation.

Probably the biggest learning experience for me from my time working with Andrew came as we were wrapping the first day of testing.  I know from personal experience that for me to be successful at something, I need to be able to find a way to make it fun.   This could be exercising, studying, whatever…  So I was asking Andrew what he did for fun on the computer and he just looked at me like I was crazy.  There is NOTHING fun about using a computer!  And then it hit me.  His only experience with a computer has been for school work or various exercises.  My aunt and uncle will help him look up sports teams / players and read him articles, but actually using it for fun?  No…  For Andrew, using a computer was more like weeding the garden.  The Styra keyboard wasn’t opening a door, it was giving him a better tool for weeding.  I could see his point.  <Completely off topic, but in Oregon and Washington I wonder if kids are now going to be sent out to weed the weed. :) >

So…  That night I searched for puzzle / strategy games that could be controlled from a keyboard. The next morning I stripped down the ShadowCat prototype and built a much simpler keyboard specifically for playing one game: Entanglement.  This really showed the power of how customizable the Styra Keyboard design is…  This complete rebuild of a keyboard only required 30 minutes. Andrew would play until he was too tired to sit up.  After some rest, he would come back and play some more.  This was great for me because it gave me a chance to see what worked and what didn’t with the Styra keyboard prototypes.  And it was cool for Andrew because for the first time he wasn’t just weeding the garden.  We left the prototype for Andrew to play with while we traveled some more and then we picked it up again on our return trip.  

Modified version of the ShadowCat Prototype

In December, Balihoo graciously awarded us a second grant to continue our work on the Styra Project.  This is a breakdown of the goals for the next stage in the process.

• Experiment with a variety of buttons to identify a good quality button
• Construct four new prototype keyboards, each with an integrated pointer control
• Build a list of games and websites that can easily be navigated using the prototypes
• Documentation.  Create tutorials and possibly YouTube videos describing how to construct the prototype keyboards.

Sharing the Story

(Annie jumping back in here for the latest update!)

With the prototypes taking shape, it was time to give them names and share what they could do. This blog and casual conversations had been the main ways people knew about the project, but next up was to talk about it more and get some feedback for the next stage of development.

First, Luke—known for finding creative names for servers and computers—wanted to christen the prototypes. He threw around a few ideas and landed on character names from the X-Men comics. Two became Shadowcat and Pyro, and James chose Phoenix for the prototype he built. We will introduce you to the fourth, later.

Shadowcat

Pyro

Phoenix

These super prototypes traveled to the Balihoo All Paws meeting to make their public debut. (Balihoo likes to use names too, and their former mascot and logo is a dog, Scout. They also have meeting rooms named for dogs. "All Paws" and "Pawsitive Projects" are their clever canine puns.)

Setting aside his dislike for public speaking, Luke gave a presentation as a way of reporting back to his employers and co-workers on a) how the team had put Balihoo's gracious grant money to use, and b) what the company could do to give input to the Styra Project. Pyro, Phoenix, and Shadowcat got to show their stuff and then stuck around for the day so that people could come by, try them out, and comment in accompanying note pads about the ease of use and any suggestions for what the devices should be able to do. All the comments are being taken into consideration for fine-tuning different keyboard functions.

At the meeting Luke also introduced Phill Thomas, James Hatmaker, Herb Ankrum, and Roy Kimball and thanked them for their contributions as part of the Styra Project.

Then it was back to the lab with kinks to flush out (also, more on that in the next post) and more ideas to fiddle with in planning for the next few months of the project. An Eastern Oregon journalist called Luke, wanting to do a story on the Styra Project for a small newspaper. He consented, always eager to talk about the project at least if he doesn't have to do an on-camera interview. A link to Brianna Walker's story in the Valley Herald follows. Here is a photo of the article.

The Valley Herald
part 1

part 2

Much more continues to generate in Luke's brain and in his lab, often keeping him awake at night and springing ideas on him in the shower. Be ready for an upcoming post.

Putting it All Together

Today's post is all about modularity. I have had a number of questions about how someone using the Styra Project designs will be able to customize their own keyboards. This post will hopefully demonstrate how different button layouts can be achieved with LEGO bricks and a little bit of imagination. All of the plates shown in this post were designed and printed by Phill Thomas.          

                       

Figure 1:  Top and bottom of button plate

The modularity of the Styra keyboards stems from the button plate design. Each button plate has a flat top and a reinforced button hole through which we can snap in an inexpensive joystick button. The bottom of the plate has a LEGO compatible interface around the edges so that it can connect with standard LEGO bricks.

Figure 2:  Assembled plate with button

Each button has a 12" cable terminated in a two pin header that can easily connect any of the custom Styra controller boards discussed in previous posts. The relatively long cable allows for a range of placement options without the need to perform any soldering. It is also possible to use extensions to move a button farther than 12" from the controller.

When working with LEGO bricks, the variety of sizes and shapes allows the creation of things limited only by one's imagination. I decided that this same philosophy would provide the greatest flexibility for the button layouts. After a bit of cajoling on my part, Phill consented to design a wide array of button plates. There is a downside to this approach; the spacing between the buttons cannot be any smaller than two lego units. This means that a keyboard layout for n buttons cannot be as compact as it could be if we custom designed a plate for each layout. The benefit is that layouts can be designed on the fly without modifying the 3D model and printing a new plate for each change.

 Figure 3:  Complete set of button plates

Figure 3 shows a complete "kit" of button plates. The contents of a kit were somewhat arbitrary; I worked through a few different layouts to figure out the quantity of each plate type. The following plates are included:

• 1x1 button plate - 8
• 1x2 button plate - 4
• 1x3 button plate - 4
• 1x4 button plate - 4
• 2x2 button plate - 2
• 2x3 button plate - 2
• 2x4 button plate - 2
• 3x3 button plate - 1
• 3x4 button plate - 1
• 4x4 button plate - 1

Phill had the 3D printer working day and night for a couple weeks printing two complete kits. These have proven extremely useful in our prototypes.

Talking about plates and layouts is one thing. Seeing them in action is another. The following images were taken while I was experimenting with a new button layout:

Figure 4:  Plates with buttons

The first step in designing a new layout is to use your imagination. Think about what might make a good button configuration. Play with the blank button plates to see what looks good and what doesn't. Once the design looks good, select the appropriate plates and snap the buttons in.  

Figure 5:  First layer of LEGO bricks

The first LEGO layer is critical. Sound LEGO engineering practices are recommended; mainly, try to make sure to provide as much connectivity between the plates as you can. Larger bricks (2x4) are really good for this for several reasons but the biggest is that the 3D printed plates don't always snap in tight with the blocks. This is par for the course, but larger blocks seem to stick better where smaller once may fall off.

The second layer is just as important. When applying the second layer of bricks, position them so that the seams between do not link up with the seams from the previous layer. In other words, span the seams.

Figure 6:  Third layer of LEGO bricks with blue accents and red cable paths

The third layer should be applied the same as the second, spanning the seams. With these shallow buttons, if the tabs on the back are bent over (gently), then it is possible to stop after three layers of bricks. It is tight and there is not a lot of room to route wires, but it is doable. For style, I like to add a splash of color to one of the layers. In this case I used blue bricks in the third layer. This layer is also where I routed the button cables and used red bricks as caps.

Figure 8:  Fourth layer of LEGO bricks

Finally, add a fourth layer in the same way as layers two and three. At this point the framework is strong enough to be entirely self-supporting. For my prototyping work I prefer to have a LEGO base to work with, so I built several base plates. They consist of 16x32 LEGO pads glued to inexpensive bamboo cutting boards. Figure 9 shows the completed button layout.

Figure 9:  Completed 16 button controller layout

Just connect a controller and assign macros to each button and this keyboard is ready for action! 

Styra Navigation Keyboard Prototype

Now presenting the Styra Navigation Keyboard prototype. This keyboard is designed to allow someone to browse websites without use of a mouse.

Figure 1:  Styra Navigation Keyboard

This keyboard required several iterations before we had a working prototype. It all began with Phill making a list of the keyboard shortcuts he uses most frequently. Then he added a couple extra functions that he typically uses a mouse for. Armed with that information I built the first draft of this keyboard. Using the keyboard helped to refine the list further. Figure 1 shows the prototype I have been working with. And yes, that is dry-erase marker on the buttons. We still need to figure out the button labeling, but for now it works. The following is a list of the functions this keyboard can perform:

• cursor up
• cursor down
• previous page (back button)
• next field (tab)
• select (enter)
• previous field (shift tab)
• open bookmark manager
• add bookmark
• new browser tab
• close browser tab
• next tab
• previous tab
• reload page
• zoom in
• zoom reset
• cancel (esc)

After using the keyboard, we have identified several more functions that can easily be added to the navigation keyboard:

• username - automatically type out a preconfigured username
• email - automatically type out a preconfigured email address
• password - automatically enter a preconfigured password.  Not secure, I know, but very convenient.

Beyond the basic functionality shown above, a lot of work has gone into the internal software that makes this keyboard function. First is the keyboard library itself. Each button press invokes a macro that consists of two components. The first is the action—or key strokes—that will be sent to the computer. The second is the action type, or how the action will be performed. For instance, sending the action CTRL + ALT + DEL to the computer one key-press at a time doesn't do anything except possibly delete a character. However, pressing them all simultaneously triggers a reboot. The Styra keyboard library supports up to 28 characters in the action field and the following five action types:

• Click - simultaneously send all action characters to the computer in a single event.
• Double-click - simultaneously send all action characters to the computer twice in two separate events.
• Press - simultaneously press and hold all action characters until the button is released.
• Latch - simultaneously press and hold all action characters.  Buttons are released when the button is pressed a second time.
• Sequence - send each action character one at a time to the computer

The second library developed for this prototype is the controller library. This simplifies working with the custom 16 input modules and includes a routine to scan the i2c bus and automatically detect connected controllers (as long as they have unique addresses). This is the first step toward being able to modify the hardware configuration of the keyboard without recompiling the Arduino firmware.  This library handles polling the buttons and managing the button numbering to make it easier to assign configuration macros to a button.

The third library is an interface that defines the macro actions and action types and how the keyboard library can interact with them. The first implementation of this interface uses an eeprom chip to store the macro definitions. I'm really proud of this library because it was my first time using the SPI bus and my first time working with an eeprom. I am using the SPI bus because I had a bunch of these chips laying around from a kit I purchased several years ago that I never built. :)

Figure 2:  Case for the 16 input controller

The Navigation Keyboard Prototype also debuts the first draft of a case Phill is designing for the 16 input controller boards. The camera flash really accents the printing lines, but the case is a perfect fit with a lego compatible bottom and Roy's Styra logo embossed on the top!

Styra Joystick Prototype

The first fully functional prototype to roll off the production line is the Styra Joystick. The concept with this prototype is to provide a mouse alternative for pointer control. After several trial runs, I decided on the following criteria for this prototype:

1. The prototype needed to provide a perfect "click" and "double-click," regardless of how long a button was pressed. 
2. The prototype needed to provide a way to grab, aka click-and-drag, an item.
3. The prototype needed a way to adjust pointer speed on-the-fly, not through an OS control panel setting.
4. This prototype should not require any modification on the host PC to function.  Just plug it in and go. Since the design is based on the Arduino Leonardo, the requirement is easy.

Figure 1:  State machine for processing clicks

I worked through several different approaches for implementing the desired click functionality before I finally decided on a state machine. Figure one shows the three different flows for processing click, double-click, and latch functionality.

Figure 2:  Flex sensor

Figure 3:  Flex sensor in action

Mouse pointer movements for the Arduino are sent to the computer as X,Y position changes relative to the current pointer position. To implement on-the-fly pointer speed control, we read a value from a sensor connected to an analog input pin on the Arduino. The value from the sensor controls the speed of the pointer.

There are quite a few types of sensors to choose from, but the one that shows the most promise is a flex sensor, shown in figures 2 and 3. This sensor changes its value based upon how much it is bent.  Bend it a little, it returns a small value. Bend it a lot, much greater value. (That could be backward, but you get the point.) A sensor like this could be sewn into something attached to a wrist, ankle, foot, finger, etc. Just bending a finger or moving a foot could control the speed of the pointer.

We have two of these sensors to play with. If you have suggestions on how they might be used, leave some feedback. We're open to ideas.

Figure 4:  Styra Joystick Prototype

Figure 4 shows the completed Styra Joystick prototype. This prototype uses an Arduino Leonardo connected to one of the custom 16 input modules, though it only requires 8 inputs. The knob at the upper left corner of the prototype controls the pointer speed. The four buttons perform the following: left-click, double left-click, left-click hold/release, right-click.

The Styra Joystick requires a little getting used to, but I have used it for several hours to browse the Internet and it really isn't too bad.

We're still getting caught up on the blog posts.  More coming soon!

PCB Modules

Playing catch up again...  The PCBs arrived from Germany on August 21st, professional and practically perfect. Somehow the 'T' in Styra didn't show up, but the design for the electronics printed out just as it should. The boards were assembled and tested in short order and prototyping work is well under way. Before we jump into the prototyping work, I wanted to show off the beautiful printed circuit boards from Fritzing!

Figure 1: 32 input shield for Mini-PAC

Figure 2: 16 input module (version 2)

Figure 3: 16 input module (version 3)

Figure 4:  $250 worth of printed circuit boards

I ordered these boards through the Fritzing project from Germany. I paid a little more for shipping than I would have if I had ordered the boards from a fab in the US, but they were able to process the native Fritzing file format AND part of the proceeds goes to fund further development of the open source Fritzing development environment. I only ordered a total of eight boards, but they included an extra board of each design without any additional charge. The boards turned out beautifully.

Figure 5:  Assembled 16 input module (version 2) 

The fully assembled through-hole version of the 16 input PCB module shown in figure 5 was easy to put together and only required about 25 minutes each to assemble.

Figure 6:  Assembled 16 input module (version 3)

The fully assembled surface-mount version shown in figure 6 required nearly 45 minutes to assemble due to the amount of time I spent checking and double checking the solder connections. Both versions work exceptionally well and they both can be used interchangeably (and simultaneously) in keyboard prototypes. The four pin headers in the lower left of the PCBs allow up to 8 input modules to be daisy-chained together, providing up to 128 button inputs for a single keyboard.

The first batch of Styra 16 input modules is ready for integration into prototypes!  :)

Opening Doors

Tonight marks a first for the Styra Project: We ordered a batch of custom printed circuit boards (PCBs). For perfectionists like James and I, it is hard to make the call that something is "good enough." And since this was our first attempt at designing custom PCBs, we wanted to get it right.

For the keyboard controller prototypes, we're investigating two different approaches. The first approach repurposes a device designed for building arcade cabinets. The Mini-PAC is a fully integrated keyboard controller designed for attaching arcade buttons, joysticks and other devices to a computer. This device is compact and inexpensive. It supports a large number of buttons and is potentially a fantastic fit for our project. The downside is that it is closed-source and only produced by a single manufacturer in Europe. The PCB board shown in Figure 1 will plug onto the Mini-PAC as a "shield" and allow the connection of up to 32 buttons using standard two-pin headers.

Figure 1: 32 input shield for Mini-PAC

The second approach we're investigating for the keyboard controller is one that leverages open-source hardware and software, specifically the Arduino Leonardo. There are several advantages to this approach, specifically that it allows for a great deal of flexibility in sourcing components as well as customization. Unfortunately the Leonardo does not have sufficient I/O inputs, and while there are techniques that can be used to add more buttons (charlie plexing / scanning/ bit shifting), I have decided that for the Styra Project it makes sense to create input modules that attach to an I2C bus.

Due to the limited scalability of using software debouncing, I decided to add hardware debouncing to the modules as well. Figure 2 shows the design for a 16 input module with hardware debouncing using entirely through-hole components. This design requires more components on the board, but is easy to solder. The EDE2008 chips used in this design have a debounce interval of 25ms, which is more than enough for the buttons we are using.   

Figure 2: 16 input module (version 2)

Figure 3 shows a second 16 input module but with a couple of surface mount components. This design utilizes the MAX6818 chips, which have a 50ms debounce interval and may feel sluggish to people who push buttons lightning fast. The layout is cleaner but the surface mount components can be a pain to solder.

Figure 3: 16 input module (version 3)

The PCBs will be on their way soon, and we will only have a couple more orders to make to have all our hardware for the project.

Finally, I would like to give a shout out to Roy Kimball for his design of the new Styra Project logo. His design was able to capture what this project is all about: opening doors.  :) We got it just in time to get it onto the PCB boards.