Success! I got the latest PCBs over the holidays, and finally had a chance to assemble a couple of them. Using the new decoder chip instead of the i2c GPIO expander is way faster. Faster enough that for the first time in this project, I had to worry about putting in a debouncing delay. (Keyswitches like these “stutter” a bit when closing or opening, so if you’re fast enough at scanning them, you’ll get repeated presses.) The loop that scans through the rows/columns has a 1ms delay at the end, so I just added an additional delay any time the key state changed, and adjusted that until things seemed stable (around 10ms). At that point most of the keys were fine, but a few of them sporadically stuttered. I popped the keycaps off and cleaned the switch contacts with sandpaper, and now they’re fine. (One nice feature of the NMB/Hi-Tek “space invaders” keyswitches is that you can do this without disassembling everything.)

The PCB I made stacks onto the Adafruit Feather microcontroller board, and then attaches to the keyboard PCB. For testing purposes, I have all of these things socketed, but at that point the whole mess sticks up enough that the top of the keyboard frame won’t fit. I haven’t found detachable connectors that were low-profile enough to fit, so for actual deployment I’m soldering everything down pretty much flat. Here’s the Feather and adapter board soldered together, and stuck into a testing socket:

In fact, there are two general models of keyboard that I’m working with; that one is the more compact layout (the RT8756C+ and similar). It predates the existence of the Windows modifier keys. The other models are the RT8255C+ and RT8255CW+, the latter of which is the only one of these keyboards I’m aware of that includes the Windows key. These models have a larger, flatter case, and they have no space on top for the adapter. They do have a bit of space underneath, though. On the wrong side of the PCB. So, for those it’s a bit of a hack. Here’s the top side:

In place of the original 40-pin microcontroller, there’s just some header pins, pushed all the way down so they stick on the other side enough that I can solder them in place and then still stick a connector onto them, like so:

That’s the same Feather/adapter combo as before, but assembled upside-down, with a 40-pin socket soldered to the adapter so it can connect with the pins hanging down from the keyboard PCB:

I had to trim the socket to fit around the decoder chip, but it all fits, and works. Here’s what it looks like installed:

It’s a pretty minimal installation; just the adapter/Feather combo, a Li-Poly battery, and a microUSB extension leading to the outside, so it can be plugged in for charging and reprogramming.

Now that the keyboard is functional, I’ve been able to start going through the rest of the to-do list for features. First up was making the capslock light work. I could make the NumLock and Scroll Lock lights work, but nothing really uses them. One of the to-do items is to repurpose the Scroll Lock light as a battery indicator (flash slowly at 50%, quickly at 10%). I need to put in some ability to have a hotkey for “Consumer Control” keys like volume up/down and playback control. I don’t know how long this battery lasts with the current arrangement, but I may want to put in some sort of sleep mode. Unfortunately, one drawback of the current arrangement with the decoder chip is that there’s no way to select all columns at once so I can have the microcontroller go into sleep mode until any key at all gets hit and generates an interrupt. I may be able to rework the design to add in transistor logic to accomplish that, but it would undoubtedly be more complex. Worst-case, I can designate a special “wake key” that is the only one that would wake up the keyboard and just monitor that one.

I also need to come up with a good solution for filtering out ghosted keys; since keyboards like this work by connecting rows and columns of connections, if you press some arrangements of keys, they’ll short out and make it look like an additional key is pressed. If you press the keys in position A1 and A4, and then press the key in the C1 position, it will also appear that you pressed the C4 key; column C and row 4 are connected because C and 1 are connected, and 1 and 4 are both connected together through A. Newer keyboards that support effectively unlimited simultaneous keypresses (NKRO, or N-key rollover) work by having a diode inline with each keyswitch, preventing it from shorting out backwards like that. These keyboards don’t have any of that. My plan for this so far is to take advantage of the fact that when a ghosted key appears, it should always be two new keys being detected in the same scanning pass, which is unlikely to happen otherwise. So if it sees two new keys being pressed, it should just ignore them until the next key state change occurs.

Anyway, unless I think of a clever arrangement for supporting interrupt-driven wakeup, the hardware side might be done.

I realized that the last time I mentioned my home server, it was when I switched away from unRAID. I did so because I really wanted dual-drive redundancy, and more powerful options for running services other than basic file-sharing on it. That was some years ago, and I’ve since switched back to unRAID, because starting with version 6, it basically checks all the boxes. It now supports dual-drive parity, mirrored cache drive pools, and a very impressive collection of software add-ons made possible by docker. I’m very happy with it.

Case-wise, I think I started out using an old Antec P180 tower case, then moved things into the Norco 4U rackmount case mentioned in the older posts. It was dirt-cheap for rackmount gear, and it was just impossibly loud. You could easily hear it from another floor of my house. But it was cheap, and fit a lot of drives, and like a lot of hobbyists, all I really want is enterprise-grade hardware for cheapo prices.

After the Norco case, I moved things into the NZXT H2 tower case; I could fit 13 3.5″ hard drives into it along with a few SSDs, and it was nearly silent. I ended up making my own SATA power cables for my modular power supply to clean up the airflow inside. A nice case, but once I moved into a place where I could stash the server in the basement, I was ready for something bigger. I briefly expanded by picking up a dual external-port SAS controller card, and connecting it to a couple of 4x drive pods in an old multi-drive SCSI enclosure — you can get replacement plates for the back that swap out the old SCSI connectors with SAS, eSATA, USB, or whatever.

After doing some research online, I concluded that the cheap enterprise-grade solution I was looking for was used Supermicro gear. I really liked the look of the 933T chassis, a 3U server with 15 vertical hard drives lined up in a row. They’re all hot-swappable with a SATA backplane. (There are SCSI and SAS backplane options, but I believe the SAS version requires interposers for SATA drives, so whatever.) The 933T occasionally showed up on eBay for ~$150, often with an old Opteron-based motherboard. One nice feature of Supermicro’s hardware like this is that it’s all ATX standard, so you can swap it out for whatever else. The included power supply supported 3 redundant hotswap PSU modules, so that’s a plus.

It was pretty noisy, so I swapped out various fans (including some 40mm fans in the PSU modules, probably the noisiest of them). Now it’s nice and quiet.

Another nice feature of Supermicro stuff is that they sell a ~$20 board designed to convert an ATX server case into a JBOD drive enclosure; it hooks up to the ATX power connector and various case connectors (power & reset button, power LED, etc.) and lets you power on/off the device without a full motherboard in it. So I bought two 933T’s! One of them contains the motherboard (an Intel DZ77GA-70k) and SAS controller, the other contains the JBOD “motherboard” and a SAS expander hooked up to the backplanes. So now I have a quiet 6U server with room for 30 hotswappable drives.

I built a simple wooden rack using some 12U brackets to put everything in, and eventually added in a couple of 2U UPS’s and an old 1U console. At some point I’d like to replace the rack with something solid (with removable panels), but otherwise I’m thrilled with the current arrangement.

After discovering that the MCP23017 I2C-based GPIO expander was not going to be reliable, my first thought was to try out its SPI-based equivalent, the MCP23S17. In theory the same bug affecting the MCP23017 doesn’t affect the MCP23S17, and the communication overhead would be much lower because SPI is so much faster. I ordered some of these but haven’t experimented with them yet, because my other idea showed up first: the 74HC154, a 4-to-16 decoder IC, often used for memory address bus decoding. It has 4 input pins and 16 output pins; you set a 4-bit number (0-15) on the input pins, and it grounds the corresponding output pin while the rest remain high.

I really liked the idea of this chip not needing any communication protocol beyond 4 GPIO pins; it would eliminate the kind of problem I was having with the I2C-based expander, and should speed up the keyboard scanning loop. When I started working with it, I had two problems. The first was just an annoyance: the previous version of the adapter was cycling through the rows (via native GPIO), grounding one at a time (and leaving the rest floating), while reading in the 16 columns. This new method would be outputting the columns and reading in the rows, so I had to rework the somewhat-optimized code to all work in the opposite direction.

The other bump I encountered was more problematic. In the previous version, every time I switched which row was being scanned, I set the selected row to output low, and set all other rows to be floating inputs. Then the columns were set to input with pullups, so they’d register as low if they were connected to the selected row, and high otherwise. The reason the non-selected rows were set to floating inputs was because if you pressed two keys at once, it might connect two rows on the same column, and I didn’t want to short out a low and high output on the row pins. But the new decoder IC doesn’t leave non-selected outputs floating; they’re set as high. After testing, I confirmed what I feared, which was that the chip really did not like having those output pins shorted out.

The solution in this case was kind of brute-force: 16 diodes, one per output pin of the decoder, making sure no current could ever flow out of them. It could still flow in, so I could still detect when one of the rows was connected to the selected column. The breadboard prototype for this was messy with diodes, but it worked. (Side note: fancy new mechanical keyboard with NKRO (n-key rollover) have diodes at each keyswitch to prevent ghosting of keys, but these vintage keyboards do not. Ghost avoidance will be problem for future updates.)

Anyway, here’s what the PCB design for that looks like. Now I wait for the updated boards (and surface-mount diodes) to arrive.

(I have yet to spend the time to figure out how to make my silkscreen layer not be a mess in EAGLE, but the design should work.)

Things have been Eventful since the last post, so not a lot of time for tinkering. After discovering that the GPIO expanders I was using were too slow for handling all of the IO, I figured I had to do most of it via native GPIO pins on the microcontroller board, but I wanted a cleaner solution than I had before. So, I made this:

It’s a custom PCB with pins to interface directly with both the keyboard’s PCB (taking the place of the original 40-pin microcontroller) and the modern bluetooth controller (an Adafruit Feather nRF52832 in this case). The PCB layout for stacking onto a Feather (a “FeatherWing”) is a standard template that only had to be modified to remove some support holes to make room for the edges of the 40-pin connector.

This design uses the same I2C MCP32017 GPIO connector, and uses its 16 pins for connecting to the 16 columns of the keyboard PCB, and uses the Feather’s native GPIO pins for the 8 rows (and 3 LEDs). This way I can leave the 16 expander pins in input mode (with pullup), and cycle through the row pins on the expander for scanning the keys. Then I can read all 16 pins of the expander in one operation, to minimize the overhead. The 3 resistors on the board are for I2C pullups and the RESET line of the expander. The JP1 connector is an external connector for the ENABLE pin of the Feather board (which is also linked to the expander); this gives me an optional hookup for an external reset button if needed.

Here’s what it looks like with the Feather stacked on top, mounted on a keyboard:

Unfortunately there’s so little clearance in the keyboard case that I had to solder everything together directly; I haven’t found a removable header connector that is low-profile enough to fit. For prototyping I have one of them set up with everything socketed, but I have to use it with the top of the keyboard case removed. I really like the clean solution, though; the only wiring visible after this is the LiPo battery and a microUSB extender for charging/programming.

The only problem I had with it was that it would occasionally lock up, and require a reset. This only seemed to happen during typing, and occurred more often the more I typed. While trying to figure this out, I discovered that there was actually a design flaw in the original version of the NRF52832-based Feathers that caused them to use a lot more power than they should while in low-power mode (it was powering the USB->RS232 chip even while on battery power). I hadn’t done any power optimizations yet, but that made it kind of pointless. A newer revision of the board was available, but around that time Adafruit also released the successor to this, the NRF52840. It was faster, and had native USB support instead of requiring the RS232 converter chip. I didn’t intend to make the keyboard work as a native USB HID device, but I liked having the option. I ordered a few of those.

The other benefit of the newer NRF52840-based Feathers is that they had a preinstalled SWD debugging connector; the previous version just had the pads for it, and required you to surface-mount solder the connector. After finding that the newer Feather board periodically locked up just like the old ones, I picked up a SWD debugging interface to figure out what was going on.

Somewhat unhelpfully, the code was locking up in the same place every time, in the I2C communication library, where it waits for a response. The library doesn’t have any sort of timeouts for waiting, so it gets stuck forever. From what I’ve seen, that seems to be deliberate, as I2C devices can stretch their clock out (arbitrarily in some cases?) if they need more time to respond. Anyway, I didn’t get anywhere finding a solution to this; I considered just living with it and setting up a watchdog timer to reboot the Feather if it got stuck, but this was happening too often for that to be acceptable while typing.

I eventually found some threads online discussing a problem one user was having with the MCP32017 GPIO expander chip where he had a clock signal connected to pin 16, and the expander was occasionally giving bad responses when the level on pin 16 transitioned. In his tests, increasing the frequency of the clock signal increased how often he got corrupted responses from the chip. Other users replicated these effects, and the user reported that the manufacturer eventually confirmed that it was a bug in the chip itself, and would be addressed in a future errata publication. The official recommendation: don’t use pins 8 or 16 for input, or you will occasionally get corrupted responses.

I don’t know for sure if what I was experiencing was related to this bug, but it sure meant I was done trying to make this expander work, and ready for a new plan.

For this one I wanted to clean up the wiring to the keyboard PCB, and work towards making something that was a drop-in replacement for the original microcontroller. To this end, I bought tiny surface-mount versions of the IO expander chips, which were small enough to fit 2 on a board the size of the original microcontroller:

First surface-mount board. Looks nicer when not magnified (it’s 2″ long).

The 32 IO pins from the expanders were sufficient to connect to all of the rows and columns and LEDs. It has an 8-pin ribbon cable connector for power, ground, clock, data, and four more pins for interrupts from the expander chips. The keyboard firmware I was working with wasn’t interrupt-driven as of yet, but I wanted to have the option.

Up until this time, the pinouts I was using varied only slightly, so I had a few #defines in the code to switch between models. Since this one offloaded everything onto the expander chips, I reworked the code to be more abstracted and cleaned up.

Unfortunately, this one didn’t initially work. This turned out to be for two reasons. First, sometimes the expander chips would get into a state where the startup code wouldn’t get them working. Cycling power to everything would return them all to a known state, but that was going to be complicated once it had battery power. I considered adding a button to the bluetooth board’s Enable pin, so I could power-cycle everything even when it was closed up with a battery, but some experimentation showed the better answer was to tie the Reset pins for the bluetooth board and expanders together, so the expanders got reset every time the bluetooth board rebooted.

The other problem was that it was just too slow. The abstracted code was definitely a lot slower, but even after optimizing it, it wasn’t performing to what I needed. Hitting a key very rapidly would sometimes get missed by the matrix scanning. I had assumed the IO operations going through the expander chips would be slower than the built-in IO pins, but when I did some quick benchmarking, they appeared to be between 200 and 500 times slower. Using one expander chip for the columns was fine before, because I could leave them all as inputs, and read all 16 pins in one operation. But the rows were more problematic, because they had to be cycled through, and each time the pin had to be switched between input and output; the currently-selected row had to be set as output-low, and the non-selected rows had to be set as floating inputs, while the columns could all be set as inputs with pullups. I figured even if I set things up to be interrupt-driven, the rows would still need to be scanned through, and the tests indicated it would probably be too slow.

This was disappointing, because I liked the idea of just having the option of being able switch bluetooth microcontrollers with anything else that had i2c connectivity. If I was going to go back to just using a single expander chip for the columns, I’d have to come up with a different solution for cleaning up the wiring.

The obvious next step in cleaning up the wiring was to connect the IO expander chip directly to the keyboard PCB. Doing it with protoboard would have been a tight fit, so this was my first experience designing a custom PCB. I drew it up in EAGLE CAD, and sent if off to OSHPARK to be manufactured for ridiculously cheap. ($9 for 3, if I recall correctly).

The board held the IO expander chip, three resistors needed to make it work, and a 4-pin connector for the power/ground/clock/data lines to the bluetooth board. It cleaned up a lot of the wiring, but there was still the mess of the rows/LEDs running to the bluetooth board:

Better, but still yuck.

Once again it worked, but I knew I could do better.

There seem to be a few different models of NMB keyboards, but they mostly boil down to some that are more wedge-shaped but have thinner bezels, and some that are flatter but have a larger bezel. I’ve only ever found one model that was new enough to have Windows keys, and it’s one of the thin ones that takes up a lot of space.

I started by mapping out the pinout of the microcontroller; I think I even found some official documentation at some point. But I ended up with a list of which pins on the 40-pin chip were rows, which were columns, which were capslock/numlock/scrolllock LEDs, and which were power and ground. The rest didn’t really matter.

The NMB keyboards have a matrix with 8 rows and 16 columns. Unfortunately, the bluetooth microcontrollers available didn’t have enough IO pins to connect directly to all of these. My solution was to pick up some IO expander chips (the MCP23017). It’s connected via i2c, so all it needs from the microcontroller is four pins (power, ground, clock, and data). It provides 16 IO pins.

The bluetooth microcontroller I picked was the (then relatively new) NRF52 Feather board from Adafruit. The first version of the board had a bug that prevented it from going into very low power mode (because it was powering the USB->serial chip off of the battery), but it was still a nice board.

For the first version of this, I put one of the expander chips on some prototype board, connected it to the bluetooth microcontroller, and then connected the expander chip to the columns, and the bluetooth board to the rows and LEDs.

IO Expander on protoboard.
Battery, bluetooth controller, USB connector (charging/programming), and expander board.
16 wires from the expander lead to the column pins.
The rows and LEDs were connected from the back of the board, leading directly to the bluetooth board.

This worked, but I really wasn’t happy with the mess of wires on both sides of the board. I was determined to follow it up with a version that cleaned up or eliminated a lot of the wiring.

I’m picky about my keyboards. I like them wireless, I like a full layout, and I like good mechanical switches. This is a hard combination to find for sale, as most mechanical keyboards nowadays are marketed towards gamers, and they don’t like the potential latency of wireless keyboards. Even the hobbyist DIY keyboard resources typically revolve around minimalist layouts.

So fine, I’ll make something. This is probably for the best anyway, since my favorite keyswitches are no longer being made. My favorites were the original Apple Extended Keyboard (using undamped ALPS switches), and the lesser-known line of keyboards using the NMB “space invaders” switches. My college roommate had one of those NMB keyboards with his 80286 PC, and I kept that keyboard to this day; a few years ago I hardwired a PS2-to-USB adapter inside it.

I wanted wireless, though, and there were a few different ways of achieving this. The first pass was very hacky. I picked up a cheap $20 logitech wireless keyboard that had a full layout, and mapped out the matrix that its keys used. (Keyboards work by having a grid of rows and columns, and every time you press a key it connects that row and that column. A microcontroller scans across them, looking for connections. The layout of rows/columns to keys varies widely across models, based on PCB layout and other factors.) At this point I could have designed a new PCB for a keyboard that used the same matrix, and attached it to the controller board from the logitech keyboard, but PCB manufacturers charge by the square inch, and these keyboards are large.

Instead, I took my test Apple keyboard, and removed its PCB entirely, and hand-wired connections between keys to match the Logitech matrix. Lest things get too crowded between keys, I used thin magnet wire to connect them. It was not pretty on the back. At this point the only thing holding the keys in was the fact that some (especially early) mechanical keyboards have the keys mounted through a metal plate before being connected to the PCB.

I’m not proud of this, but it worked.

Well, it worked, but the metal plate didn’t provide enough stability; I had to pop a keycap off to fix a flaky keyswitch, and it ended up pulling the whole keyswitch out of the plate, ripping the thin magnet wires out of the back in the process.

If I were to try this approach again, I’d grab a bunch of the enabler boards and connect them with hookup wire or something. But I wasn’t going to try this approach again; for subsequent versions, I wanted to replace the original keyboard’s microcontroller with something that would interface with a modern bluetooth-enabled microcontroller. I had all the information I needed about the Apple keyboard from this site, but those are more expensive for tinkering, so I focused on the NMB models. (Quick note, though: if you like the “clacky” feel of the first Apple Extended Keyboard as opposed to the softened sound and feel of the AEK2, you can take the AEK2, open up each keyswitch, and remove a couple of rubber dampers from each one, and reassemble.)

I’d meant to post about this a while ago, but lost the relevant link until now.

When I was in college, I wrote a tiny DOS Tetris clone called Blocks from Hell. I was an avid player of the game, and there were already many freeware and commercial clones around, but I was frustrated that they generally couldn’t keep up with really fast playing, and many of them seemed like they tried to change up aspects of the game, always to their detriment. My goal in writing my version was to make one that handled championship-level performance, and was as vanilla-standard as possible in its implementation of the game rules.

Well, the problem I ran into was that it wasn’t clear what “standard” meant regarding the rules of the game. I got my hands on every version and variant I could find (in 1989 or so), and they were kind of all over the map in their gameplay. Some things were mostly agreed-upon (like the size of the gameplay area and shapes of the blocks), but things like scoring and level advancement were clearly not. (For example, some of them gave a fixed score per piece played which didn’t reward the player for playing it early, while some of them did crazy things like have a maximum number of points that you could get for playing a piece, but subtracted from that every time the piece was moved or rotated, making it easy for an indecisive player to get no points.) Even things like how the pieces rotated varied among them.

Lacking an authoritative model, I set about trying all of them, and taking notes as to how they handled all of these game aspects. This meant playing random freeware Tetris games while mostly paying attention to the score, to reverse-engineer how some of them worked if they weren’t documented in that much detail. Armed with the information of the dozen-plus versions, I picked the aspects that seemed like the best-fit, or that felt like they made the most sense gameplay-wise. (In the case of the scoring, I went with a small fixed number of points per piece played, plus a bonus based on height from which the piece was dropped immediately down.) I was very pleased with the outcome, and it’s held up well enough that it’s still being played by some enthusiasts, 24 years later.

Well, a few years ago I ran across this article by Colin Fahey, which (along with a great history of the game) attempts to nail down an official set of gameplay rules for “Standard Tetris”, in part so that it could be more usefully used as an artificial intelligence arena. For this, Colin goes to the purest source — the pre-commercial DOS version of Tetris written by Alexey Pajitnov and Vadim Gerasimov in 1986, which I unfortunately never had the chance to test.

It turns out that I could have saved a lot of time if I had seen it, because it tracks almost perfectly with the choices that I used for Blocks from Hell. The scoring differs mostly because the original version didn’t reward the player for clearing lines(!), and the level advancement and speed control are basically identical, except the original starts at the equivalent of level 10 on Blocks.

Overall, I’m very pleased with how close to “pure” Tetris my efforts turned out to be, and I’m seriously impressed at how well-tuned the initial version by Pajitnov and Gerasimov was. Well, except for the lack of a line-clearing bonus. I just can’t get behind that.

While migrating data from my ReiserFS-formatted disks over to ext4 volumes, I ran into a weird issue with a Seagate drive. It’s a Barracuda 7200.14, model ST3000DM001, with the latest firmware. It’s been running fine, and I just copied all of its data off with no problems. Copying new data onto it, though, a short bit into the transfer it slows way down, to below 1MB/s, and eventually drops off of the SATA link entirely. Upon a reboot, it’s all back, and SMART diagnostics show no errors ever detected by the drive. Doing a diagnostic test on the drive shows nothing wrong. Reading the data works fine. I’ve tried the drive in 3 different drive controllers so far, disabled Native Command Queuing (NCQ), replaced cables, no difference. At this point I can just power up the system (which contains multiple drives of the same model that don’t exhibit this problem), and start writing information to that drive without ever reading it, and it starts to slow down within 30 seconds. It drops offline a few minutes later. When I turned off NCQ, it didn’t drop offline during the time I tested it, but it did slow way down, then speed back up, then slow way down again, repeatedly.

It’s not just that this is not how drives are supposed to behave. This isn’t how drives are supposed to fail, either. If there’s a defect on the media, it’s detected when the drive tries to read that section, then reported as a failure and put on a list of sectors pending relocation to a spare area on the disk. The relocation doesn’t happen until that section is overwritten, because the drive then knows that it’s safe to give up on ever reading the old data. None of this explains the behavior of reading being fine, and writing hosing everything without logging a problem on the drive.

I’ve seen 2 or 3 posts online from people clearly describing the exact same problem with this model of drive, but never with a solution; the thread either never went anywhere, or the poster RMA’d the drive. Mine isn’t under warranty according to Seagate’s web page.

At this point, the easy options seem to be exhausted. The next things I can think of to try are:

  • Downgrade the firmware to an older version, if it will let me.
  • Connect a TTL RS232 adapter to the diagnostic port on the drive’s board and see what it says during powerup, and during failure. I haven’t delved into Seagate’s diagnostic commands before, so maybe there’s something there to help.
  • Pull out my new hot air rework station, swap the drive’s BIOS chip with a spare board from a head-crashed drive, and see if that’s any better.

I am vexed by this drive.