sasheldon.com

Designing my own keyboard alpha layout for comfy inrolls

2025-12-29T00:00:00-06:00

Last year, the discomfort in my arms after using the computer got too bad to continue ignoring, and I ordered my first ergonomic keyboard, the MoErgo Glove80. This was my first time using a columnar keyboard, and my first programmable keyboard. I expected to make some layout customizations; obviously the layout on this new keyboard couldn’t match the keyboards I was used to. But I didn’t expect how, when you’re using a totally new form factor and can customize every part of the layout, there’s really no reason not to customize deeply!

Ultimately this led me to designing my own alpha layout with inspiration from other alternative layouts. I switched to it at the start of the year and have been enjoying typing with it! I prioritized finger utilization I found comfortable and inward rolling:

x c l d k  z y o u q
r s t h f  p n e i a
w v g m j  ↵ b ' . ,

Background
Learning
Basics of layout design
My design goals
Design process
- Vowels
- Consonants
- Punctuation
- Enter?
Metrics
Thoughts from a year of usage
Why I wrote this
Resources and additional reading

Background

I originally intended to keep using QWERTY on my new keyboard. Surely my existing muscle memory would be far more valuable than any minor improvements to be gained from a new alpha layout. Until I started using a columnar keyboard for the first time, with fingers strictly assigned to keys, and realized: my typing previously on QWERTY was nothing like this, and using QWERTY this way kind of sucked.

Using QWERTY with strictly assigned fingers raised a lot of questions. Why did I so often have to type multiple letters in a row with the same finger? Why did I so frequently have to reach with my index fingers for t, n, and h? Why was I resting my right hand on these home keys if they are so rarely used? Why did I have to reach my right pinky to the top row for p when its home position was the rarely used ;?

I realized that I had been avoiding a lot of these issues by not strictly using fingers for certain keys. I learned that it’s common among QWERTY typists to work around using the same finger repeatedly by using different fingers to hit a key, a technique called “alt fingering”. Even beyond this, my QWERTY typing had idiosyncrasies: I type the entirely with my left hand (middle-index-ring), I barely use my right ring finger and never use my right pinky, and I even hit the space bar with my right index finger.

This meant that, for me, typing QWERTY on columnar was basically learning an entirely new layout. I was typing really slowly and would need a lot of practice, and I would be putting in all this effort to re-learn to type with a layout that didn’t even perform well with strictly assigned fingers.

Learning

Before we get into this process of designing a keyboard layout, if you haven’t learned to use an alternative layout before, you may be thinking that switching is prohibitively difficult. The process of learning a new layout wasn’t as bad as I expected!

When I got my first split columnar keyboard, I re-learned how to type three times over the following months. I started out using columnar QWERTY for ~1 month, then switched to an alternative layout designed to be similar to QWERTY for ~3 months, then finally switched to the custom layout described in this post. Learning for the third time seemed easier than the previous times. I started practicing during a lazy holiday week off work and then went back to work using it (clumsily). After a little over a month I started hitting 70 wpm in typing tests, and now after a year I’ve hit 90 wpm in tests.

To learn, I started out using keybr, which gradually adds additional letters for you to learn. Once I had unlocked all letters there, I switched to Monkeytype, which offers more options and has a fun collection of quotes to practice with.

The process of learning a new layout is frustrating at times, like when I wanted to quickly type out an idea but was still too clumsy to do so. But I also found it fun, and, strangely, I kind of miss it. Typing practice was meditative for me, a good way to clear my head, and it reminded me of practicing a musical instrument.

I’ve only use alternative keyboard layouts on my split columnar keyboards; I continue to use QWERTY on my laptop and any other unibody row-staggered keyboards. I experienced a bit of confusion during the learning process, but that cleared up as my muscle memory solidified, so now I can easily switch between using the different kinds of keyboards. Having such different form factors between the two seems to help maintain separate muscle memory.

I believe I still type faster on QWERTY than on my custom layout (~100 wpm), but I don’t regularly take QWERTY typing tests to know for sure. This is a little demoralizing after so much practice, but it makes sense given the many years I’ve been typing QWERTY. And I remind myself that the switch wasn’t about speed; I may type faster on my laptop keyboard, but after using it for a while I can feel the familiar forearm and wrist aches returning.

Basics of layout design

When I started researching alternative keyboard layouts, I thought I’d just find the best, most optimal one and pick that. Well, it turns out that’s not so easy to define. There’s a passionate community using metrics to rigorously analyze layouts, but there’s no obvious way to define a metric for how “good” a keyboard layout is. Instead, we usually look for patterns that feel comfortable or uncomfortable, and then try to formally define them and measure how frequently they occur. Each layout optimizes for a different balance of these metrics.

Ec0’s Keyboard layouts doc is the the best source for understanding how modern layouts are designed and the metrics used. One of the basic metrics is the frequency of “same finger bigrams” (SFBs) which measures how often the same finger has to type two keys in a row. There are metrics to measure how often fingers have to stretch apart vertically (scissors) or horizontally (lateral stretch bigrams, LSBs). More advanced metrics look at trigrams, 3-key sequences, to evaluate how often typing switches between hands (alternation), or uses the same hand flowing in a consistent direction (rolling), or uses the same hand but with awkward changes in direction (redirects). Some analyzers assign effort values to key positions to quantify how difficult typing common words feels.

All of these metrics make layout design a fascinating puzzle, where it’s impossible to maximize every metric at once, and the layout designer must decide what trade-offs to make to maximize metrics for their preferred patterns.

My design goals

When I first started using my new keyboard, I quickly found that I was not accustomed to typing with all 10 fingers, and some felt strained from the adjustment. I found that I had used my middle fingers for many keys previously, but now they were down to only 3 letters. Maybe I have short pinkies or suboptimal hand positioning, but I found the top row pinky keys required a stretch that was not comfortable for me to perform frequently.

These observations led to comfortable finger utilization being my primary goal. I wanted my middle fingers to have the most utilization, followed by index, ring, and pinky, and I wanted to only have low frequency keys for the top row pinky keys.

In hindsight, I’m not sure utilization was exactly the right metric. I now think that finger movement (moving between keys) is more impactful than utilization; I feel less strain on a finger that mostly presses a single key than a finger that regularly switches between multiple, even if the single key is frequently used. Fortunately, utilization does correlate somewhat with movement, so I ended up in roughly the right place. Over time I have found that my weaker fingers have grown more comfortable typing, so I think I could have adapted to a layout that used them more, but perhaps it was good to not push my luck.

After finger utilization, I decided I wanted to prioritize inward rolling. When I first read about inward rolling (from pinky to index) vs outward rolling (from index to pinky), I was surprised to compare how much clumsier it felt to roll my fingers outwards. I’m not sure how much this actually matters in typing, so I didn’t zealously avoid all outward rolling, but it was at least a fun little challenge to optimize around.

Although I wanted to prioritize inward rolling, I didn’t want redirects to become too frequent. If some rolls came with too many redirects, I would prefer having alternation instead.

Like all modern layouts, I wanted to minimize same finger bigrams, but I found I wasn’t so sensitive to them that I’d prioritize this over my other goals. I was hoping for SFBs a bit better than Colemak, but I knew I wouldn’t beat the SFB rates of the newest layouts (especially ones with thumb alphas or magic keys). I didn’t even consider same finger skipgrams; just avoiding SFBs already felt luxurious enough coming from QWERTY.

In the end, my priority ranking came to:

Finger utilization
High in-rolls
Low redirects
Low same finger bigrams

Design process

With these goals established, how do we design a layout optimized for them?

First, we should quantify the finger utilization targets. I chose:

Index: 12-14%
Middle: 16-18%
Ring: 10-12%
Pinky: 8-9%

The foundation of a layout is its placement of the 9 most common letters. Modern layouts place all vowels on the same hand (in order to reduce redirects), with the 4 most common vowels on three fingers (in order to leave more room for placing the many consonants). Let’s start designing the layout as recommended with the vowel hand!

Vowels

Laying out the vowel hand starts with choosing the vowel block.

The vowel block with highest inrolls is yi oe ua. The oe stack would have utilization far past the target for ring, so this would require ua on index. This means we’d have to push more consonants off the index finger to keep SFBs low, and ultimately this leads to violating the finger utilization goal, like adding more letters onto pinky; therefore, let’s only consider vowel blocks that do not include index.

The next best vowel block for inrolls is yi oa ue, and while oa has less utilization than oe, that’s still a lot for ring, especially with a very common letter off the home row. i ue oa is next; this has the ring finger moving off home row far less often, so it could be viable, but it still already means over 14% utilization for ring, which is higher than the target.

This leads us to a ui oe. Although it doesn’t have the highest inrolls, its utilization is closer to what we’re looking for. oe alone is over 19% utilization, but as long as we don’t add anything else common to the column, it’s not too far past the middle finger utilization target’s 18% upper bound.

With the vowel block chosen, we can begin laying out consonants on the vowel hand. High-inroll layouts (usually) have either n on the vowel hand index or h on the vowel hand pinky, so n index it is! Let’s not stack another common consonant like r on the index, because that would exceed the utilization target and limit our options for the remaining 4 keys.

I paired y with n, even though that leads to some moderate SFBs, because then we have a you three-roll, and that just feels so dang nice (even as an outroll). We’ll fill out the rest of this index later when laying out the consonants.

For a low-utilization key for top vowel pinky, I chose q. This does mean a redirect for equ, an SFB for aqu, and an SFS for qua, but the guaranteed qu inroll is nice, and quo feels great and que feels pretty good.

At this point, we’ve laid out most of the vowel hand:

☐ ☐ ☐ ☐ ☐  ☐ y o u q
☐ ☐ ☐ ☐ ☐  ☐ n e i a
☐ ☐ ☐ ☐ ☐  ☐ ☐ ☐ ☐ ☐

Consonants

With the vowel hand mostly established, and only 4 consonants remaining from the 9 most common letters, it’s time to start on the consonant hand. By this point we can see that this layout will be a member of the “N + vowels” family of inward rotation layouts in Ec0’s Keyboard layouts doc.

Let’s use the remaining 4 most common consonants for the home row of the consonant hand; the arrangement with the highest inrolls is rsth. This is perfect, because it puts the most frequent consonant, t, on middle finger, which we’ll need in order to have higher utilization on the middle finger, in just 3 keys, than the index finger, with its 6 keys.

With this home row, c must stack with s on ring finger to minimize SFBs. Let’s place l, the highest frequency consonant that we haven’t already given a home row spot, on middle finger. tl does mean some SFBs, but this gives us more flexibility with what letters can be placed on the index. d then goes on the index, and by placing these in the top row we now have nice adjacent same-row inrolls for ld and cl.

For the consonant pinky’s rare top key, I placed x to have xc and xt inrolls, plus a fun three-roll for xcl in exclude.

x c l d ☐  ☐ y o u q
r s t h ☐  ☐ n e i a
☐ ☐ ☐ ☐ ☐  ☐ ☐ ☐ ☐ ☐

From here we should decide how to finish the consonant pinky. xr is still well below the 8-9% utilization target. w or v are the natural choices to reduce SFBs. I chose w to get closer to the 8% utilization goal. Fortunately a wc pair almost never appears, so we avoid an uncomfortable scissor, and the wl and wd pairs are infrequent, so we don’t have many row skips. I find the sw, tw, and wh pairs to be comfortable.

There’s another nice aspect of the rw pinky: I especially dislike redirects where the pinky is the finger upon which direction changes, and thankfully r and w don’t often appear in the middle of consonant clusters. They appear at the beginning (rst, rts, wds) or at the end (str, thr, sw), but not in the middle (srt, trs, dws).

g shouldn’t go with n or h, so it should go on ring or middle. Putting g on middle doesn’t maximize middle finger utilization (m would), but it results in fewer SFBs than ring (gl vs gs), so let’s settle for middle. The ltg column is a little below the middle finger utilization target, but it’s close enough. We can then finish off ring as csv to keep its SFBs low, since it already has the sc pair to deal with.

With m and pb remaining, we should put them on separate indexes. by has lower SFBs than my (plus ph), so pb goes to the vowel hand index and m to the consonant hand index. Of p and b, p pairs more frequently with o, while b mostly pairs with e, so let’s place p in the inner column and b in the bottom row for fewer scissors (despite slightly more lateral stretches from po).

x c l d ☐  ☐ y o u q
r s t h ☐  p n e i a
w v g m ☐  ☐ b ☐ ☐ ☐

At this point, we have 5 spots left on the index fingers, 4 of which are the inner column top/bottom row corner positions that are more than 1u from the index finger’s home position. Let’s assign f, the last letter with > 1% utilization, to the consonant hand index so that it need not be in one of these corner positions. f does not cause SFBs here, and although it could lead to higher inrolls on the vowel hand index, at least on the consonant hand it mostly contributes to alternation rather than outrolls. I placed f in the inner column and m in the bottom row since f has less utilization and this avoids fl scissors.

k, the last letter besides the rare ones, can be assigned to the consonant index to avoid nk SFBs. Putting it in the top row means less of a stretch for ck and lk.

At this point we just have j and z to fill in the gaps. I put j on the consonant hand so that j down and k up vim commands in the terminal are in a convenient place. z gets a vowel hand index spot, and now we’ve placed all the letters!

x c l d k  z y o u q
r s t h f  p n e i a
w v g m j  ☐ b ☐ ☐ ☐

Punctuation

We still have some open spots on the vowel hand to place punctuation. The most frequent . and , deserve a spot on the base layer, but on middle finger they would cause higher SFBs with e and raise utilization even higher, so let’s place them on ring and pinky instead.

I chose ring for . because, in contrast to ,, it is tapped repeatedly for ..., and is often interleaved with letters for programming, acronyms, and domains. Putting it on ring also gives us the freedom to put / on the pinky outer column without creating an SFB for ./ paths. This does swap the order of ,. and the shifted <> from QWERTY, but I’m not sure that matters when we’re deviating so much from QWERTY and when angle brackets are often better positioned on a symbol layer.

' could go on the pinky outer column without causing many SFBs, but I instead chose middle finger. This does raise middle finger utilization somewhat higher, but I prefer that to increased pinky lateral movement. This also means that ' is still available on the base layer of 3x5 keyboards.

Placing ' on middle causes an e' 1u SFB on contractions like he's or we're, and an o_' 2u SFS on contractions like don't or you're, but I don’t find these so unpleasant. Surprisingly, the layout analyzers say the overall SFBs are the same as / or ; on middle and ' on pinky, and I’d rather handle SFBs with the more dexterous middle finger.

x c l d k  z y o u q
r s t h f  p n e i a
w v g m j  ☐ b ' . ,

Enter?

You may notice this leaves one spot unaccounted for on the vowel hand index. Normally a layout would include one more punctuation key, usually either ; or /, but I use these infrequently enough that they seem like a waste of a base layer key for the strong index finger. Yet this position isn’t ideal for keys that would regularly be used in the middle of typing prose: it’s more than 1u from the index home position, requires a lateral stretch, and can cause scissors given this layout’s top row bias on right hand. Can we find a key that’s used regularly, but not in the middle of typing prose?

The whitespace/control keys seemed like they might be a nice fit. Placing one of them here could allow for lower pinky and thumb movement.

Space would obviously be a bad choice, since it’s used constantly when typing prose.
Backspace is an option I thought might be good, but after trying it, I found that I use it too frequently in the middle of typing prose to quickly correct mistakes. (I put backspace on a thumb home key instead.)
Delete and Escape could be fine options, but I don’t use them that regularly.
Enter and Tab seem like good options.

I use Enter more regularly than Tab, so I chose it.

With that, we’ve reached the completed layout as shown at the start of this post!

x c l d k  z y o u q
r s t h f  p n e i a
w v g m j  ↵ b ' . ,

Metrics

But enough of me justifying these choices; the best way to see if this layout succeeded at its goals is to turn to an analyzer and look at the metrics!

My favorite analyzer for evaluating and experimenting with layouts is Cyanophage’s, and you can see its analysis of this layout here. Some other great interactive playground analyzers are Oxey’s and Layouts Wiki’s, though these don’t support direct linking to custom layouts.

A caveat: these analyzers do not have data on usage of the Enter key, so for this analysis I have replaced it with ;. This means the usage of the right index will be undercounted, but hopefully it does not have a large impact on bigram/trigram analysis.

We can benchmark by comparing metrics against other layouts. I’ll compare with:

Colemak-DH: a version of Colemak, slightly modified and generally considered better for columnar keyboards. Colemak is one of the most popular alternative layouts and is designed to be similar to QWERTY.
APTv3: a well known inroll-focused layout from which I took inspiration.
Gallium: a well-rounded layout often recommended as an upgrade from Colemak.
QWERTY: mostly just for laughs.

Finger utilization

The reported finger utilization is roughly in line with the targets:

	Target	Left	Right
Pinky	8-9%	8.01%	9.23%
Ring	10-12%	10.58%	11.00%
Middle	16-18%	15.12%	19.88%
Index	12-14%	13.66%	12.52%

Left middle is a little under, right middle is over, and right index may actually be over when the true utilization of Enter is considered.

Like mentioned earlier, finger movement is another useful measure of the work performed by each finger:

	Left	Right
Pinky	9.95	6.05
Ring	21.07	18.87
Middle	28.56	39.99
Index	41.43	28.33

For the most part, the movement of the fingers look good, but it shows some interesting differences from finger utilization. Although left pinky has lower utilization than right pinky, it has more movement. And similarly, although left index has lower utilization than left middle, it has significantly higher movement. There may be more work required of these fingers than I hoped.

In-rolls

	2roll in	2roll out	3roll in	3roll out	Alt
This layout	30.07%	14.62%	1.33%	1.07%	33.45%
Gallium	21.35%	23.06%	0.45%	1.21%	37.55%
APTv3	32.63%	14.12%	2.26%	0.54%	30.73%
Colemak-DH	25.18%	21.54%	1.50%	0.98%	25.43%
QWERTY	20.38%	21.38%	1.32%	1.48%	21.38%

The trigram metrics show we did a pretty good job of prioritizing inrolls! Bigram inrolls are nearly twice as common as bigram outrolls and are noticeably higher than Gallium and Colemak-DH. We get close to APTv3, but cannot match it because of our focus on finger utilization, falling behind especially on inward 3-rolls. We can see that alternation also falls between Gallium and APTv3, which makes sense due to the inverse relationship between alternation and rolling.

Redirects

	Redirect	Weak Redirect
This layout	2.53%	0.30%
Gallium	1.93%	0.26%
APTv3	3.60%	0.42%
Colemak-DH	5.33%	1.09%
QWERTY	6.22%	0.44%

Similarly, redirects are situated between Gallium and APTv3, but significantly better than Colemak-DH. We won’t be able to beat Gallium on redirects because we’re prioritizing inrolls, and rolling correlates with redirects.

Additionally, I’m happy to see only +0.04% weak redirects over Gallium. Cyanophage classifies redirects that occur on the middle, ring, and pinky (i.e. without the index) as weak redirects, the redirects that feel clumsiest.

Same finger usage

	SFBs	SFSs
This layout	0.84%	3.50%
Gallium	0.64%	2.74%
APTv3	0.81%	3.09%
Colemak-DH	0.91%	4.24%
QWERTY	4.38%	5.45%

Same finger bigrams look acceptable. As expected, we do not beat Gallium, though we get close to APTv3 and do beat Colemak-DH. Index/middle SFBs are 0.58%, so most are on capable fingers.

Since same finger skipgrams were not a design priority, it’s understandable that they lag behind Gallium and APTv3, though it’s good to see that they do still beat Colemak-DH. A considerable portion of SFSs come from the oe stack.

Scissors and lateral stretches

	Scissors	LSBs
This layout	0.31%	0.98%
Gallium	0.95%	0.96%
APTv3	0.11%	0.33%
Colemak-DH	0.15%	1.27%
QWERTY	1.46%	4.55%

Although scissors and lateral stretches weren’t a focus, we did consider them during design, so it’s worth checking on them.

Fortunately, the numbers seem reasonable! Scissors are lower than Gallium, with most of the scissors coming from bo (like Gallium’s of). Lateral stretch bigrams are comparable to Gallium and lower than Colemak-DH.

Thoughts from a year of usage

A layout can’t be completely evaluated from its design justification or analyzer metrics; the real test is how it feels in real usage. I’ve been using this layout for a year now, and I’ve been happy with it!

It has largely succeeded at my goal of comfort. I no longer feel the strain on my index fingers that I started to feel from using columnar QWERTY, though perhaps that would have faded in time regardless. The high middle finger utilization has not caused me any discomfort.

Unfortunately, the layout did have one noticeable point of discomfort: the wr pinky. It seems that my left pinky has been messed up by a lifetime of using only left shift on QWERTY, including weird contortions like Shift+1 and long stretches like Shift+Y with my left index. I started experiencing discomfort in my left pinky on the new layout, though fortunately I was able to mitigate it by adjusting my navigation and symbol layers for less pinky usage. It has lessened over time and now I notice it only rarely if I’ve been typing quickly for long periods of time.

In hindsight, I wonder if finger movement would have been a better design goal than utilization. For a while I considered swapping to a vr pinky, which would reduce usage, but it comes with an increase in pinky SFBs and SFSs. I considered something like an xrz pinky, but I was worried it’d lead to index discomfort, and I never found a solution I liked before the pinky discomfort started improving. I’m not sure if the more common bn pinky would have fared better for me; I can only imagine how bad I would have felt with Dvorak’s ls pinky.

The relatively high SFBs of the left middle and right index haven’t bothered me. I don’t notice the by 2u SFB, which I had expected to be annoying. ltg is a bit busy, but nothing compared to QWERTY’s edc; typing digital for the first time was a funny experience, though. The ght SFS occasionally trips me up during fast typing, so maybe g on ring could have helped with that.

One pleasant realization I had: of the 5 most repeated letters, this layout puts 4 (all but s) on the middle fingers, which I find to be the most comfortable finger to tap in quick succession. Nice!

The inroll focus doesn’t feel as notable as I first imagined. I’m excited whenever I get to type rst or ien, but for the most part the inrolls fade into the background. I can see why most layouts these days aren’t designed for directionality, especially with the wide variance of the English language fighting against you.

I’ve liked having Enter on an index finger. It feels nice for sending quick messages and has helped me balance the load on my thumbs. It doesn’t feel great if you have do repeated patterns of single letter + Enter, like with yes/no CLI prompts… which are especially annoying because y and n are on the same finger, so I usually alt-finger this case.

I don’t think I’ll be hopping to a new layout any time soon; any improvement it provides would have to be dramatic. I’ve considered layouts with a thumb letter, but I’m concerned about thumb strain if I have backspace and a letter on the same thumb. Maybe someday a magic key will lure me into learning a new layout, but not yet.

Why I wrote this

Designing this layout was a fun project and I wanted to share about the process! I don’t think I’ve designed the world’s best layout, and I don’t think that everyone should use it; rather, I hope more people explore alternate layouts, tweak existing ones to their preferences, and design their own.

If you’re switching to an ergonomic keyboard, it’s a great time to try an alternative. Every layout is made with certain preferences, and if you can’t find one that matches yours, it’s not so hard to tweak an existing layout or design your own.

Resources and additional reading

If you want to learn more about alternative layouts:

Pascal Getreuer’s guide to alt keyboard layouts - a great introduction that overviews many alternative layouts
Layouts Wiki’s guides - includes recommendations for the latest layouts with reasons you may (or may not) prefer them
Ec0’s Keyboard layouts doc - already linked many times in this post, but worth repeating; an in-depth resource for understanding and creating layouts

And, in addition to the layouts linked earlier for benchmarking, here are some layouts with neat ideas and writeups that I’ve enjoyed reading:

Thraeg’s Mumak - prioritizes middle finger utilization and adjacent finger rolls
Acas’s Vylet - prioritizes inrolls with an added magic key
Valorance’s Night - achieves very low SFBs and SFSs with a thumb letter, plus its writeup includes a nice overview of other layouts with thumb letters
Simon Zeng’s Afterburner - a really neat idea for multiple magic keys to dramatically reduce same finger usage

How I Broke Rust's Package Manager for All Windows Users

2017-05-07T12:07:40-05:00

Last weekend I was playing around with a way to represent null-terminated UTF8 strings in Rust. Rather than just toying with it forever, I decided to clean up a minimal version and publish it to crates.io. Creating the crate went smoothly, exactly the same as the 11 prior crates I’ve published. I closed up my laptop and called it a day.

Well, everything only seemed the same until an hour later when I got this tweet:

Hi Steven, Rust team here :) We’re having some technical difficulties with the name of one of your crates… feel free to DM

— @bstrie April 30, 2017

The impact

The crate I had just published was named nul, after the null terminator character which is abbreviated as NUL. I hopped onto Rust’s IRC channel to figure out what I had done, and the friendly Rust devs informed me that they were going to delete nul from crates.io.

While trying to catch up at this point, I saw that there was an issue filed on my repo. And an issue filed on cargo. And a reddit thread. Whoops.

Whenever any users on Windows attempted to download new crates or update their dependencies, they were greeted with this error:

Updating registry https://github.com/rust-lang/crates.io-index
error: [20/-1] Cannot checkout to invalid path '3/n/nul'

As kmc described the status of cargo on Windows:

But why?

Turns out that “NUL” (including “nul”) is a reserved filename on Windows. NUL is the Windows equivalent of Unix’s /dev/null, but instead of existing at a specific directory you can write to NUL in any directory, and even with any extension!

That might sound surprising, but Raymond Chen has an illuminating explanation:

Because DOS 1.0 didn’t have subdirectories. There was only one directory, which today we would call the root directory…

And so, since these DOS 1.0 days, tons of batch files exist on Windows redirecting with >NUL, and the reserved filenames remain.

The aftermath

Once my nul crate was deleted, Windows users were back in business. The Rust team quickly moved to prevent this issue from happening again by adding “NUL” and Windows’ 21 other reserved filenames to the list of reserved crate names. Guess I won’t get to publish that aux crate now 😉

Apologies to Carol Nichols and any other Rust devs whose weekends I interrupted, thanks for your quick action!

Although, I mean, like withoutboats said:

I thought Rust was supposed to prevent nul errors! ;-)

— @withoutboats April 30, 2017

Objective-C from Rust: Statically Verified Type Encodings without Allocation

2017-02-21T01:39:39-06:00

Lately I’ve been thinking about a new way to generate Objective-C type encodings from Rust. The Objective-C runtime needs these encodings when declaring methods and instance variables, and in Objective-C they’re simply generated via the @encode() compiler directive, but that’s not available in Rust!

The current approach used in the objc crate basically just treats them as strings, like they are in Objective-C. The intention was: when you want the encoding of a type, just go run the Objective-C compiler and copy the result of @encode().

However, the encoding for a type is not the same on all platforms; certain types have different encodings on different architectures. @encode(NSInteger) is "i" on 32-bit systems and "q" on 64-bit systems, while @encode(BOOL) is "c" on most platforms but "b" on ARM64.

With these discrepancies, it’s really convenient to have ways of building up a type encoding from its parts. A first approach could look something like:

fn encode_struct(name: &str, fields: &[&str]) -> String {
    let mut encoding = format!("{{{}=", name);
    encoding.extend(fields.iter().cloned());
    encoding.push('}');
    encoding
}

encode_struct("CGPoint", &[GCFloat::encode(), CGFloat::encode()])

This makes it easier for us to generate the struct’s encoding and use the appropriate encoding for CGFloat on our platform. However, it’s a stringly-typed API; encodings have a well-defined format, but this code accepts inputs that can make it produce invalid encodings, like:

encode_struct("CGPoint", &["Hello, World!"])

Can we do better? Is there a design that will allow us to say “I accept valid encodings and will produce a valid encoding from them”?

Abstract syntax

Looking at the documentation for encodings, we can see that their grammar is pretty simple. It’s not hard to represent the abstract syntax with Rust’s enums:

enum Encoding {
    Char,
    Int,
    ...
    Pointer(Box<Encoding>),
    Struct(String, Vec<Encoding>),
    ...
}

With this, it’s even easier to build the encoding of a struct from its parts, because the type system won’t let you build a struct encoding with an invalid format!

There’s something I’m curious if we can improve on, though: see Box and Vec in that definition? Each of those will involve a memory allocation. Representing as simple an encoding as "^i" will allocate additional memory, and complex encodings like "{CGRect={CGPoint=dd}{CGSize=dd}}" require more.

Zero allocation

Our new challenge is composing these encodings together without allocation or virtualization. That problem reminds me of the design of iterators and futures in Rust; rather than representing these as a single concrete type, there is a trait defining the core functionality and then iterators/futures can be combined through generic structs which also conform to the trait.

Let’s take some inspiration from this design and say that there is an Encoding trait which every type representing a valid encoding implements (we’ll come back later to exactly what functionality it should expose). Modelling a pointer encoding can be as simple as:

// T is the type of the target's encoding
struct Pointer<T: Encoding>(T);

Simple, and it avoids allocating!

Things get trickier when we want to model a struct, because we need to hold the encoding of its multiple fields which may be of differing types. My go-to solution for representing heterogenous collections is using tuples with a trait they implement exposing the functionality we need, like the MessageArguments trait in the objc crate. With such a trait, we could be able to create struct encodings like:

Struct { name: "MyStruct", fields: (Float, Pointer(Int)) }

Again, no allocations needed. But what functionality will be needed in our encoding-tuple trait?

Encodings trait

In the end, the functionality we’ll need from encodings will include writing out their string representation and comparing them for equality. For a struct encoding to do this, it will need access to its fields. If we could iterate over the fields of the struct, that should be sufficient!

Unfortunately, we can’t create an Iterator in this case; what would be the type of the items it yields? As mentioned earlier, we’re working with tuples that contain encodings of differing types, so there is no single type we could choose for the type of the iterator’s items.

There’s a different iteration technique we can fall back to: internal iteration. Internal (or push-based) iteration differs from external (or pull-based) iteration in that, instead of asking for each next element and then doing something with it, you provide an action to perform and that action is called for each item in the collection.

Internal iteration works for us because, instead of needing to say explicitly which type of item will be returned, we can just ask for an action that can be performed on any type that implements Encoding!

There’s one obstacle here: we can’t use Rust’s usual Fn types because when the closure is passed, we don’t know exactly which type of encoding it will be called with. In fact, we expect it to be called with different types of encoding! In effect, we want a closure that hasn’t yet been monomorphized, which doesn’t appear to be possible. If it were, the declaration would look something like this:

trait Encodings {
    fn each<F>(&self: callback: F) where for<T: Encode> F: FnMut(T) -> bool;
}

But instead we settle for defining the callback as a custom trait:

pub trait EncodingsIterateCallback {
    fn call<T: Encoding>(&mut self, &T) -> bool;
}

trait Encodings {
    fn each<F: EncodingsIterateCallback>(&self: callback: F);
}

And voilà! Now we have a way to iterate over a heterogenous collection of encodings. We can implement this trait for encodings of tuples and be on our way.

Encoding trait

You may have noticed that we’ve so far dodged an important question: what does the Encoding trait look like? Let’s figure that out!

As mentioned earlier, one of the things we’ll want to do with encodings is compare them for equality. But how can we compare with an encoding that could be any type? The Encoding trait will need to provide some sort of “common language” through which encodings of different types can interoperate.

Fortunately, the kinds of encodings that exist are constrained and well-known. We would be able to compare if we could ask encodings something like: are you a pointer? what’s the encoding of your target? are you an array? what’s your length?

To solve this, I used a Descriptor enum that describes which kind an encoding is and provides its properties. In pseudo-Rust, it looks like:

enum Descriptor {
    // Primitive is an enum of all encodings not composed from others
    Primitive(Primitive),
    Pointer(&(impl Encoding)),
    Array(u32, &(impl Encoding)),
    Struct(&str, &(impl Encodings)),
    Union(&str, &(impl Encodings)),
}

The real implementation looks gnarlier and is generic rather than using impl Trait, but it’s the same idea.

With this descriptor we are effectively able to “downcast” encodings and figure out more about them, allowing us to compare encodings or convert them to strings. The Encoding trait ultimately ends up looking like:

pub trait Encoding {
    type PointerTarget: ?Sized + Encoding;
    type ArrayItem: ?Sized + Encoding;
    type StructFields: ?Sized + Encodings;
    type UnionMembers: ?Sized + Encodings;

    fn descriptor(&self) -> Descriptor<Self::PointerTarget,
                                       Self::ArrayItem,
                                       Self::StructFields,
                                       Self::UnionMembers>;

    fn eq_encoding<T: ?Sized + Encoding>(&self, other: &T) -> bool {
        /* implementation provided based on the descriptor */
    }

    fn write<W: fmt::Write>(&self, writer: &mut W) -> fmt::Result {
        /* implementation provided based on the descriptor */
    }
}

Parsing encodings

As cool as the encoding structs we’ve made so far are, we can’t build encodings like them from parsing a string. This is because at compile time we have know idea how big the result would be, and with monomorphization the compiler would have to generate code for creating every possible kind of encoding struct, and there are infinitely many possible encodings. Believe me, I tried:

What we can do, though, is make a new kind of encoding that is initialized from a string and parses itself lazily; I ended up calling this type StrEncoding. When StrEncoding’s descriptor method is called, it determines which kind of encoding it is and creates more StrEncodings for any sub-encodings it contains. These sub-encodings reference safely into the original buffer, no copying required, thanks to Rust’s lifetimes and ownership system.

There is one gotcha: our Encoding trait was designed for structs with encodings inside them, and so was designed where references to the contained encodings are returned. This doesn’t seem to work with StrEncoding; it doesn’t own its sub-encodings (remember, it creates them on demand), and how could we return a reference to something that was created inside the method and is about to drop out of scope?

The key to solving this was realizing that the StrEncoding doesn’t actually need any data other than its input string. What if we were able to convert a reference to a str into a reference to a StrEncoding? Turns out this is possible if StrEncoding is defined as a dynamically-sized type wrapping str:

struct StrEncoding(str);

And with that, when a StrEncoding is parsed, it is able to return references to sub-encodings with the same lifetime!

Owned StrEncoding

As nice as our dynamically sized StrEncoding is, sometimes it’d be beneficial to have a parsed encoding which owns its buffer and can be passed around without worrying about lifetimes.

Complicating this is the fact that I was aiming to have this encodings crate be no_std, relying only on libcore. String doesn’t even exist in libcore! How can we make a parsed encoding which owns a String when it’s not available?

Since we can’t access String ourselves, I figured we’d have to use a generic type parameter through which users could specify that String is used. I scoured the docs for a trait that is implemented by String and accessible in libcore and found just what we need in AsRef<str>.

A further bonus is that str itself implements AsRef<str>, so we don’t actually even need a new type for this; we can modify StrEncoding to support its original use case and this new one:

struct StrEncoding<S = str>(S) where S: ?Sized + AsRef<str>;

Now we can have multiple flavors of string encodings: the original, dynamically sized version as &StrEncoding<str>, or one with an owned buffer as StrEncoding<String>.

The end result

I’ve published the end result as the objc-encode crate, which allows creating encodings that are statically known to be valid and can be compared against encodings parsed from a string representation, all without memory allocation.

The encodings compose well, albeit with a bit of boilerplate for more complex structs. The good news is that this boilerplate is very formulaic, and I think generating it would be a great candidate for a custom derive macro, something I’m eager to try in the future!

Testing Rust on iOS with Travis

2016-02-23T00:10:43-06:00

Since I started automatically testing my Rust crate with Travis, I had always wanted to test on iOS as well. Unfortunately, the default Rust compiler on Travis doesn’t support iOS, and compiling one from scratch takes a prohibitively long time. With Rust’s improved cross-compilation support, it’s now possible!

Travis configurations

We’ll be running our tests on Xcode’s iOS simulator, which means that these tests have to run on Travis’s OSX image. We’ll start with these Travis configurations for iOS builds:

language: rust
rust: nightly
os: osx
osx_image: xcode7.2

If you’ll be testing your library on OSX and iOS, I’ve found it convenient to use an IOS_ARCHS environment variable; this variable lists all the iOS architectures we should build, and will be empty for our OSX builds. To build for all currently supported iOS architectures, IOS_ARCHS can be defined as IOS_ARCHS="i386 x86_64 armv7 armv7s aarch64".

By using an IOS_ARCHS environment variable, we can take advantage of Travis’s build matrix functionality to have separate iOS and OSX builds:

env:
  - IOS_ARCHS=""
  - IOS_ARCHS="i386 x86_64 armv7 armv7s aarch64"

This will trigger two separate builds in Travis’s matrix: one for no iOS architectures (i.e. an OSX build) and one for all iOS architectures.

Setting up the cross compiler

The Rust compiler included on Travis is not able to compile for iOS. With Rust’s cross compiler support, though, compiling for iOS is just a matter of downloading the iOS versions of the standard library.

Following the rust-cross guide, we can add a Travis install step to install all the architectures we’ll be building for:

for arch in $IOS_ARCHS; do
    ios_stdlib="rust-std-nightly-${arch}-apple-ios"
    curl -O "http://static.rust-lang.org/dist/${ios_stdlib}.tar.gz"
    tar xzf "${ios_stdlib}.tar.gz"
    "./${ios_stdlib}/install.sh" --prefix=$(rustc --print sysroot)
done

Building an iOS library

The trickiest part about testing Rust on iOS currently is that you cannot directly run the #[test]s you’ve written in Rust. Rust’s testing infrastructure compiles tests into an executable to be run, but executables on iOS have to be packaged as apps and signed properly, a process best handled by Xcode.

I chose to extract some of my testing code into a library that could be called from iOS, and then wrote a simple iOS test project that would link and call these tests. The objc crate actually has a Python script to parse out the tests from the project to be included in a crate for external tests; ideally this will someday be replaced with a custom test harness.

The code we’d like to test on iOS has to be compiled to a static library, as support for dynamic libraries on iOS was only added recently and Rust does not yet support them. To compile to a static library, simply add the crate-type key to the lib section of the Cargo.toml for your test library:

[lib]
crate-type = ["staticlib"]

The test code that will be called from our iOS app must be properly exposed like any Rust code called from C:

#[no_mangle]
pub extern fn rust_lib_test_functionality() -> c_int {
    ...
    // Returns a non-zero code for test failure
    if success { 0 } else { 1 }
}

Great! Now we have a static lib exposing some test functionality that we’ll call on iOS.

One last thing: iOS libraries support multiple architectures, so to test on multiple architectures we’ll want to combine the different builds of our library using the lipo tool:

for arch in $IOS_ARCHS; do
    cargo build --target "${arch}-apple-ios"
done
lipo -create -output "${LIB}" target/*-apple-ios/debug/"${LIB}"

iOS test project

Now that we have our library of test code to call from iOS, we can create an iOS test project. Starting out from an empty project in Xcode, we’ll add a new iOS unit testing target:

Next, drag your compiled library of Rust tests into the “Link Binary With Libraries” section of the “Build Phases” for your test target.

Now we’re ready to write an XCTestCase verifying the functionality of the Rust library:

int rust_lib_test_functionality();

@interface RustLibTests : XCTestCase
@end

@implementation RustLibTests

- (void)test {
  int result = rust_lib_test_functionality();
  XCTAssertEqual(result, 0, "Test failed with a non-zero code");
}

@end

At this point, we can run our tests from within Xcode and should see a nice green checkmark if it’s passing on the iOS simulator!

Running on Travis

The last step is simply adding a Travis script step to run our iOS tests.

One caveat, first: running tests from a script requires a shared scheme, so make sure the test target scheme is shared and the .xcscheme file committed. This can be done in Xcode from the “Product” > “Scheme” > “Manage Schemes…” menu:

Now, we can add a script step using xcodebuild. xcodebuild takes a multitude of options, but if your project and scheme were named RustLibTests, running your tests on an iPhone 5 and 5s simulator (32 and 64-bit, respectively) could be done with the following command:

xcodebuild \
    -project RustLibTests.xcodeproj \
    -scheme RustLibTests \
    -destination 'platform=iOS Simulator,name=iPhone 5' \
    -destination 'platform=iOS Simulator,name=iPhone 5s' \
    test

And there we have it! Now, Travis will run our tests on an iOS simulator to verify that our crate is working.

For an example of this in action, check out the objc crate. It may seem like a lot of hoops to jump though, but I am impressed with the ease of setting up Rust itself for cross-compilation.

Objective-C from Rust: objc_msgSend

2015-08-03T01:53:50-05:00

I previously wrote about how to interoperate between Objective-C and Rust and use a Rust macro to pass a variable number of arguments to objc_msgSend. Well, there’s a problem with this: objc_msgSend isn’t a variadic function!

objc_msgSend is actually a “trampoline” that works by jumping directly to the implementation of the method, not calling it and passing parameters. Safely calling it requires first casting it to the type of the underlying method implementation, like:

// id result = [obj description];
id result = ((id (*)(id, SEL))objc_msgSend)(obj, @selector(description));

If you try to call objc_msgSend as if it were a variadic function, weird things can happen!

Handling arguments

How can we handle objc_msgSend in Rust? Let’s try to write a wrapper function that calls it correctly. It’s easy to see how we could do this for, say, two arguments:

unsafe fn msg_send<A, B, R>(obj: *mut Object, op: Sel, arg1: A, arg2: B) -> R {
    // Transmute objc_msgSend to the type of the method implementation
    let msg_send_fn: unsafe extern fn(*mut Object, Sel, A, B) -> R =
        mem::transmute(objc_msgSend);
    msg_send_fn(obj, op, arg1, arg2)
}

But our function needs to be able to accept one argument, or no arguments, or more than 2. This sounds a lot like function overloading, which Rust doesn’t support. However, there’s a pattern that can emulate function overloading in Rust: functions with a generic parameter. In our case, our function could take a generic parameter that is the arguments, represented as a tuple:

unsafe fn msg_send<T, R>(obj: *mut Object, op: Sel, args: T) -> R { ... }

How do we implement this now? Well, it was easy to implement for a fixed number of arguments, so we can let the arguments themselves handle calling objc_msgSend. Let’s add a MessageArguments trait and implement it for tuples:

impl<A, B> MessageArguments for (A, B) {
    unsafe fn send<R>(self, obj: *mut Object, op: Sel) -> R {
        // Transmute objc_msgSend to the type of the method implementation
        let msg_send_fn: unsafe extern fn(*mut Object, Sel, A, B) -> R =
            mem::transmute(objc_msgSend);
        let (arg1, arg2) = self;
        msg_send_fn(obj, op, arg1, arg2)
    }
}

Using a macro, we can easily implement this trait for tuples from size 0 to some upper bound. Our function simply becomes:

unsafe fn msg_send<T, R>(obj: *mut Object, op: Sel, args: T) -> R
        where T: MessageArguments {
    args.send(obj, op)
}

And now we’re no longer pretending that objc_msgSend is variadic!

Return types

There’s one other caveat with objc_msgSend, though: different versions of it are used for different return types. Which version is used depends on the calling conventions of the architecture.

Let’s encapsulate this with a simple function that returns the correct version of objc_msgSend for the return type:

fn msg_send_fn<R>() -> unsafe extern fn(*mut Object, Sel, ...) -> R { ... }

We can implement this for different architectures with a cfg attribute. x86’s calling conventions are arguably the most complicated, so for it this function would look like:

#[cfg(target_arch = "x86")]
fn msg_send_fn<R: Any>() -> unsafe extern fn(*mut Object, Sel, ...) -> R {
    use std::any::TypeId;

    let type_id = TypeId::of::<R>();
    let size = mem::size_of::<R>();
    if type_id == TypeId::of::<f32>() || type_id == TypeId::of::<f64>() {
        unsafe { mem::transmute(objc_msgSend_fpret) }
    } else if size == 0 || size == 1 || size == 2 || size == 4 || size == 8 {
        unsafe { mem::transmute(objc_msgSend) }
    } else {
        unsafe { mem::transmute(objc_msgSend_stret) }
    }
}

With this msg_send_fn function defined for each architecture, we can use it to get the correct version of objc_msgSend for our return type.

At this point we’re invisibly handling objc_msgSend correctly for any combination of argument types and any return type. We can further wrap it with a macro to make it more ergonomic; in the end, calling Objective-C methods is nearly as easy from Rust as it is in Objective-C!

Interoperating Between Objective-C and Rust

2014-11-28T03:52:17-06:00

Since the Objective-C runtime exposes a C interface, it’s actually pretty easy to interact with from Rust. Over the past months I’ve worked on a Rust wrapper around the Objective-C runtime and some classes of the Foundation framework, creatively called rust-objc. I had hoped to learn more about Rust’s foreign function interface and the Objective-C runtime, but along the way I also encountered some interesting challenges in API design.

Calling Objective-C methods

If we want to interact with Objective-C from Rust, one of the first things we’ll need to be able to do is call methods on Objective-C objects. Let’s consider this example where we have an NSString pointer, string:

const char *c_string = [string UTF8String];

Since the Objective-C runtime actually has a C interface, we can invoke methods in C using the objc_msgSend function. Our previous Objective-C code is equivalent to this C:

SEL selector = sel_registerName("UTF8String");
const char *c_string = (const char *)objc_msgSend(string, selector);

Now that we see the C code, the translation into Rust is pretty straightforward using Rust’s foreign function interface. Once we’ve set up an interface for the functions of the Objective-C runtime, we can write:

let selector = "UTF8String".with_c_str(|name| unsafe {
    sel_registerName(name)
});
let c_string = unsafe {
    objc_msgSend(string, selector) as *const c_char
};

Nice! But this is Rust, we can make this better with Rust’s powerful macros. We can even support methods with arguments using a macro like this:

macro_rules! msg_send(
    ($obj:expr $($name:ident : $arg:expr)+) => ({
        let sel_name = concat!($(stringify!($name), ':'),+);
        let sel = sel_name.with_c_str(|name| {
            sel_registerName(name)
        });
        objc_msgSend($obj, sel $(,$arg)+)
    });
)

By adding a special case to our macro for the no-argument case, we can rewrite our example as:

let c_string = unsafe {
    msg_send![string UTF8String] as *const c_char
};

And so we have a convenient way to call Objective-C methods with a syntax that feels comfortable for Objective-C developers.

Representing Objective-C objects

In our previous examples, we’ve been working with a variable named string, but what is the type of this variable? Well, in Objective-C it’d be declared like this:

NSString *string;

The identical declaration in Rust would look like this:

let string: *const NSString;

Okay, so we’ll need some sort of NSString type in Rust. Since we don’t actually know or care about the memory layout of the NSString, we could declare it simply as a unit struct:

struct NSString;

There’s a problem with this, though: it allows user to construct an NSString on the stack, and Objective-C objects only live on the heap!

let string_on_stack = NSString;
let c_string = unsafe {
    msg_send![&string_on_stack UTF8String] as *const c_char // Oops!
};

We don’t actually want users to be able to construct our NSString, we’ll just be giving them pointers and references to one. To avoid this, we could use a phantom type:

enum NSString { }

By using an enum with no variants, NSString will be a valid type but there is no way for users to instantiate an instance of one.

Unfortunately, this still has a problem; if a user has a reference to this NSString, they can still dereference it in safe code:

let string: &NSString;
let string_on_stack = *string;

This happens because the Rust compiler sees that our enum has no fields that can’t be copied, and therefore infers that our enum is copyable as well. To solve this, we must use the NoCopy marker:

struct NSString {
    nocopy: NoCopy,
}

This also lets us use a struct again; now that it has a private field, users cannot construct an NSString themselves. As long as we don’t construct an NSString on the stack in our module, there will be no way in safe code for users to end up with a stack-allocated NSString.

Drawbacks of this representation

This isn’t a perfect solution, because even if there’s no way to get a stack-allocated NSString, the compiler will still accept definitions like:

let string: NSString;
let vector: Vec<NSString>;

Additionally, the following code will compile and run without doing anything:

let a: &mut NSString;
let b: &mut NSString;
mem::swap(a, b); // Doesn't actually do anything

Ideally, we would opt out of the Sized trait so that the compiler would disallow these types as local variables, but unfortunately it doesn’t seem possible to have an unsized type without all references to it becoming “fat” two-word references.

Why not just wrap the pointer?

If an NSString can never exist on the stack, why don’t we just prevent that by making a struct that wraps a pointer?

struct NSString {
    ptr: *mut c_void,
}

Let’s consider the case of an NSArray of NSStrings. If we want to get a string from the array, our array can’t return references to this NSString struct:

fn object_at(array: &NSArray, index: uint) -> &NSString {
    let string_ptr = unsafe {
        msg_send![array objectAtIndex:index]
    };
    let string = NSString { ptr: string_ptr };
    &string // Oops! string doesn't live past this method
}

Instead, we’d have to return this NSString struct by value, and then it’s not tied to the lifetime of our array at all. This would allow us to get multiple copies of an NSString from our array and try to mutate them simultaneously, which would cause a race condition. To fix this we’d need to add a lifetime parameter to indicate that the string is only valid as long as the array is mutably borrowed. Not all strings should have this lifetime parameter, though, so we’d actually end up needing 3 different NSString representations: an owned string (NSString), one representing an immutable borrow (NSStringRef<'a>), and one representing a mutable borrow (NSStringRefMut<'a>). This results in an interface that looks odd to both Rust and Objective-C developers.

I felt that, despite the imperfections of representing Objective-C objects as structs in Rust, it makes for a much more usable API.

A safe Rust interface

Now that we’ve got a struct for representing our NSString, we can implement some methods on it. For example, we can wrap the UTF8String method using idiomatic Rust types:

impl NSString {
    fn as_str(&self) -> &str {
        unsafe {
            let c_string = msg_send![self UTF8String] as *const c_char;
            c_str_to_static_slice(c_string)
        }
    }
}

Here we can also see one of the challenges of wrapping Objective-C with a safe interface. Of the C string returned by UTF8String, the docs say:

This C string is a pointer to a structure inside the string object, which may have a lifetime shorter than the string object and will certainly not have a longer lifetime.

We’ve assumed that as long as the string isn’t mutated, the internal pointer is still valid, but since Foundation is closed source there isn’t really a way for us to verify this.

Inheritance

What happens when we decide to implement a safe interface for NSMutableString? Since NSMutableString inherits from NSString, it should also have this method, but Rust structs don’t allow inheritance. Instead of just duplicating the method, we can implement it in a trait:

trait INSString {
    fn as_str(&self) -> &str {
        unsafe {
            let c_string = msg_send![self UTF8String] as *const c_char;
            c_str_to_static_slice(c_string)
        }
    }
}

impl INSString for NSString { }

Now if we just implement INSString for NSMutableString, it’ll get this functionality, too. This trait is also useful for generic programming; with it, we can write functions that take any type that implements the INSString trait and will accept either an NSString or an NSMutableString.

There is a drawback to this approach, though: users could implement this trait for any type inappropriately. Since it doesn’t require any other methods implemented, I could, in safe code, just implement the INSString trait for int and then have undefined behavior by sending Objective-C messages on an int. I don’t know of a way to prevent this without losing the convenience of only declaring these methods once.

Objective-C memory management

Great, at this point we can call methods from an NSString reference, but where does this reference come from? What’s its lifetime?

Our Objective-C objects must be retained while we’re using them and released when we’re done with them, so this is a great fit for a custom smart pointer in Rust:

struct Id<T> {
    ptr: *mut T,
}

impl<T> Drop for Id<T> {
    fn drop(&mut self) {
        unsafe { msg_send![self.ptr release]; }
    }
}

impl<T> Deref<T> for Id<T> {
    fn deref(&self) -> &T {
        unsafe { &*self.ptr }
    }
}

Now we can use this to create safe wrappers over an object’s initializers:

impl NSString {
    fn new() -> Id<NSString> {
        unsafe {
            let cls = "NSString".with_c_str(|name| {
                objc_getClass(name)
            });
            let obj = msg_send![class alloc];
            let obj = msg_send![obj init];
            Id { ptr: obj }
        }
    }
}

This finally allows us to work with an NSString without any unsafe blocks!

let string = NSString::new();
println!("{}", string.as_str());

When the Id goes out of scope, the object will automatically be released. With just a few lines of Rust code, we’ve implemented our own simplified version of Objective-C’s automatic reference counting.

Mutability

Sometimes we may want to retain a shared object, but it wouldn’t be safe to do this if we implement DerefMut for any Id, because if it is mutably dereferenced in multiple places we’d have aliasing mut references. Similarly, it’d be safe to implement Clone when the object is shared, but an Id that implements DerefMut shouldn’t implement Clone.

I chose to resolve this was by adding a phantom type parameter to Id which is either Owned or Shared. Then, we can implement Clone only for a shared Id, and we can implement DerefMut only for an owned Id.

enum Owned { }
enum Shared { }

impl<T> Clone for Id<T, Shared> {
    fn clone(&self) -> Id<T, Shared> {
        unsafe { msg_send![self.ptr retain]; }
        Id { ptr: self.ptr }
    }
}

impl<T> DerefMut<T> for Id<T, Owned> {
    fn deref_mut(&mut self) -> &mut T {
        unsafe { &mut *self.ptr }
    }
}

We can also allow an owned Id to be “downgraded” to a shared Id and then cloned.

impl<T> Id<T, Owned> {
    fn share(self) -> Id<T, Shared> {
        Id { ptr: self.ptr }
    }
}

Thinking about Objective-C in terms of Rust’s memory semantics leads to some interesting questions, and these phantom types will be used again. For example, unlike a Vec in Rust, an NSArray can be copied without copying all of its elements. If we consider the array to own its objects, this isn’t safe because it could create aliasing mut references. However, it’s totally fine if the array’s objects are shared. We can resolve this by using an approach similar to Id: if our NSArray has a type parameter for Owned or Shared, we only implement copying for the shared array.

Simplifying JSON Response Mocks With Jinja

2013-12-14T15:43:01-06:00

For documenting and testing large APIs, response mocks can be a very useful tool. They not only provide examples for the users of your API, but they can be used in automated tests as if you had received real data without actually hitting the network.

Creating and maintaining a large collection of response mocks can be greatly simplified by preprocessing them with a template engine like Python’s Jinja. Although this adds another step to your build process, it allows some really useful functionality!

Variables

One of the first advantages of processing response mocks with Jinja is that we can use variables when generating them. For example, let’s consider an API that can respond with information about a person. One of our mock responses is a Person object, like so:

{
    "first_name": "John",
    "last_name": "Smith",
    "phone_number": "415 555-1234"
}

However, developers in different locations require that the phone number have a local area code. In this situation, we could simply change the mock to be:

{
    "first_name": "John",
    "last_name": "Smith",
    "phone_number": "{{ area_code }} 555-1234"
}

Now, we only need to process the mocks through Jinja with the area code in the context to generate mocks using the new area code. We can render the mock with area_code='312' in the context to get the phone number "312 555-1234".

Conditionals

We can further take advantage of Jinja to template our mocks using more complicated control flows. For example, suppose that we wanted to format our mock from the previous example to include a Person’s last name sometimes, but not in other cases. We can accomplish this simply in Jinja with an if block:

{
    "first_name": "John",
    {% if last_names_included %}
    "last_name": "Smith",
    {% endif %}
    "phone_number": "415 555-1234"
}

If this mock is rendered with last_names_included=True in the context, it will include the last_name field, and if not it will be absent!

Including mocks

The previous examples might feel a little contrived, but importing one mock from another is one of the greatest benefits of Jinja. A large API will likely have objects with a common scheme that are used in multiple responses, and it can be very convenient and greatly simplify our mocks if we can define an object once and reference it in many places.

Perhaps multiple responses from our example API include addresses in a common format. We can provide an example address in its own file:

{
    "street_address": "21 2nd Street",
    "city": "New York",
    "state": "NY",
    "postal_code": 10021
}

Then, in any response that includes an address, instead of copy-pasting this example address or creating an entirely new address, we can simply include our address in the mock with Jinja’s include tag:

{
    "first_name": "John",
    "last_name": "Smith",
    "address": {% include 'address.json' %},
    "phone_number": "415 555-1234"
}

The include tag does not only let us avoid duplicating data; if each object is stored in its own file, it can turn our response mocks into a simple composition of objects where the structure is clear at a glance.

Extending mocks

I commonly need to create a response mock that is modified slightly from an existing one, like a version of the response that has a parameter with a different value or that specifies some optional parameters. I think of this like JSON “inheritance”, where an object gets all the values from a base object but can then override them.

With Jinja, we call allow this “inheritance” by creating a way to extend mocks. We can simply model the changes we want and the base object as Python dicts and use the update method to override the items in the base dictionary with our changes. The dictionary of updates will be specified in JSON, so a function to return the JSON that results from updating looks like this:

def json_update(base, updates):
    result = json.loads(base)
    result.update(json.loads(updates))
    return json.dumps(result, sort_keys=True, indent=4)

We can then include this function in Jinja’s context to access it within our response mocks. This allows us to write an extend_json macro:

{% macro include_json(path) %}{% include path %}{% endmacro %}

{% macro extend_json(base_path) %}
    {{ json_update(include_json(base_path), caller()) }}
{% endmacro %}

This macro will be called with the JSON updates as its body so that they are accessible within the macro through the caller() function. The include_json macro just allows us to use an included file as an argument to the json_update function.

To see this macro in action, let’s consider our Person object with an age parameter:

{
    "first_name": "John",
    "last_name": "Smith",
    "age": 25,
    "phone_number": "415 555-1234"
}

To create another Person object with a different age, we only have to do this:

{% call extend_json('person.json') %}
{
    "age": 70
}
{% endcall %}

It’s useful to note that the extend_json macro must be available in this file, so we’ll likely need to import it from the file in which it is declared.

The functionality of Jinja doesn’t end with macros; after all, it’s a full-fledged template engine with many built-in filters and a powerful (if complicated) extension system. This article just covered the functionality that I’ve found useful, but Jinja can surely adapt to meet whatever your needs are.

Waiting for Octopress 2.0's Successor

2013-07-07T02:24:00-05:00

The next version of Octopress is packed full of awesome improvements. Octopress will change massively to be more extensible and customizable while still being as powerful and easy as ever; as such, this update has been in development for a long time, still not having released officially.

In anticipation of this update, I periodically check the Octopress Twitter for news and have encountered many release date estimates. Estimating time for development is notoriously difficult, so I’ve compiled a lighthearted Twitter history of Octopress 2.0’s successor and its estimated release dates.

The follow-up to Octopress 2.0 was originally envisioned as a much smaller set of changes to be called Octopress 2.1. The release of this version was first mentioned in March of 2012 as “soon”.

That will be shipping in 2.1 (soon).

— @octopress March 27, 2012

By May, the outlook for Octopress 2.1 was looking good enough that its release seemed imminent, within a “day or two”.

Have you tried 2.1 yet? I’m hoping to release it in the next day or two.

— @octopress May 23, 2012

But it was not to be! A week and a half later, in June, 2.1 was still looking as though it would release “soon”.

2.1 should be releasing soon. It has lots of goodies and fixes.

— @octopress June 3, 2012

Next month, July, enough progress had been made to offer a new estimate that 2.1 would release in a “week or two”.

It’ll be released once I finish a couple of remaining features. I’m hoping that’ll be in the next week or two.

— @octopress July 2, 2012

It was not until 4 months later in November that the next estimation was made that 2.1 would be released before the end of 2012.

I can’t give a solid date right now. But I expect to have a release before years end.

— @octopress November 14, 2012

With an acknowledgement of the difficulty in estimating, by January 2013 Octopress 2.1 was predicted to be completed “soon”.

We’re working on an asset pipeline, fixing any critical bugs and setting up a migration for 2.0. It’s hard to estimate, but soon.

— @octopress January 10, 2013

In February, it seemed that the eagerly awaited update would be finalized within a “month or two”.

right now [the 2.1 branch is] not entirely stable. We’re working on a few large features then we’ll release it in the next month or two.

— @octopress February 8, 2013

With the tremendous amount of effort that had gone into the update, it seemed more fitting that the major version change, so it would instead be released as Octopress 3.0.

By March, this 3.0 update’s release date was further clarified to “mid April”.

Right now, 2.1 is under pretty active development. We’re shooting for a mid April release, so it might be a good idea to wait.

— @octopress March 20, 2013

My mid April, though, it seemed more likely that the release would occur later in the month.

[The next release is] in flux right now. We’re hoping for late April.

— @octopress April 15, 2013

Although an April release had been missed, by late May it seemed that there would be a 3.0 release candidate available within a “month or so”.

We’re currently working through getting some core functionality working, but I’ll be announcing an RC in the next month or so.

— @octopress May 27, 2013

At the end of June, the ambitious Octopress 3.0 was still predicted to arrive during the summer.

[3.0 is] Going well. We’re targeting a summer release, still lots to do though. :)

— @octopress June 30, 2013

And so we still anticipate the official release of this update that will redefine Octopress. This does showcase a neat benefit of open source; despite not being officilly released, you can try out 3.0 and closely follow its development on GitHub.

Regardless of Octopress’ history of imprecise estimates, I have great faith in its developers to deliver such an ambitious update. I’ll keep an eye out on Twitter for an announcement that Octopress 3.0 has finally been perfected!

Update (Aug 18, 2013): By the end of July, there was still “a ways to go” on Octopress 3.0, but version 2.1 of Octopress was resurrected as a smaller, incremental upgrade on the way to Octopress 3.0.

we still have a ways to go. Right now I’m working on 2.1 which adds support for Jekyll 1.0 and uses Guard for watching files.

— @octopress July 27, 2013

Update (Feb 15, 2014): In September, Octopress 3.0 still wasn’t ready for release, but it seemed mature enough that a blog post was promised detailing the roadmap leading up to its release.

We’ve finally nailed it down. Blog post coming soon after @imathis gets back from BlendConf.

— @octopress September 4, 2013

Despite it originally being anticipated that this blog post would be published within a week, by mid October there were still details to be ironed out.

There’s a draft. Still ironing out some details. :)

— @octopress October 21, 2013

By the new year, it appeared that version 3.0 was still “a couple of months” away from being released.

The update is coming with Octopress 3.0 (which is probably a couple of months out) you should be able to migrate content easily.

— @octopress January 7, 2014

It also appears that the previously proposed version 2.1 was abandoned, so no new version is slated before Octopress 3.0.

Octopress

2012-11-15T22:14:00-06:00

I really like git; if I could keep everything under version control, I would. I’ve started writing all my papers in LaTeX instead of Word, not only because it’s cleaner and can provide more control, but because then I can version control them. Hell, I’ve even started keeping my Borderlands 2 game saves backed up in a git repository.

Imagine my delight, then, when I discovered Octopress. Octopress is a framework that makes it really easy to generate a blog just from a set of markdown files that can be hosted on GitHub pages. I’ve already been using GitHub to host my personal website, and using it to host a blog would be pretty swell.

I’ve worked through the documentation to get a blog up and running; after all, I have homework to be working on right now, so I need to find something else to do. Nevermind the fact that I never actually put in the effort to blog or can come up with interesting topics; if nothing else, this was just an exercise to test out Octopress and to become more familiar with the features of GitHub pages.