Pseudo Associative

Random thoughts on software development

0 notes

Get Started Writing TextMate 2 Bundles

TextMate 2 is my favorite editor. In many ways it is the modern decendant of vim and emacs. Like those editors it is very flexible when it comes to extending it while following modern GUI conventions. One of the things I really like is the extension mechanisms. You can create macros and snippets easily. But what I want to talk about in this article is TextMate commands. Commands are external scripts you can invoke from TextMate. The cool thing about these scripts compared to vim and emacs is that you can write them in any language.

However information about how you write these scripts seem to be scattered, so this is my attempt to help you get started writing scripts to extend TextMate 2. This allows you to do all kinds of cool stuff like creating command completion like you find in modern IDEs.

The Unix philosophy of TextMate

TextMate is different from how you make plugins in a lot of IDEs. You don’t create a dynamically loaded library which adheres to some binary interface. The downside of that is that you have to develop your plugin in a particular programming language.

TextMate bundles’s on the otherhand are based on the unix philosophy of creating software by combining small programs which process text.

A TextMate command is a script which receives its input from TextMate through regular environment variables and standard input. TextMate will setup its own environment variables before running your script.

Variable Description
TM_CURRENT_LINE Text found on the line the caret is on
TM_CURRENT_WORD Word at location of caret
TM_SELECTED_TEXT Contains text currently selected in document
TM_LINE_INDEX Zero based position of caret on line
TM_LINE_NUMBER Line number at caret. Counts from one.

There are a lot more environment variables. You can read about them in the Pragmatic Programmers TextMate book. Although it is for TextMate 1 it is very usefull for TextMate 2.

Environment variables are not suited for large amounts of data, so the bulk of data is sent in through STDIN.

For each command you can configure how information about current line, current selection etc is sent to your script.

Configuration of a TextMate Command

TextMate store a configuration file with your command which it reads to figure out how to pass data to it what to do with the data received back through STDOUT. It configuration file might say to replace the current selection with the output from the script or insert the output at the caret. There are a lot of choices.

How to Debug and Develop a Command

Just as when debugging any other program without a dedicated debugger you can use printf style debugging. Anything you send to STDOUT can be shown by TextMate. E.g. you can configure your command to display output from STDOUT in a separate window, tooltip or just insert it right in your document at the caret.

Example of a TextMate Command is Implemented

This example is taken from the bundle which comes for the programming language Go. This bundle provides function and type completion through a command. It is implemented as a ruby script which calls a unix command named gocode which you need to install like any other Go command. gocode can be invoked at the command line:

gocode -f=json --in=foobar.go autocomplete 449

This will look at at the 449th character in the source code file foobar.go and produce a list of possible completions formated in JSON.

Our TextMate command consists of a Ruby script which invokes this command. First the script load a whole bunch of ruby scripts which come bundled with TextMate:

require ENV['TM_SUPPORT_PATH'] + '/lib/ui.rb'

This loads the ui.rb file. The TM_SUPPORT_PATH environment variable will always contain the path to scripts bundled with TextMate to help you make bundles.

Next step is to read in the whole file. This script does not pass the file in through STDIN.

document = []['TM_FILEPATH'], "r+") do |file|
  document = file.readlines

Since gocode can’t work with line and column number directly which is what TextMate sends our script through environment variables it has to calculate a byte offset like this:

cursor = document[ 0, ENV['TM_LINE_NUMBER'].to_i - 1].join().length + ENV['TM_LINE_INDEX'].to_i

Now we have enough information to execute the gocode command and get a list of completions. This script gets the result in CSV format and not JSON:

output = `$TM_GOCODE -f=csv -in=#{e_sh ENV['TM_FILEPATH']} autocomplete #{cursor}`

output now contains a string formated in CSV with our completion suggestions. The completions are sent to a ruby function provided by TextMate called TextMate::UI.complete. It pops up a list of our choices and let users choice a completion. However complete expects the completions in its own format:

They need to be formated as array of dictionaries with the following keys:

  • display The title to display in the suggestions list
  • insert Snippet to insert after selection
  • image An image name, see the :images option
  • match Typed text to filter on (defaults to display)

Trying to complete on fmt. I get the following CSV back (only first 4 lines):

func,,Errorf,,func(format string, a ...interface{}) error
func,,Fprint,,func(w io.Writer, a ...interface{}) (n int, err error)
func,,Fprintf,,func(w io.Writer, format string, a ...interface{}) (n int, err error)
func,,Fprintln,,func(w io.Writer, a ...interface{}) (n int, err error)

To work with TextMate::UI.complete it needs to be changed to an array of dictionaries like this (just 4 first matches):

{"match"=>"Errorf", "display"=>" Errorf(format string, a ...interface{}) error", "insert"=>"(${1:format string}, ${2:a ...interface{\\}})$0", "image"=>"func"}
{"match"=>"Fprint", "display"=>" Fprint(w io.Writer, a ...interface{}) (n int, err error)", "insert"=>"(${1:w io.Writer}, ${2:a ...interface{\\}})$0", "image"=>"func"}
{"match"=>"Fprintf", "display"=>" Fprintf(w io.Writer, format string, a ...interface{}) (n int, err error)", "insert"=>"(${1:w io.Writer}, ${2: format string}, ${3:a ...interface{\\}})$0", "image"=>"func"}
{"match"=>"Fprintln", "display"=>" Fprintln(w io.Writer, a ...interface{}) (n int, err error)", "insert"=>"(${1:w io.Writer}, ${2:a ...interface{\\}})$0", "image"=>"func"}

TextMate::UI.complete works by spawning a new executable bundled with TextMate called tm_dialog2 found under in the TextMate app. This is called with the following arguments:

tm_dialog2 popup --returnChoice --alreadyTyped '' --additionalWordCharacters _

The array of dictionaries are sent over to the tm_dialog2 process through its STDIN. Before it is sent it is formated as a OS X property list (plist). TextMate comes bundled with a function to_plist attached to array objects to do this.

To test this functionality you can write a simple script which executes this line:

tm_dialog2 popup --suggestions '( { display = law; }, { display = laws; insert = "(${1:hello}, ${2:again})"; } )'

And configure the command to output to caret position. Make sure the output of this command is send to STDOUT. When you run this script a list with two choices will popup at the caret:

  • law
  • laws

If you select law then law will be inserted at caret. If you pick laws then you will get a snippet inserted:

laws(hello, again)

Where you can tab between hello and again. Here is an example of a Julia script which does exactly this:

#!/usr/bin/env julia
tm_dialog = ENV["DIALOG"]

print(readall(`$tm_dialog popup --suggestions '( { display = law; }, { display = laws; insert = "(${1:hello}, ${2:again})"; } )'`))

Open TextMate from HTML document

If you have done iOS development you would be familiar with the concept of URL Schemes. This is not limited to iOS, but also supported on Mac OS X. TextMate registers in its Info.plist (contained in the Application bundle) the URL Scheme txmt. So any URL starting with txmt will cause a method in TextMate to be called which will be passed the URL. Thus any program which handles URLs in OS X can potentially open a TextMate document. E.g. the Safari Web browser or the Unix command open:

$ open "txmt://open?line=14&url=file://filename.txt"

This opens the textfile filename.txt at line number 14. This is the basis for how TextMate uses HTML pages with errors to let you jump to a line number.


Commands are regular Unix commands which you send and reveive data to through STDIN and STDOUT. Input can be provided by environment variables as well. For each command you provide a configuration file which you can setup with a GUI in TextMate. The configuration file says what data is transfered on STDIN to your command and what is doine with the text output from your script.

Filed under textmate textmate2 julialang unix bundle golang

0 notes

When adapting an iOS6 app for iOS7, you need to move any immediate child of your top level view 20 pixels down to make way for the status bar. In iOS6 the status bar area is not part of your top level view, so this change will cause everything to get shifted 20 pixels below the status bar. To account for this you need to set the iOS 6/7 deltas. Set dY on your child view to -20 to move it up 20 pixels when running under iOS6.

However this will shrink your view by 20 pixels, so you need to set dHeight to +20 pixels.

When adapting an iOS6 app for iOS7, you need to move any immediate child of your top level view 20 pixels down to make way for the status bar. In iOS6 the status bar area is not part of your top level view, so this change will cause everything to get shifted 20 pixels below the status bar. To account for this you need to set the iOS 6/7 deltas. Set dY on your child view to -20 to move it up 20 pixels when running under iOS6.

However this will shrink your view by 20 pixels, so you need to set dHeight to +20 pixels.

Filed under programming objective-c ios7

0 notes

Objective-C refactoring: Extract function

Extract function is the refactoring I do most frequently. It is easy to do and has a very immediate payoff in reduced code and increased clarity. In this example I will follow the same approach as I would in C++. In both languages I prefer using static free functions over methods. I’ll show why as we progress.

The code we are refactoring is a Objective-C wrapper API around Apple’s C API to the addressbook. The method we are trying to cleanup looks like this:

+ (void)addContact:(ABContact *)contact successBlock:(SuccessBlock)successBlock {
    [self createABRef:^(ABAddressBookRef abRef) {

        ABRecordRef aRecord = ABPersonCreate();
        CFErrorRef *anError = NULL;

        ABRecordSetValue(aRecord, kABPersonFirstNameProperty, (__bridge CFTypeRef)(contact.firstName), anError);
        ABRecordSetValue(aRecord, kABPersonLastNameProperty, (__bridge CFTypeRef)(contact.lastName), anError);
        ABRecordSetValue(aRecord, kABPersonOrganizationProperty, (__bridge CFTypeRef)(, anError);

        ABMutableMultiValueRef multiPhone = ABMultiValueCreateMutable(kABMultiStringPropertyType);
        ABMultiValueIdentifier multivaluePhoneIdentifier;
        bool didAdd;

        for (int i = 0; i < contact.phoneNumbers.count; i++) {
            didAdd = ABMultiValueAddValueAndLabel(multiPhone, (__bridge CFTypeRef)(contact.phoneNumbers[i]), [self phoneLabelWihtIndex:i], &multivaluePhoneIdentifier);

        ABRecordSetValue(aRecord, kABPersonPhoneProperty, multiPhone, anError);


        ABMutableMultiValueRef multiEmail = ABMultiValueCreateMutable(kABMultiStringPropertyType);
        for (int i = 0; i < contact.emails.count; i++) {
            didAdd = ABMultiValueAddValueAndLabel(multiEmail, (__bridge CFTypeRef)(contact.emails[i]), [self genericLabelWihtIndex:i], NULL);
        ABRecordSetValue(aRecord, kABPersonEmailProperty, multiEmail, anError);


        ABMutableMultiValueRef multiUrl = ABMultiValueCreateMutable(kABMultiStringPropertyType);
        ABMultiValueIdentifier multivalueUrlIdentifier;
        for (int i = 0; i < contact.urls.count; i++) {
            didAdd = ABMultiValueAddValueAndLabel(multiUrl, (__bridge CFTypeRef)(contact.urls[i]), [self urlLabelWihtIndex:i], &multivalueUrlIdentifier);
        ABRecordSetValue(aRecord, kABPersonURLProperty, multiUrl, anError);

        NSData *dataRef = UIImagePNGRepresentation(contact.image);
        ABPersonSetImageData(aRecord, (__bridge CFDataRef)dataRef, nil);

        ABAddressBookAddRecord(abRef, aRecord, anError);
        ABAddressBookSave(abRef, anError);

        BOOL result = (anError ? NO : YES);

You probably notice that you can’t see at a glance what is going on here or what is being repeated. But if you look carefully at the for loops you will see that the same pattern of code is repeated 3 times in this function.

Here is an example of the chunk of code which gets duplicated:

    ABMutableMultiValueRef multiEmail = ABMultiValueCreateMutable(kABMultiStringPropertyType);
    for (int i = 0; i < contact.emails.count; i++) {
        didAdd = ABMultiValueAddValueAndLabel(multiEmail, (__bridge CFTypeRef)(contact.emails[i]), [self genericLabelWihtIndex:i], NULL);
    ABRecordSetValue(aRecord, kABPersonEmailProperty, multiEmail, anError);


This same chunk of code is being used to fetch email, urls and phone numbers and add them to a record in our addressbook.

Why should we factor out this code?

This constitutes 5 lines of code which gets repeated with different parameters. Sometimes it is far more lines. But what is common to all of them is that you don’t immediatly see if it is the exact same code. There could be minor variations you don’t see. This makes code slower to read, than if this was just one function call. With one function call you would know that the same code was being run in each case.

This is also important for bug fixes. You discover a bug, but don’t notice that the exact same chunks of code is used elsewhere and thus your bugfix is only applied in one place. Now your bugfix actually makes matters worse. Because the next time somebody comes looking at that code, all those copy pasted chunks of code look different, because one has a bug fix in it. That person is now less likely to identify the possibility for a refactoring, or that yet another bugfix needs to be applied in multiple places.

Reduce coupling, replace methods with free functions

When factoring out our function we want it to depend on as few objects and other functions as possible. This will make it easier to test our new function and reason about it. Our function does not depend on any methods or instance variables except there is a [self genericLabelWithIndex:i]. Looking at this method definition we can see that it does not need to be a method at all. We can make it a regular C function. This is the implementation:

+ (CFStringRef)genericLabelWithIndex:(NSUInteger)index{
    switch (index) {
        case 0: return kABHomeLabel;break;
        case 1: return kABWorkLabel; break;
        default: return kABOtherLabel; break;

Since it doesn’t depend on any instance variables or other methods calls we can change it into a static function:

static CFStringRef genericLabelWithIndex(NSUInteger index) {
    switch (index) {
        case 0: return kABHomeLabel;break;
        case 1: return kABWorkLabel; break;
        default: return kABOtherLabel; break;

You’ll see how this will helps us in a moment.

Extract function

Now we extract our function from the duplicated code. You can notice one unusual type LabelForIndex in the function signature.

static void AddMultiValuePropertyToRecord(ABRecordRef aRecord, ABPropertyID propID, NSArray *values, LabelForIndex labelForIndex, CFErrorRef *anError) {
    ABMutableMultiValueRef multiValue = ABMultiValueCreateMutable(kABMultiStringPropertyType);
    ABMultiValueIdentifier multiValueID;
    for (int i = 0; i < values.count; i++) {
        ABMultiValueAddValueAndLabel(multiValue, (__bridge CFTypeRef)(values[i]), labelForIndex(i), &multiValueID);
    ABRecordSetValue(aRecord, propID, multiValue, anError);

This is a typedef I have introduced to make it possible to pass functions finding correct label as a function pointer:

typedef CFStringRef (*LabelForIndex)(NSUInteger index)

This matches exactly the signature of our genericLabelWithIndex. That allows us to pass the function which gets the appropriate label in the function call, otherwise we would have had to extract 3 different function to make 3 different function calls for getting the labels. Email, phone numbers and URLs all use a different method for getting labels. Now you can see why it made sense to change these methods to static functions. Static functions can be passed to other functions. We could not have passed the method selector without also passing the instance or class objects which would have introduced an extra dependency in our function.

Replace duplicated code with function calls

So with this function extracted we can replace the following code from our example:

ABMutableMultiValueRef multiPhone = ABMultiValueCreateMutable(kABMultiStringPropertyType);
ABMultiValueIdentifier multivaluePhoneIdentifier;
bool didAdd;

for (int i = 0; i < contact.phoneNumbers.count; i++) {
    didAdd = ABMultiValueAddValueAndLabel(multiPhone, (__bridge CFTypeRef)(contact.phoneNumbers[i]), [self phoneLabelWihtIndex:i], &multivaluePhoneIdentifier);

ABRecordSetValue(aRecord, kABPersonPhoneProperty, multiPhone, anError);


ABMutableMultiValueRef multiEmail = ABMultiValueCreateMutable(kABMultiStringPropertyType);
for (int i = 0; i < contact.emails.count; i++) {
    didAdd = ABMultiValueAddValueAndLabel(multiEmail, (__bridge CFTypeRef)(contact.emails[i]), [self genericLabelWihtIndex:i], NULL);
ABRecordSetValue(aRecord, kABPersonEmailProperty, multiEmail, anError);


ABMutableMultiValueRef multiUrl = ABMultiValueCreateMutable(kABMultiStringPropertyType);
ABMultiValueIdentifier multivalueUrlIdentifier;
for (int i = 0; i < contact.urls.count; i++) {
    didAdd = ABMultiValueAddValueAndLabel(multiUrl, (__bridge CFTypeRef)(contact.urls[i]), [self urlLabelWihtIndex:i], &multivalueUrlIdentifier);
ABRecordSetValue(aRecord, kABPersonURLProperty, multiUrl, anError);

With these 3 calls:

AddMultiValuePropertyToRecord(aRecord, kABPersonPhoneProperty, contact.phoneNumbers, phoneLabelWithIndex, anError);
AddMultiValuePropertyToRecord(aRecord, kABPersonEmailProperty, contact.emails, genericLabelWithIndex, anError);
AddMultiValuePropertyToRecord(aRecord, kABPersonURLProperty, contact.urls, urlLabelWithIndex, anError);

Now we can clearly see that phone numbers, urls and emails are added the same way, and it is also clear what parameters the adding depends on. We pass 3 different functions with the same signature for getting labels:

CFStringRef phoneLabelWithIndex(NSUInteger index)
CFStringRef genericLabelWithIndex(NSUInteger index)
CFStringRef urlLabelWithIndex(NSUInteger index)


When removing code duplication through method or function extraction it is preferable to first see if it is possible to do so using a regular static function rather than a method. Also see if you can change dependent methods into static functions, since that allows easier composition and better decoupling than using methods.

Filed under objective-c refactoring

0 notes

Keep it simple: Don’t be Framework happy

In the Objective-C world with Cocoapods it is extremely easy to pull in small third party libraries. There is a library for almost any kind of imaginable thing. That however does not mean you should pull in a library for every task you need to do.

I recently took over development of a 10 000 line app which used about 20 different third party libraries. An interesting experience from this is that pulling in that much libaries instead of writing your own code often neither:

  • Require less coding
  • Ease maintenance
  • Or is easer to do

I had two parts of the appliation using third party libraries where I replaced it with my own handwritten code.

Storage of cached data

The app talks to a REST server and caches some data. The original version used Magical Record which is a thin layer on top of Core Data. Now that might seem like an okay idea until you realize the app only has two entities which are not related and we only need to store miniscule amounts of data.

Core Data is there to simplify managing large amounts of data you can not read in and out of memory in one big chunk without getting serious performance problems. The data being stored in this app was about 40 KB if we used an extremely verbose XML format. In binary it would probably be just a few hundred bytes. Hardly something that requires something as complex as Core Data.

I threw out the use of Magical Record and replaced it with storage based on NSKeyedArchiver. Excluding the code for Magical Record that reduce my code by half. Meaning if my Magical Records related code added about 100 lines, then my new NSKeyedArchiver code was written in about 50 lines of code.

There are several benefits to this:

  1. I now have less code to maintain.
  2. NSKeyedArchiver code is quite straightforward and easy to understand and debug.
  3. It is easy to evolve the storage. Adding and removing data fields is much easier than with Core Data.
  4. I now have one less library dependency.

On the fly validation of input text

A common issue in iOS apps is to have some on the fly validation of the text users input, e.g. making sure that they write valid email or phone numbers. PMValidation is a fancy text validation library that was used in the app. It allows you to string together different text validation modules etc. But again our requirements were quite modest, we just needed to validate email, phone number, zip code, first name and last name.

Fancy implementation is often not a good thing. In Unix programming tradition emphasis is placed on simplicit of implementation. And I could see the value of that when dealing with PMValidation. PMValidation would use an elaborate system of concurrently computer text validations using Grand Central Dispatch. That would sometime go wrong and prove very tricky to debug. After spending a lot of time trying to figure out what was going wrong, I asked myself how hard is text validaton anyway and why the hell do we need to do it on separate threads?

So I threw out PMValidaton and wrote my own simple validation code with no multiple threads. Again I ended up with half has much code without counting the library itself. Because the code was simple I basically wrote a couple of throw away classes which would validate and visually indicate valid text in different ways. The validation itself was done in a baseclass TLTextValidator implementing the UITextFieldDelegate delegate method:

- (BOOL)            textField:(UITextField *)textField 
            replacementString:(NSString *)string {

    NSString *resultString = 
        [textField.text stringByReplacingCharactersInRange:range withString:string];
    [self setVisualState:[self isTextValid:resultString]];

    return YES;

Checking if text is valid is just a simple regexp:

- (BOOL)isTextValid:(NSString *)text {
    NSInteger num_matches = [_regexp numberOfMatchesInString:text
                                                       range:NSMakeRange(0, text.length)];
    return num_matches == 1;

Then using the template method pattern I had two subclasses TLRedBorderValidator and TLRedLabelValidator implement the setVisualState: method, so that depending on whether text was valid or not it would highlight that in two different ways depending on chosen text validator subclass. Using the fancy off the shelf PMValidation the app was using these 7 lines for each text field I was validating:

PMValidationRegexType *val_name_regex = [PMValidationRegexType validator];
val_name_regex.regexString = kRERegexUserName;
NSSet *valtypes = [NSSet setWithObjects:val_name_regex, nil];
PMValidationUnit *name_unit = [self.validationManager registerTextField:self.nameTextField
[self.nameTextField registerWithValidationUnit:name_unit];

By rolling my own classes I cut that down to 3 lines for each field:

_nameValidator = [[TLRedBorderValidator alloc]


Libraries often not only increase the complexity of your code but it also easily ends up requiring you to write more code which is harder to understand and maintain. The is exactly the opposite of why you want to use a library in the first place.

So before you discover that you need feature X and then right away go searching github for a library that implements X. Ask yourself how complex and involved is it to create this feature from scratch with the libraries I already have?

Filed under ios cocoa objective-c refactoring

1 note

Common misconception about static typing in Objective-C

When looking at the Objective-C code of people who have gotten into Objective-C the last years it is not uncommon to see that they have learned to use all kinds of advance stuff like KVO, method swizzling and Core Data.

Yet most people don’t seem to be aware something as fundamental as how the type system works in Objective-C.

Consider this example code of a closure:

[anObj doStuff:^id(id obj) {
    NSDictionary *dict = (NSDictionary *)obj
    MyClass *myObj = [[MyClass initWithDictionary:dict]];
    [myObj doMoreStuff];
    return myObj;

For people comming from a Java or C++ background this makes perfect sense. Except Objective-C isn’t really statically typed like Java/C++. It is dynamically typed with type annotations. That is not strictly true, but helps to think about it that way.

So first of all, casting is typically pointless. So you could change:

NSDictionary *dict = (NSDictionary *)obj


NSDictionary *dict = obj

To take that further we could just let initWithDictionary take obj directly:

MyClass *myObj = [[MyClass initWithDictionary:obj]];

Since there is no need of a cast. The second point to make about this code snippet is that because we are dealing essentially with type annotations, we can freely change the types in method calls. That does not change the method. At runtime Objective-C does not see what objects we pass to a function or method. It only sees that a object is passed. Obviously this does not apply to primitive types such as int, float, char etc.

So we can change the code example to this:

[anObj doStuff:^(NSDictionary *dict) {
    MyClass *myObj = [[MyClass initWithDictionary:dict]];
    [myObj doMoreStuff];
    return myObj;

And it compiles to exactly the same code. So what is the point of static types in Objective-C then? It is mainly for the benefit of the programmer. By using static types the compiler can warn you about possible wrong usage of on object. It also helps communicate to the reader of the code what type of object you expect to get passed to your function/method.

You will have experienced this when creating actions or outlets in interface builder. When connecting to your code you can chose the type which should be used for sender. It could be id, UISlider or whatever. It doesn’t really matter to the compiler.

Filed under objective-c programming types

0 notes

Cool things you can do in Julia

So if you are one of those cool kids meta programming in Ruby or you love all the tools, libraries and clarity of Python, why would you consider Julia. The Julia website does go into a lot of reasons why you should consider having a look at Julia, so I am not going to reiterate those here but instead give my own personal take on what I like.

A lot of this comes down to what you think about type systems. While I tend to favor dynamic languages I am not a purist. I did notice quickly how nice it was to work in Google Go comming from Python. Now I really dislike typing in languages such as C++ and Java, because I feel it always gets in my way and so much code is spent juggling around the type system.

On the other hand the downside I see with Python and Ruby is not just that they are fairly slow, but I actually quite like to specify type now and then in my programs. That is not because I think that static typing is crucial to catch all kinds of important bugs. But I believe using types helps readability.

Here is an example from C++ of copying the values of a dictionary to an array:

    std::back_inserter( v_T ), 
        &std::map< std::string, T >::value_type::second, 
        _1 ) );

It is hard to see at a glance what is going on because there is so much noise comming from spelling out to the compiler everything it needs to know about the types of everything. If you do this in Julia (similar for Python) it would look like this:

array = [value for (key, value) in dict]

You can quickly see what is going on because the code is not riddled with type information. However we should not ignore static typing completely. Consider this Ruby code from previous blog entry:

class Facility
  def initialize
    @occupier = nil
    @duration = 0

If I am to reason about this code I have a number of issues. I have no idea what sort of object goes into @occupier. I will have to locate some function which puts somethig into @occupier and see what the type of the object is there. Further I can’t know for sure whether duration is a continous or discrete value. At work I mostly look at Objective-C code which usually has some static type information. However collections are not annoted with what objects are stored in them. When I analyze some new code I usually look at the class definitions of the main objects used in the application to get an overview of what is going on. A common stumbling block is that Objetive-C collections don’t tell what type of objects they contain.

That is something I like about Julia. If you look at the definition below:

type Facility
   occupier::Union(Person, Nothing)

   Facility() = new(nothing, 0)

You can see that it tells you a lot more than the Ruby example. Duration is a discrete number and the occupier of the Facility are objects of type Person.

Multiple Dispatch

Julia is not object oriented. And I think that is a good thing. For the things I usually work with I find it constraining and unatural to model the world as if methods must always belong to one particular object. Polymorphis lets you chose what concrete method to call at runtime depending on the type of the instance you invoke the method on. But limiting yourself to deciding this based on one object is an artificial constraint. E.g. when programming a C++ game engine you might want to calculate whether two geometric objects in your scene are overlapping. You don’t know what type they are at runtime so you can’t use C++ templates. So what you end up with is some convoluted way of doing it where each of your geometric objects need to implement a collide method which gets dispatched at runtime. While you have another method overloaded for each concrete geometry type:

bool CircleShape::collide(Shape* other)
  if (!boundingBox().intersect(other->boundingBox()))
    return false;

  return other->intersection(this->circle);

Don’t worry you don’t really have to get the details of this. With Julia I could simply write a bunch of:

function collide(me::Circle, other::Rectangle)
function collide(me::Polygon, other::Circle)
function collide(me::Polygon, other::Rectangle)

And at runtime when I call:

collide(me, other)

It would dispatch to the correct implementation, because in Julia the vtable is essentially stored on each function rather than on each class. When you call a function in Julia it will lookup in a table at runtime which concrete function it should call based on the types of its arguments. I find that many design patterns such as Visitor just exist because of this flaw in how Object Oriented Programming works. While my example was C++, the problem does not go away just because you chose Ruby or Python instead.

Properly modeling your domain

In geometry a 2D point and a vector are not the same even if they both have an x and y component. You can perform many of the same operations on a point and vector but they do not always work the same. So you need to be able to distinguish them. In Julia we can model them as two distinct types:

type Point

type Vector2D

In geometry if I add a vector to a point I get a new point. But if I add a vector to a vector I get a new vector. I can model this in Julia like this:

+(p::Point,    v::Vector2D) = Point   (p.x + v.x, p.y + v.y)
+(u::Vector2D, v::Vector2D) = Vector2D(u.x + v.x, u.y + v.y)

This would not be possible in Ruby because it would not be able to distinguish between a Point and Vector2D argument. You would have to give each function a different name and keep track of when and where you are giving a point or vector as argument. One might ask if it matters. When working in C++ i often model points, vectors and unit vectors with exactly the same class. However e.g. having your own type for unit vectors gives you some neat possibilities. In code it is usually troublesome to keep apart which of your Vector2D objects represent reqular vectors and which are units vectors. Say we define a new type to represent unit vectors:

type Direction

    Direction(v::Vector2D) = (len = norm(v); new(v.x/len, v.y/len))

Unit vector are special because among other things their length is always 1. So while Vector2D would define the following functions as:

norm(v::Vector2D) = sqrt(v.x^2 + v.y^2)
sqrnorm(v::Vector2D) = v.x^2 + v.y^2
unit(v::Vector2D) = Direction(v)

Notice that there is only one constructor for a unit vector, which takes a Vector2D. There is thus no way to make a unit vector which does not have norm 1 (eucledian distance of 1). This allows us to define the previous functions for a unit vector as:

norm(v::Direction) = 1
sqrnorm(v::Direction) = 1
unit(v::Direction) = v  

Thus a unit vector can calculate norm really fast. Thus in code where it does not matter whether you are using a unit vector or a regular vector you can pass in any and perform calculations in the optimal way.

Type Unions

Although you might think all this seems very neat, you might already be thinking that defining 3 different types is going to cause a lot of code duplication because many of the functions you can perform on points, vectors and unit vector are the same. However Julia allows you to use type unions. E.g. the dot product works the same way for unit vectors and vectors. But you don’t need to define it twice. In the implementation you can say you don’t care whether the input is of type Vector2D or Direction:

VecOrDir = Union(Vector2D, Direction)
dot(u::VecOrDir, v::VecOrDir) = u.x*v.x + u.y*v.y

Tuning and Performance

So I covered a lot about why I think Julia is great at modeling your problem. Basically how powefull Julia’s abstraction mechanisms are. The other reason I like Julia is due to the promise of great performance and the ability to tune and reason about the performance of your code. There is a function code_native which lets you look at the assembly code the JIT compiler would generate for a function. With it you have a powerfull tool to inspect potential performance issues with the way you write your code. One thing I was curious about was e.g. whether using union types would be bad for performance because I would essentially box and unbox data into a sort of Variant data type. But checking the code that would be called when both arguments are of type Vector2D we can see that there is no boxing or conversion going on. Just raw calculations:

julia> code_native(dot, (Vector2D, Vector2D))
    .section    __TEXT,__text,regular,pure_instructions
    push    RBP         ; Store stack frame pointer
    mov RBP, RSP        ; Let framepointer point to stack

    ; Move data at address RDI + 8 to 
    ; floating point registers XMM0 and XMM1
    vmovsd  XMM0, QWORD PTR [RDI + 8]
    vmovsd  XMM1, QWORD PTR [RDI + 16]

    ; Multiply double number in XMM1 floating point 
    ; register with number at RSI + 16 store result at XMM1
    vmulsd  XMM1, XMM1, QWORD PTR [RSI + 16]
    vmulsd  XMM0, XMM0, QWORD PTR [RSI + 8]

    ; Add results from both multiplication and store at XMM0
    vaddsd  XMM0, XMM0, XMM1
    pop RBP

If you are familiar with assembly code it sould be clear that Julia has the potential to go almost as fast as C code despite being primarly a dynamically typed language. This sort of optimized code would be much harder to generate for Python or Ruby because the JIT compiler can’t know the type of arguments being passed.

Filed under julialang c++ performance assembly

1 note

Getting comfortable with the Julia programming language

The last couple of days I have been working with the programming language Julia, designed for scientific computing. I have been rewriting the ruby code found in the book “Exploring Everyday Things with R and Ruby” to Julia. Being more used to languages like Ruby and Python it took a little time to get the hang of the Julia way of doing things. Specifically working with collections was different, so here I have made some notes about how to work most conveniently with Julia collections.

Working with Arrays

Make an array with one dimension:

julia> xs = [1,2,3,4]
4-element Array{Int64,1}:

Make a new array by squaring the members, using list comperhension:

julia> sqr_xs = [x^2 for x in xs]
4-element Array{Any,1}:

Note that we got an array of type Array{Any,1}. We know however that in this case, it could more accuratly be described as Array{Int64,1}. However the JIT compiler does not know that. This happens often in Julia. Then you can help it by adding a type assertion.

julia> sqr_xs = [(x::Int64)^2 for x::Int64 in xs]
4-element Array{Int64,1}:

A type assertion is not a casting, but more like both a hint and assertion that the type is what we say it is.

Since all the elements in a Julia array have the same type unlike Ruby or Python we can’t simply write [] to create an empty array. We have to specify the type:

julia> ys = Int64[]
0-element Array{Int64,1}

# Add elements to ys
julia> for x in xs
         push!(ys, x^2)

julia> ys
4-element Array{Int64,1}:

# Remove element from the back
julia> pop!(ys)

# Remove from front
julia> shift!(ys)

julia> ys
2-element Array{Int64,1}:

julia> empty!(ys)
0-element Array{None,1}

Removing elements

The function for removing elements from sets is the same as for splicing new elements into an array:

julia> xs = [5,4,3,2,1]
5-element Array{Int64,1}:

julia> splice!(xs,2)

julia> xs
4-element Array{Int64,1}:

julia> splice!(xs, 2:3, [11, 12])
2-element Array{Int64,1}:

julia> xs
4-element Array{Int64,1}:

Working with Sets

You can create and add and remove elements to an empty set, much the same way as an array:

julia> s1 = Set{Int64}()

julia> push!(s1,2)

However sets don’t have order like arrays, so the order in which you remove elements with e.g. pop! will be random.

julia> s2 = Set{Int64}(1,2,3,4)

julia> pop!(s2)

julia> pop!(s2)

Of course with a set you can quickly remove specific elements:

julia> s2 = Set{Int64}(1,2,3,4)

julia> delete!(s2,3)

Cheat Sheet

Empty array of type T


Empty set of type T


Set with content of array xs

Set{T}(xs...) # to assert elements are of type T

Modify collections

push!(collection, element)
append!(collection, collection)
delete!(set, element)
splice!(array, index[, replacement])

Ruby to Julia

Here are some common patterns in Ruby and how the translate to Julia.

Ruby                        Julia

xs.each {|x|                for x in xs
xs << x                     push!(xs, x)
rand(N)                     rand(1:N)
(1..10).step(2).each {|i    for i in 1:2:10
10.times {|i|               for i in 1:10
{"one" => 1, "two" => 2}    {"one" => 1, "two" => 2}

Dealing with nil when there is no nil

Programming languages such as Python and Ruby will use nil a lot as markers. E.g. in this example we use nil to indicate that the facility as the restroom is empty (example from “Exploring Everyday Things with R and Ruby”).

class Facility
  def initialize
    @occupier = nil
    @duration = 0

Unfortunatly there is not one single ready made solution to this in Julia. The best solution depends on the particular problem. One solution is this:

type Facility
   occupier::Union(Person, Nothing)

   Facility() = new(nothing, 0)

That is we create a Union type, meaning occupier can contain an object which is either of type Nothing or Person. Later you can do a “nil” check with:

is(occupier, nothing)

This might seem like there is no difference between this and nil. However the difference is that any field in Ruby or Python may get the nil value. But in Julia you have to explicitly allow a field to take nothing. Otherwise there is no way that field can hold nothing. If you try to assign nothing to a field of type Person you will get a type exception.

This has implications for how you structure your code, since you know that from a variable of type Person you never need to write checks for nothing to avoid getting a type exception thrown in your face. This is the sort of things that makes Java code riddled with null checks. You would only need to perform this check if the type is Union(Person, Nothing).

Sometimes a better approach than Nothing is to use Null objects. People already do this in Java, C++ etc. That basically means creating one instance of a type which you designate as the Null object. It has its fields set in such a way that you can check whether it is the Null object or you simply compare identity.

An example how you can do this:

type Node{T}
    prev::Node{T}  # ref to previous node
    next::Node{T}  # ref to next node

    function Node()
        n = new()
        n.prev = n = n

isvalid(n::Node) = n != || n != n.prev

# We need this to avoid default display of value in REPL to get an endless recursion
function show(io::IO, n::Node)  # avoids recursion into prev and next
    if isvalid(n)
        print(io, "$(typeof(n))(data = $, prev = $n.prev, next = $")
        print(io, "$(typeof(n)) Null Object")

Filed under julialang programming ruby nil null

1 note

Why do iOS Provisioning Profiles exist?

If you are doing iOS development, you might ask yourself why did Apple make the whole process of signing an app putting it on the app store so bloody complicated and confusing?

You might know all the right steps you have to do to make everything work, but you have no idea what you are doing. That was me some weeks ago. For me in order to understand how something works, it helps to know why it has to be like this.

But before we get into the whys let us look at what it is and isn’t.

At first I thought a provisioning profile was for signing your applications so that iOS would know it was safe to run. Turns out I got it exactly opposite of what it is.

What it is

A provisioning profile is NOT for signing applications. It is a plist (xml format) which is installed on your iOS device. During development applications are signed with the private key in Code Signing Identity (specified in target build settings. Remember from previous post that code signing identity is a private public key pair). When the iOS kernel encounters a signed app, it checks with the installed provisioning profiles whether:

  1. Has the app been signed with a private key, which has a corresponding certificate (public key) in one of the provisioning profiles?
  2. Is the ID of the app in the provisiong profile?
  3. Are we on a device listed in the provisoning profile containing said certificate and app id?

If all of these criteria are fullfilled we can run the application.

Provisioning profiles can be used to run apps in different ways:

  1. Run on device thru xCode (iOS Team Provisioning Profile)
  2. Ad Hoc distribution
  3. App Store
  4. Enterprise build for In-House distribution

The provisioning profile is a PKCS#7 signed plist, signed by Apple. Apple works as the Certificate Authority (CA). Look at my previous post on cryptography if Certificate Authoritydoesn’t make sense to you. A plist is a dictionary in XML form. Under the key DeveloperCertificates the developer certificate is stored in base64 encoding.


Copy data into a file with ending .pem like this:


To the developer certificate stored in a file on PEM format (Privacy Enhanched Email). Inspect content of certificate with:

$ openssl x509 -text -in mycertificate.pem

iOS use this certificate to verify that code has been signed by an appropriate developer. If the application has not been signed with a private key associated with one of the certificates under DeveloperCertificates in the provisioning profile then the app can not run. And just to repeat from earlier: The private key and certificate together makes a so called Code Signing Identity.

Why do we need a provisiong profile?

So back to our original question. Can’t we just sign the apps? Why go through these extra hoops with the provisiong profile?

The reason is that Apple does not want us to be able to distribute the signed apps to any device. That would allow us to circumvent the appstore and Apple’s verification that the app is safe for anybody to use. So Apple wants us to restrict the number of devices we can place our apps on and which apps we can place there.

That is what the provisiong profile is for. We make updates to it through a web interface so Apple can sign it and we can download it. Because it is signed by Apple, iOS can trust it’s content.

How to find out which developers are included in a provisioning profile

Apple will list this on its developer portal. But you can also figure this out yourself.

Typically a team provisioning profile contains a list of certificates for every developer.

In this example we have extracted the XML part from the provisioning profile and stored in a file namedteamprovisioning.plist. Otherwise xpath will not work, because it can not deal with the binary data prefix in the provisioning profile.

for (( i = 1; i <= 16; i++ )); do
    rm cert.pem
    echo -----BEGIN CERTIFICATE----- > cert.pem
    xpath teamprovisioning.plist plist/dict/array/data[$i] | tail -n 34 | head -n 33 >> cert.pem
    echo -----END CERTIFICATE----- >> cert.pem
    openssl x509 -text -in cert.pem | ack Subject

Where in xCode are provisioning profiles configured?

Follow these steps to find which provisioning profile a given target is using.

  1. Select blue icon representing your project in project navigator (folder icon on left most side).
  2. Select your target from the leftmost dropdown menu in view of your selected project icon.
  3. Click Build Settings tab.
  4. Scroll down to the Code Signing header
  5. Here you can see both Code Signing Identify andProvisioning Profile

Your code signing identity will have to have its corresponding certificate in the selected provisioning profile. Remember code signing identity is both public and private key while the certificate stored in the provisioning profileonly contains the public key.

Getting info about how an app has been signed

Apple Root CA, has signed Apple Worldwide Developer Relations Certification Authority which has signed certificateiPhone Distribution: REAKTOR AS, which has been used to actually sign the executable.

This is valid because the certificate signing chain is no longer than three links.

$ codesign -dvvv 
Format=bundle with Mach-O universal (armv7 (12:11))
CodeDirectory v=20100 size=8950 flags=0x0(none) hashes=439+5 location=embedded
Hash type=sha1 size=20
Signature size=4299
Authority=iPhone Distribution: REAKTOR AS
Authority=Apple Worldwide Developer Relations Certification Authority
Authority=Apple Root CA
Signed Time=18. sep. 2013 10:46:08
Info.plist entries=31
Sealed Resources rules=3 files=391
Internal requirements count=1 size=172

This is how the app might look when signed locally. When placed on the appstore the certificate used to sign the executable will be from Apple. So the 3 Authority lines will look like this:

Authority=Apple iPhone OS Application Signing
Authority=Apple iPhone Certification Authority
Authority=Apple Root CA

I verified this by copying the .ipa package for dragonbox, changed suffix to .zip and unpacked it. Then I ran:

codesign -dvvv DragonBox+\ 1.1.1/Payload/ 

Filed under iOS provisioningprofile xCode codesigning codesign openssl

0 notes

Cryptography crash course

Whether you do web development, worry about the NSA, want to understand bitcoins or understand provisioning profiles in iOS development it helps to have a basic understanding of cryptography and what the different terminology means.

My intention with this post is to give you an overview in a compressed way, so you can use this as a sort of reference later.

Cipher text

Plain text is encrypted using an algorithm called acipher and a cryptographic key. This results in acipher text.


Data is transformed using a hash function into something we can either call:

  • A hash
  • Digital fingerprint
  • Message digest

hash function is different from a cipher in that it only goes one way. Multiple messages may be turned into the samedigital fingerprint. If we only need to decide whether two messages are equal we can compare just the hashes if there exists a sufficiently large number of hashes so that collision is practically impossible.

E.g. an attacker can’t trivially create a bogus message with the same hash as the valid message unless trial and error is performed, which would be impractical if the number of possible hashes is very large.

Hashes are used with digital signatures to compress a message so we only have small amounts of data to encrypt and decrypt.


Given a user A and a Certificate authority (CA) X then a certificate for A issued by X consists of:

  • Identity of A
  • A's public key
  • expiry date
  • signature created with X's private key

A signature is created by hashing the 3 first fields and then encrypting the hash value with the private key of X. If userB wants to validate that he/she has the A's public key then she uses X's public key to decrypt the signature on A's certificate. Then she hashes the 3 other fields and compares this value with the decrypted signature.

For this to work we assume that X has made an effort to make sure that A's public key belongs to A and A is who he/she says she is.

Code Signing Identity

A Certificate and its Private Key together can be referred to as “identity” or “Code Signing Identity”

Certificate level

How secure is the certificate. Meaning how did the CA (Certificate Authority) validate the identity of the owner of a public key. It could have been validated with:

  • Email
  • SMS message
  • Passport (manual process)


X could sign certificate of A with his private key, and then Acould sign certificate of B with his private key.

A VA (Validation Authority) can be used to validate a whole chain.

Filed under cryptography certificate hashing

1 note

Objective-C developers should not forget to use C

While Objective-C is an old language, it is relatively new to most people using it thanks to the popularity of iOS. Having used Objective-C long before iOS existed I have noticed people have a tendecy to code like the do in Java in it or they look upon the Objective-C extensions to C the same way they look at the extensions C++ brought to C.

Both approaches has the unfortunate effect of making people think that C is a lesser citizen in the Objective-C world and they try to shun it as much as possible. As a long time C++ developer I know this makes sense because as a C++ developer you are constantly told that C is inferior and you have to do everything the C++ way. cout instead ofprintfstd::vector instead of plain C arrays etc.

But that is not how it works in the Objective-C world. C complements Objective-C very well. Here I would like to discuss some areas where using more C will enhance your Objective-C code by making it easier to understand, faster and more maintainable.

C arrays

A downside of C arrays, is that we can not pass them around as easily as NSArrays because ARC can’t track them and release them and they don’t carry their size with them. So when you are programming in the largeNSArray is preferable.

However sometimes we just need a local array of fixed size.

Say you want to map an index to a prime number. Not a great example, but that it all I could do from the top of my head. A lot of people would simple create this array like this:

NSArray *mapper = @[@1, @3, @5, @7, @11];

The problem with this is that not only is it inefficient (which is the least of our problems), but can’t enforce anything about type with an NSArray. Your compiler will not know that your mapper object only contains numbers. By using a C array however:

int mapper[] = {1, 3, 5, 7, 11};

We can enforce that the array only contains elements of one type and the compiler will know this.

C functions

The benefit of Objective-C methods is that they can be polymorphic, meaning the exact implementation being invoked is determined at runtime depending on the concrete class of the instance we send a message to. That is great for object oriented programming.

But that also has its downsides. One is that the compiler can’t know at compile time whether a method will be called or not.

A benefit of plain C functions is that they can be used for functional programming. Meaning they are suitable for indicating to the caller that the output only depends on the input arguments. Secondly the compiler can determine at compile time whether a C function is used or not. That means the compiler can warn you about unused functions.

Maintainable code

Together these features makes it possible to make your code easier to read an more maintainable.

  1. The compiler will warn you about unused C functions so you can remove them. Ergo less code to maintain.

  2. Reasoning about a snippet of code calling C functions will be easier than code based on messaging, because you know exactly what code will be called and what goes into the function and what comes out.

With respect to point 2, there is of course nothing preventing you from modifying global data in a function. But you can make a convention to only work on inputs with C functions. You can’t make that convention methods, because methods have to be able to modify the instance they are attached to.

How to use in practice

For most classes you implement there are probably few methods which need to be public. Private methods will often be better implemented as static C functions defined inside your .m files.

Let us use AFNetworking classAFHTTPRequestOperation as an example. It needs to find a string from an index set. Rather than implement that as a method, it is defined as a C string:

static NSString *AFStringFromIndexSet(NSIndexSet *indexSet) {
    // implementation code

@implementation AFHTTPRequestOperation
    // methods go here

As you see, functions are placed outside@implementation or any other Objective-C specific keyword groupings. By prefixing static the function will be private. It will not be known outside the .m file where it was defined.

Use structs instead of classes

Objective-C classes are great because they allow us to attach methods to them, and ARC can help us with memory usage. However they also represents a bunch of overhead. Accessing class members is slower than accessing struct members, and they take up slighly more space. Structs also requires much less code to define and use.

So for small dumb objects such as points, vectors, ranges etc, it is usually better to use structs. Graphics programming will use lots of points and vectors. If all of those were Objective-C objects that would cause a lot of overhead. That is why we have NSPointNSSize etc as structures.

By dumb objects I mean objects you are not likely to attach methods to. If you made them in Objective-C you would probably only have properties on them. The objects should typically just contain 2 - 6 member variables which are either primitive types such as ints or floats or other structs consisting of primitive types. If they need to contain Objective-C types such as NSString you have to use objects, because ARC can not deal with Objective-C objects in structs.

Apple uses plenty of structs. They are not second class citizens. Use them if it makes sense. Don’t avoid them just because they seem less object oriented. Object oriented programming is used to create more maintainable and readable code, not an end goal in itself. Sometimes non object oriented approaches improves readability and maintanability. There is no silver bullet.

Filed under C objective-c programming functions arrays methods