Pseudo Associative

Random thoughts on software development

2 notes

My Predictions about Swift 4 years ago

In 2010 when development of Swift started, I was contemplating the future of Mac development on this blog.

My predictions was that Apple would not adopt an existing language, but build their own. I thought about whether they would make a new managed language platform like Microsofts .NET, but I concluded that was unrealistic:

Creating something similar to .NET is unthinkable. It is a huge undertaking and Apple is too far behind to compete with established platforms like Java and .NET.

But Apple had other strengths I concluded that they could utilize:

Cocoa is a very well designed framework and the Objective-C runtime allows fairly easy interaction with other dynamic languages. Microsoft was never in such a fortunate position. Creating a COM object was far more complicated than creating an Objective-C class. That made .NET more of a necessity

Microsoft mainly did their development in C/C++ using COM objects. But that was a much poorer basis for a language than Objective-C as COM had no concept of inheritance and is not message based. Win32 wasn’t really anything to build the next generation language on. So they had to create .NET as entirely new platform. Of course that also had the benefit of making them much freer in how to design .NET.

Apple did not have that freedom, but they had Objective-C and Cocoa, which was actually possible to build further on. Due to the message passing model it was a lot easier to integrate Objective-C with another language than C or C++.

Apple has gotten a lot of criticism for sticking with a fairly unknown language in the industry like Objective-C. Many are of the opinion that C++ would have served them better. I think Swift proved the doubters wrong. C++ with its extreme complexity is an extremely difficult language to bridge to other languages. It would have been a practically impossible task to build something like Swift on top of a C++.

It was the rising popularity of languages such as Scala and Clojure on the Java platform which made me see how alternative languages had a chance if well designed. Playing with the Nu created by Tim Burks was what convinced my that what Apple should and would do was to create an entirely new programming language build on top of the Objective-C runtime (object model). Tim Burks elaborated on the complexities of creating RubyCocoa which was trying to allow you to use Cocoa using Ruby. Because the semantics and practices of Objective-C and Ruby are quite different it is very hard to get a satisfying integration.

Apple’s previous failures trying to adopt existing popular languages

Using an existing language had been tried multiple times by Apple. Best known was their attempt to replace Objective-C with Java. In the early 2000s Java was made a first class citizen and Apple’s flagship product for creating server side software WebObjects was rewritten from Objective-C to Java. Apple even tried to make Python and Ruby first class citizens. PyObjC e.g. was included with the Mac OS X 10.5 Leopard release in 2007. However none of these alternative languages gained much popularity. To me as a Objective-C/Cocoa developer it was plain to see why. Having to write:

myLittleDuck.makeSomeNoise_eyesClosed_onOneFoot_(quack, True, True)

Instead of:

[myLittleDuck makeSomeNoise:quack eyesClosed:@YES onOneFoot:@YES];

Was not attractive. It look messy and unreadable. Java, Ruby and Java didn’t have syntax which meshed well with the Smalltalk inspired syntax in Objective-C. Java was even worse not being dynamic many common patterns in Cocoa didn’t work. Java developers have also ever since MS tried to adopt it for Windows development, been critical of using non cross platform frameworks for development like Cocoa. Apple thus effectively abandoned existing alternative languages after they failed in the market place (Java bridge deprecated in OS X 10.4). This was my thinking that lead me to conclude in my blogpost 4 years ago:

My prediction is that if Apple has a new more modern language in the works for rapid application development on Mac, it will be based on LLVM and the Objective-C runtime like MacRuby, but will take the same approach as Nu in that it will be designed from the ground up to work with Objective-C and complement it.

Swift features I correctly predicted

With respect to language features and general direction of language I got some things wrong but I’d say I got a lot right:

  1. Built on top of Objective-C runtime model.
  2. Using LLVM (but I wrongly assumed it would be JITed).
  3. Type inference using static typing.
  4. Closures.
  5. Native support for Objective-C named parameters.
  6. More functional
  7. Emphasis on shorter and simpler code as found in Go, Ruby, Python etc. In Swift they dropped parenthesis in if, while, switch and for loops.
  8. No fundamental new features not seen in existing languages.

I think the last point is important, because I’ve seen comments about Swift were people criticize it for not bringing anything fundamentally new to programming. It was obvious 4 years ago and it should be obvious today as well that you don’t bring in new experimental features in a programming language for industry. Swift only contains features which have already proven themselves.

What I got wrong

JITing

Swift isn’t JITed as I suspected. Knowing the iOS security model I would not have predicted JITing today, as it makes checking signatures of running code impossible and requires code pages in memory to be read and write. But I didn’t know anything about this in 2010.

Native Concurrency support

No native support of Concurrency. I actually think my predictions will be right eventually with this regard. It is just that this will be added later. Having GCD syntax for concurrency might not be first priority.

Garbage collection

In my defense ARC didn’t exist (released 2011) when I made my prediction and nobody had expected it to work as well as it did. Certainly not Apple otherwise they would not have created GC for Objective-C and then deprecating it a few years later.

Multi language platform

I thought LLVM might get some higher level abstractions to work a bit more like .NET as a platform for multiple languages. I don’t believe this will happen anymore. For a couple of reasons:

  1. The promise of .NET never really panned out. Few projects ever used multiple languages. C# completely dominated. Adding other languages proved harder than expected. E.g. .NET was made specifically for statically typed languages to adding stuff like IronPython was hard. You see the same on the Java platform in that Groovy gets bad performance.
  2. Apple are not really into offering a lot of alternatives. They have very few options for computers and cell phones. They like to find the best approach and stick to that.

Future predictions for Swift

As I said I am certain concurrency will come. It is simply such a big thing now than no modern language is going to not support this.

However there are some things which doesn’t exist which I am certain wont come either. E.g. Exceptions because:

  1. Cocoa is not exception safe.
  2. New languages in recent years don’t seem to embrace exceptions as happily as they once did.
  3. The support for optionals, multiple return arguments and enum types provides many potentially good facilities for handling errors.

I think we will see Apple making changes to Cocoa in the coming years to see error handling more hashed out.

As for peoples usage of Swift, I predict a very rapid adoption. How can I be sure?

  1. A lot of Objective-C developers today came to the platform because they wanted to do iOS development not because they wanted to use Objective-C. As much as I love Objective-C I realize that most people really dislike it. Swift is much more what people are used to and like.
  2. Objective-C doesn’t offer anything that Swift doesn’t do better. Well Objective-C offers seamless integration with C, but that matters just to perhaps less than 10% of developers.
  3. Unlike when .NET was introduced or Go for that matter. There are no new APIs to learn. Okay it is a new language, but pretty much everything else stays the same. This lets developers keep their multiple years of investment into learning Apple API’s. The risk is low.
  4. Apple has clearly said this is the future.

They didn’t say it was a supplement or alternative to Objective-C. From experience we know Apple is serious when they say something is the future. In the past they have shown themselves willing to lose customers going forward according to plan despite protests. They lost a lot of customers in their hardware transitions, Motorola to PowerPC and PowerPC to intel. They lost customers when deprecating Carbon in favor of Cocoa. Even though it meant one of the platforms most prominent products Adobe Photoshop would not get an upgrade.

And finally Swift isn’t just minor improvement like Python 2 to Python 3. Swift is a large improvement, so developers will make the switch because they will see real tangible benefits in terms of productivity and lower bug count.

Filed under swift objc

1 note

Microservices and Unix Philosophy

Over the years I’ve gotten more and more convinced of the Unix philosophy of building systems by composing small components and integrate them through a very generic interface such as text protocol.

Of course the irony of this is that the Unix philosophy is really old, and yet it took me a long time to see its merits. I don’t remember any particular quote by “We are doomed to repeat the mistakes of our forefathers” is the essence.

After I got all excited about the Unix approach and seeing it in practice in how systems like git is built and well as how TextMate is structured as I have written about earlier. I wondered why Server side software is not built that way.

Or rather because I have not worked professionally in this field I don’t know much about what is going on there but when peering over a Java developers shoulder it always looked like they built huge monolithic systems running in a single process. Possibly multiple processing for scaling purposes, but still the same code.

Now I discover they have in fact started building software on the server like that. I came across something called Microservices described very well by Martin Fowler.

Components vs Services

One of the first stubling blocks for me is the jargon used. With Microservices they used the term services a lot. With my background as desktop software developer interested in Game Engines and Computer Graphics, services is not a term we use much.

Why not say component I wonder? Their thinking however is that components are the static parts you compose a system out of. You bundle a bunch of components to build a service, or a service might just consist of one component. But it is when you start running the component and let it respond to requests over the network that it turns into a service.

So one way of looking at it is that services are instances of components. With that perspective I guess you could say that individual unix programs which communicate through pipes also are services, but I believe it is understood that services are more long running. Traditionally a piece of unix software runs for a short time.

The Benefits

With TextMate plugins we saw the benefit of being able to write plugins in any language and reuse them for completely different text editors. For large monolithic desktop software, what made me gravidate to towards the unix approach was:

  1. Being stuck in an old technology: C++. Old libraries and frameworks. Storage done in files rather than databases etc.
  2. Executable being too large to be pratical to run through something like valgrind.
  3. Development got very slow due to high turn over time. It takes time to compile and launch a large executable.

It turns out that on the server side they have many of the same challenge but also some unique ones which makes modularization using something like Microarchitectures beneficial:

  1. They got a whole stack of software: Databases, application servers, JVMS and what not. Complicated software and complicated deployment.
  2. People are typically using webservers all the time. Unlike Desktop apps which I work on uptime is important. If you release code that doesn’t work it affects a lot of people immediately. Desktop users can just revert to a previous version why you fix the software.

So deployment, testing and all that is complicated. But with Microservices they can replace one single service at a time. If it fails it doesn’t bring the whole system down.

And because each service is simple it is easy to write test. Just like with how git is made. You could create script that pushes a bunsh of data into a git command or into a microservices and then you can validate the data comming out.

Why did the modular approach get forgotten?

So Unix is old, why did the unix philosophy have to get reinvented again the Unix philosophy?

I have some personal ideas about this. One is that the lingua franca of Unix the C programming language doesn’t scale very well. It has few abstractions it its disposal, so if you want to build large scalable applications you need to modularize it at the process level.

So in a sense the invention of Object Oriented programming allowed us to replace the process as a component with the class.

On the server side I think it is the practice of continous integration and frequent releases and rollouts which resurected the unix way. There was simply no simple way you could achieve rapid changes with the monolithic approach. So Microservices was an answer to a strong need that developed due to the spread of practices such as continuous integration.

Silver Bullet?

Now of course it easy to turn this into a receipt and say you must always build software this way. I really hate that. I hate when people come and tell you you need to do it this way, and they don’t even know why. It is just that they have stone tablets of 10 commandments of software development and it tells them to alway do like this. E.g. never use goto because Dijkstra sort of said that once. Something taken completely out of context because he lived in a world where goto was used all the time for even basic control structures such as if and for loops.

Anyhoo back to my main point. The linux kernel, developmed by the same guy Linus Torvalds who also happened to make the beautifully modularized software git. He went completely agains this sort of modularization when making the Linux kernel. Everybody believed Microkernels were the way to go back in the early 90s. They are essentially the same idea as microservices. Yet that failed miserably and the monolithic kernel was a resounding success. Somehow the Linux kernel team manages to build a huge monolith and release it frequently.

Of course the kernel isn’t all spagetti. It does have layering and modules internally. But the point is that we have to understand the problems of our domain. It is not given that a modularized approach is going to work in every circumstances.

But I’d wager it would work great in a lot more domains than it is currently being used.

Filed under microservice unix software architecture

0 notes

Get Started Writing TextMate 2 Bundles

TextMate 2 is my favorite editor. In many ways it is the modern decendant of vim and emacs. Like those editors it is very flexible when it comes to extending it while following modern GUI conventions. One of the things I really like is the extension mechanisms. You can create macros and snippets easily. But what I want to talk about in this article is TextMate commands. Commands are external scripts you can invoke from TextMate. The cool thing about these scripts compared to vim and emacs is that you can write them in any language.

However information about how you write these scripts seem to be scattered, so this is my attempt to help you get started writing scripts to extend TextMate 2. This allows you to do all kinds of cool stuff like creating command completion like you find in modern IDEs.

The Unix philosophy of TextMate

TextMate is different from how you make plugins in a lot of IDEs. You don’t create a dynamically loaded library which adheres to some binary interface. The downside of that is that you have to develop your plugin in a particular programming language.

TextMate bundles’s on the otherhand are based on the unix philosophy of creating software by combining small programs which process text.

A TextMate command is a script which receives its input from TextMate through regular environment variables and standard input. TextMate will setup its own environment variables before running your script.

Variable Description
TM_CURRENT_LINE Text found on the line the caret is on
TM_CURRENT_WORD Word at location of caret
TM_SELECTED_TEXT Contains text currently selected in document
TM_LINE_INDEX Zero based position of caret on line
TM_LINE_NUMBER Line number at caret. Counts from one.

There are a lot more environment variables. You can read about them in the Pragmatic Programmers TextMate book. Although it is for TextMate 1 it is very usefull for TextMate 2.

Environment variables are not suited for large amounts of data, so the bulk of data is sent in through STDIN.

For each command you can configure how information about current line, current selection etc is sent to your script.

Configuration of a TextMate Command

TextMate store a configuration file with your command which it reads to figure out how to pass data to it what to do with the data received back through STDOUT. It configuration file might say to replace the current selection with the output from the script or insert the output at the caret. There are a lot of choices.

How to Debug and Develop a Command

Just as when debugging any other program without a dedicated debugger you can use printf style debugging. Anything you send to STDOUT can be shown by TextMate. E.g. you can configure your command to display output from STDOUT in a separate window, tooltip or just insert it right in your document at the caret.

Example of a TextMate Command is Implemented

This example is taken from the bundle which comes for the programming language Go. This bundle provides function and type completion through a command. It is implemented as a ruby script which calls a unix command named gocode which you need to install like any other Go command. gocode can be invoked at the command line:

gocode -f=json --in=foobar.go autocomplete 449

This will look at at the 449th character in the source code file foobar.go and produce a list of possible completions formated in JSON.

Our TextMate command consists of a Ruby script which invokes this command. First the script load a whole bunch of ruby scripts which come bundled with TextMate:

require ENV['TM_SUPPORT_PATH'] + '/lib/ui.rb'

This loads the ui.rb file. The TM_SUPPORT_PATH environment variable will always contain the path to scripts bundled with TextMate to help you make bundles.

Next step is to read in the whole file. This script does not pass the file in through STDIN.

document = []
File.open(ENV['TM_FILEPATH'], "r+") do |file|
  document = file.readlines
end

Since gocode can’t work with line and column number directly which is what TextMate sends our script through environment variables it has to calculate a byte offset like this:

cursor = document[ 0, ENV['TM_LINE_NUMBER'].to_i - 1].join().length + ENV['TM_LINE_INDEX'].to_i

Now we have enough information to execute the gocode command and get a list of completions. This script gets the result in CSV format and not JSON:

output = `$TM_GOCODE -f=csv -in=#{e_sh ENV['TM_FILEPATH']} autocomplete #{cursor}`

output now contains a string formated in CSV with our completion suggestions. The completions are sent to a ruby function provided by TextMate called TextMate::UI.complete. It pops up a list of our choices and let users choice a completion. However complete expects the completions in its own format:

They need to be formated as array of dictionaries with the following keys:

  • display The title to display in the suggestions list
  • insert Snippet to insert after selection
  • image An image name, see the :images option
  • match Typed text to filter on (defaults to display)

Trying to complete on fmt. I get the following CSV back (only first 4 lines):

func,,Errorf,,func(format string, a ...interface{}) error
func,,Fprint,,func(w io.Writer, a ...interface{}) (n int, err error)
func,,Fprintf,,func(w io.Writer, format string, a ...interface{}) (n int, err error)
func,,Fprintln,,func(w io.Writer, a ...interface{}) (n int, err error)

To work with TextMate::UI.complete it needs to be changed to an array of dictionaries like this (just 4 first matches):

{"match"=>"Errorf", "display"=>" Errorf(format string, a ...interface{}) error", "insert"=>"(${1:format string}, ${2:a ...interface{\\}})$0", "image"=>"func"}
{"match"=>"Fprint", "display"=>" Fprint(w io.Writer, a ...interface{}) (n int, err error)", "insert"=>"(${1:w io.Writer}, ${2:a ...interface{\\}})$0", "image"=>"func"}
{"match"=>"Fprintf", "display"=>" Fprintf(w io.Writer, format string, a ...interface{}) (n int, err error)", "insert"=>"(${1:w io.Writer}, ${2: format string}, ${3:a ...interface{\\}})$0", "image"=>"func"}
{"match"=>"Fprintln", "display"=>" Fprintln(w io.Writer, a ...interface{}) (n int, err error)", "insert"=>"(${1:w io.Writer}, ${2:a ...interface{\\}})$0", "image"=>"func"}

TextMate::UI.complete works by spawning a new executable bundled with TextMate called tm_dialog2 found under TextMate.app/Contents/PlugIns/Dialog2.tmplugin/Contents/Resources in the TextMate app. This is called with the following arguments:

tm_dialog2 popup --returnChoice --alreadyTyped '' --additionalWordCharacters _

The array of dictionaries are sent over to the tm_dialog2 process through its STDIN. Before it is sent it is formated as a OS X property list (plist). TextMate comes bundled with a function to_plist attached to array objects to do this.

To test this functionality you can write a simple script which executes this line:

tm_dialog2 popup --suggestions '( { display = law; }, { display = laws; insert = "(${1:hello}, ${2:again})"; } )'

And configure the command to output to caret position. Make sure the output of this command is send to STDOUT. When you run this script a list with two choices will popup at the caret:

  • law
  • laws

If you select law then law will be inserted at caret. If you pick laws then you will get a snippet inserted:

laws(hello, again)

Where you can tab between hello and again. Here is an example of a Julia script which does exactly this:

#!/usr/bin/env julia
tm_dialog = ENV["DIALOG"]

print(readall(`$tm_dialog popup --suggestions '( { display = law; }, { display = laws; insert = "(${1:hello}, ${2:again})"; } )'`))

Open TextMate from HTML document

If you have done iOS development you would be familiar with the concept of URL Schemes. This is not limited to iOS, but also supported on Mac OS X. TextMate registers in its Info.plist (contained in the Application bundle) the URL Scheme txmt. So any URL starting with txmt will cause a method in TextMate to be called which will be passed the URL. Thus any program which handles URLs in OS X can potentially open a TextMate document. E.g. the Safari Web browser or the Unix command open:

$ open "txmt://open?line=14&url=file://filename.txt"

This opens the textfile filename.txt at line number 14. This is the basis for how TextMate uses HTML pages with errors to let you jump to a line number.

Summary

Commands are regular Unix commands which you send and reveive data to through STDIN and STDOUT. Input can be provided by environment variables as well. For each command you provide a configuration file which you can setup with a GUI in TextMate. The configuration file says what data is transfered on STDIN to your command and what is doine with the text output from your script.

Filed under textmate textmate2 julialang unix bundle golang

0 notes

When adapting an iOS6 app for iOS7, you need to move any immediate child of your top level view 20 pixels down to make way for the status bar. In iOS6 the status bar area is not part of your top level view, so this change will cause everything to get shifted 20 pixels below the status bar. To account for this you need to set the iOS 6/7 deltas. Set dY on your child view to -20 to move it up 20 pixels when running under iOS6.

However this will shrink your view by 20 pixels, so you need to set dHeight to +20 pixels.

When adapting an iOS6 app for iOS7, you need to move any immediate child of your top level view 20 pixels down to make way for the status bar. In iOS6 the status bar area is not part of your top level view, so this change will cause everything to get shifted 20 pixels below the status bar. To account for this you need to set the iOS 6/7 deltas. Set dY on your child view to -20 to move it up 20 pixels when running under iOS6.

However this will shrink your view by 20 pixels, so you need to set dHeight to +20 pixels.

Filed under programming objective-c ios7

0 notes

Objective-C refactoring: Extract function

Extract function is the refactoring I do most frequently. It is easy to do and has a very immediate payoff in reduced code and increased clarity. In this example I will follow the same approach as I would in C++. In both languages I prefer using static free functions over methods. I’ll show why as we progress.

The code we are refactoring is a Objective-C wrapper API around Apple’s C API to the addressbook. The method we are trying to cleanup looks like this:

+ (void)addContact:(ABContact *)contact successBlock:(SuccessBlock)successBlock {
    [self createABRef:^(ABAddressBookRef abRef) {

        ABRecordRef aRecord = ABPersonCreate();
        CFErrorRef *anError = NULL;

        ABRecordSetValue(aRecord, kABPersonFirstNameProperty, (__bridge CFTypeRef)(contact.firstName), anError);
        ABRecordSetValue(aRecord, kABPersonLastNameProperty, (__bridge CFTypeRef)(contact.lastName), anError);
        ABRecordSetValue(aRecord, kABPersonOrganizationProperty, (__bridge CFTypeRef)(contact.company), anError);

        ABMutableMultiValueRef multiPhone = ABMultiValueCreateMutable(kABMultiStringPropertyType);
        ABMultiValueIdentifier multivaluePhoneIdentifier;
        bool didAdd;

        for (int i = 0; i < contact.phoneNumbers.count; i++) {
            didAdd = ABMultiValueAddValueAndLabel(multiPhone, (__bridge CFTypeRef)(contact.phoneNumbers[i]), [self phoneLabelWihtIndex:i], &multivaluePhoneIdentifier);
        }

        ABRecordSetValue(aRecord, kABPersonPhoneProperty, multiPhone, anError);

        CFRelease(multiPhone);

        ABMutableMultiValueRef multiEmail = ABMultiValueCreateMutable(kABMultiStringPropertyType);
        for (int i = 0; i < contact.emails.count; i++) {
            didAdd = ABMultiValueAddValueAndLabel(multiEmail, (__bridge CFTypeRef)(contact.emails[i]), [self genericLabelWihtIndex:i], NULL);
        }
        ABRecordSetValue(aRecord, kABPersonEmailProperty, multiEmail, anError);

        CFRelease(multiEmail);

        ABMutableMultiValueRef multiUrl = ABMultiValueCreateMutable(kABMultiStringPropertyType);
        ABMultiValueIdentifier multivalueUrlIdentifier;
        for (int i = 0; i < contact.urls.count; i++) {
            didAdd = ABMultiValueAddValueAndLabel(multiUrl, (__bridge CFTypeRef)(contact.urls[i]), [self urlLabelWihtIndex:i], &multivalueUrlIdentifier);
        }
        ABRecordSetValue(aRecord, kABPersonURLProperty, multiUrl, anError);
        CFRelease(multiUrl);

        NSData *dataRef = UIImagePNGRepresentation(contact.image);
        ABPersonSetImageData(aRecord, (__bridge CFDataRef)dataRef, nil);

        ABAddressBookAddRecord(abRef, aRecord, anError);
        ABAddressBookSave(abRef, anError);

        CFRelease(aRecord);
        BOOL result = (anError ? NO : YES);
        successBlock(result);
    }];
}

You probably notice that you can’t see at a glance what is going on here or what is being repeated. But if you look carefully at the for loops you will see that the same pattern of code is repeated 3 times in this function.

Here is an example of the chunk of code which gets duplicated:

    ABMutableMultiValueRef multiEmail = ABMultiValueCreateMutable(kABMultiStringPropertyType);
    for (int i = 0; i < contact.emails.count; i++) {
        didAdd = ABMultiValueAddValueAndLabel(multiEmail, (__bridge CFTypeRef)(contact.emails[i]), [self genericLabelWihtIndex:i], NULL);
    }
    ABRecordSetValue(aRecord, kABPersonEmailProperty, multiEmail, anError);

    CFRelease(multiEmail);

This same chunk of code is being used to fetch email, urls and phone numbers and add them to a record in our addressbook.

Why should we factor out this code?

This constitutes 5 lines of code which gets repeated with different parameters. Sometimes it is far more lines. But what is common to all of them is that you don’t immediatly see if it is the exact same code. There could be minor variations you don’t see. This makes code slower to read, than if this was just one function call. With one function call you would know that the same code was being run in each case.

This is also important for bug fixes. You discover a bug, but don’t notice that the exact same chunks of code is used elsewhere and thus your bugfix is only applied in one place. Now your bugfix actually makes matters worse. Because the next time somebody comes looking at that code, all those copy pasted chunks of code look different, because one has a bug fix in it. That person is now less likely to identify the possibility for a refactoring, or that yet another bugfix needs to be applied in multiple places.

Reduce coupling, replace methods with free functions

When factoring out our function we want it to depend on as few objects and other functions as possible. This will make it easier to test our new function and reason about it. Our function does not depend on any methods or instance variables except there is a [self genericLabelWithIndex:i]. Looking at this method definition we can see that it does not need to be a method at all. We can make it a regular C function. This is the implementation:

+ (CFStringRef)genericLabelWithIndex:(NSUInteger)index{
    switch (index) {
        case 0: return kABHomeLabel;break;
        case 1: return kABWorkLabel; break;
        default: return kABOtherLabel; break;
    }
}

Since it doesn’t depend on any instance variables or other methods calls we can change it into a static function:

static CFStringRef genericLabelWithIndex(NSUInteger index) {
    switch (index) {
        case 0: return kABHomeLabel;break;
        case 1: return kABWorkLabel; break;
        default: return kABOtherLabel; break;
    }
}

You’ll see how this will helps us in a moment.

Extract function

Now we extract our function from the duplicated code. You can notice one unusual type LabelForIndex in the function signature.

static void AddMultiValuePropertyToRecord(ABRecordRef aRecord, ABPropertyID propID, NSArray *values, LabelForIndex labelForIndex, CFErrorRef *anError) {
    ABMutableMultiValueRef multiValue = ABMultiValueCreateMutable(kABMultiStringPropertyType);
    ABMultiValueIdentifier multiValueID;
    for (int i = 0; i < values.count; i++) {
        ABMultiValueAddValueAndLabel(multiValue, (__bridge CFTypeRef)(values[i]), labelForIndex(i), &multiValueID);
    }
    ABRecordSetValue(aRecord, propID, multiValue, anError);
    CFRelease(multiValue);
}

This is a typedef I have introduced to make it possible to pass functions finding correct label as a function pointer:

typedef CFStringRef (*LabelForIndex)(NSUInteger index)

This matches exactly the signature of our genericLabelWithIndex. That allows us to pass the function which gets the appropriate label in the function call, otherwise we would have had to extract 3 different function to make 3 different function calls for getting the labels. Email, phone numbers and URLs all use a different method for getting labels. Now you can see why it made sense to change these methods to static functions. Static functions can be passed to other functions. We could not have passed the method selector without also passing the instance or class objects which would have introduced an extra dependency in our function.

Replace duplicated code with function calls

So with this function extracted we can replace the following code from our example:

ABMutableMultiValueRef multiPhone = ABMultiValueCreateMutable(kABMultiStringPropertyType);
ABMultiValueIdentifier multivaluePhoneIdentifier;
bool didAdd;

for (int i = 0; i < contact.phoneNumbers.count; i++) {
    didAdd = ABMultiValueAddValueAndLabel(multiPhone, (__bridge CFTypeRef)(contact.phoneNumbers[i]), [self phoneLabelWihtIndex:i], &multivaluePhoneIdentifier);
}

ABRecordSetValue(aRecord, kABPersonPhoneProperty, multiPhone, anError);

CFRelease(multiPhone);

ABMutableMultiValueRef multiEmail = ABMultiValueCreateMutable(kABMultiStringPropertyType);
for (int i = 0; i < contact.emails.count; i++) {
    didAdd = ABMultiValueAddValueAndLabel(multiEmail, (__bridge CFTypeRef)(contact.emails[i]), [self genericLabelWihtIndex:i], NULL);
}
ABRecordSetValue(aRecord, kABPersonEmailProperty, multiEmail, anError);

CFRelease(multiEmail);

ABMutableMultiValueRef multiUrl = ABMultiValueCreateMutable(kABMultiStringPropertyType);
ABMultiValueIdentifier multivalueUrlIdentifier;
for (int i = 0; i < contact.urls.count; i++) {
    didAdd = ABMultiValueAddValueAndLabel(multiUrl, (__bridge CFTypeRef)(contact.urls[i]), [self urlLabelWihtIndex:i], &multivalueUrlIdentifier);
}
ABRecordSetValue(aRecord, kABPersonURLProperty, multiUrl, anError);
CFRelease(multiUrl);

With these 3 calls:

AddMultiValuePropertyToRecord(aRecord, kABPersonPhoneProperty, contact.phoneNumbers, phoneLabelWithIndex, anError);
AddMultiValuePropertyToRecord(aRecord, kABPersonEmailProperty, contact.emails, genericLabelWithIndex, anError);
AddMultiValuePropertyToRecord(aRecord, kABPersonURLProperty, contact.urls, urlLabelWithIndex, anError);

Now we can clearly see that phone numbers, urls and emails are added the same way, and it is also clear what parameters the adding depends on. We pass 3 different functions with the same signature for getting labels:

CFStringRef phoneLabelWithIndex(NSUInteger index)
CFStringRef genericLabelWithIndex(NSUInteger index)
CFStringRef urlLabelWithIndex(NSUInteger index)

Conclusion

When removing code duplication through method or function extraction it is preferable to first see if it is possible to do so using a regular static function rather than a method. Also see if you can change dependent methods into static functions, since that allows easier composition and better decoupling than using methods.

Filed under objective-c refactoring

0 notes

Keep it simple: Don’t be Framework happy

In the Objective-C world with Cocoapods it is extremely easy to pull in small third party libraries. There is a library for almost any kind of imaginable thing. That however does not mean you should pull in a library for every task you need to do.

I recently took over development of a 10 000 line app which used about 20 different third party libraries. An interesting experience from this is that pulling in that much libaries instead of writing your own code often neither:

  • Require less coding
  • Ease maintenance
  • Or is easer to do

I had two parts of the appliation using third party libraries where I replaced it with my own handwritten code.

Storage of cached data

The app talks to a REST server and caches some data. The original version used Magical Record which is a thin layer on top of Core Data. Now that might seem like an okay idea until you realize the app only has two entities which are not related and we only need to store miniscule amounts of data.

Core Data is there to simplify managing large amounts of data you can not read in and out of memory in one big chunk without getting serious performance problems. The data being stored in this app was about 40 KB if we used an extremely verbose XML format. In binary it would probably be just a few hundred bytes. Hardly something that requires something as complex as Core Data.

I threw out the use of Magical Record and replaced it with storage based on NSKeyedArchiver. Excluding the code for Magical Record that reduce my code by half. Meaning if my Magical Records related code added about 100 lines, then my new NSKeyedArchiver code was written in about 50 lines of code.

There are several benefits to this:

  1. I now have less code to maintain.
  2. NSKeyedArchiver code is quite straightforward and easy to understand and debug.
  3. It is easy to evolve the storage. Adding and removing data fields is much easier than with Core Data.
  4. I now have one less library dependency.

On the fly validation of input text

A common issue in iOS apps is to have some on the fly validation of the text users input, e.g. making sure that they write valid email or phone numbers. PMValidation is a fancy text validation library that was used in the app. It allows you to string together different text validation modules etc. But again our requirements were quite modest, we just needed to validate email, phone number, zip code, first name and last name.

Fancy implementation is often not a good thing. In Unix programming tradition emphasis is placed on simplicit of implementation. And I could see the value of that when dealing with PMValidation. PMValidation would use an elaborate system of concurrently computer text validations using Grand Central Dispatch. That would sometime go wrong and prove very tricky to debug. After spending a lot of time trying to figure out what was going wrong, I asked myself how hard is text validaton anyway and why the hell do we need to do it on separate threads?

So I threw out PMValidaton and wrote my own simple validation code with no multiple threads. Again I ended up with half has much code without counting the library itself. Because the code was simple I basically wrote a couple of throw away classes which would validate and visually indicate valid text in different ways. The validation itself was done in a baseclass TLTextValidator implementing the UITextFieldDelegate delegate method:

- (BOOL)            textField:(UITextField *)textField 
shouldChangeCharactersInRange:(NSRange)range 
            replacementString:(NSString *)string {

    NSString *resultString = 
        [textField.text stringByReplacingCharactersInRange:range withString:string];
    [self setVisualState:[self isTextValid:resultString]];

    return YES;
}

Checking if text is valid is just a simple regexp:

- (BOOL)isTextValid:(NSString *)text {
    NSInteger num_matches = [_regexp numberOfMatchesInString:text
                                                     options:0
                                                       range:NSMakeRange(0, text.length)];
    return num_matches == 1;
}

Then using the template method pattern I had two subclasses TLRedBorderValidator and TLRedLabelValidator implement the setVisualState: method, so that depending on whether text was valid or not it would highlight that in two different ways depending on chosen text validator subclass. Using the fancy off the shelf PMValidation the app was using these 7 lines for each text field I was validating:

PMValidationRegexType *val_name_regex = [PMValidationRegexType validator];
val_name_regex.regexString = kRERegexUserName;
NSSet *valtypes = [NSSet setWithObjects:val_name_regex, nil];
PMValidationUnit *name_unit = [self.validationManager registerTextField:self.nameTextField
                                                     forValidationTypes:valtypes
                                                             identifier:@"uname"];
[self.nameTextField registerWithValidationUnit:name_unit];

By rolling my own classes I cut that down to 3 lines for each field:

_nameValidator = [[TLRedBorderValidator alloc]
                   initWithRegexpString:kRERegexUserName
                         validatedField:_nameTextField];

Conclusions

Libraries often not only increase the complexity of your code but it also easily ends up requiring you to write more code which is harder to understand and maintain. The is exactly the opposite of why you want to use a library in the first place.

So before you discover that you need feature X and then right away go searching github for a library that implements X. Ask yourself how complex and involved is it to create this feature from scratch with the libraries I already have?

Filed under ios cocoa objective-c refactoring

1 note

Common misconception about static typing in Objective-C

When looking at the Objective-C code of people who have gotten into Objective-C the last years it is not uncommon to see that they have learned to use all kinds of advance stuff like KVO, method swizzling and Core Data.

Yet most people don’t seem to be aware something as fundamental as how the type system works in Objective-C.

Consider this example code of a closure:

[anObj doStuff:^id(id obj) {
    NSDictionary *dict = (NSDictionary *)obj
    MyClass *myObj = [[MyClass initWithDictionary:dict]];
    [myObj doMoreStuff];
    return myObj;
}];

For people comming from a Java or C++ background this makes perfect sense. Except Objective-C isn’t really statically typed like Java/C++. It is dynamically typed with type annotations. That is not strictly true, but helps to think about it that way.

So first of all, casting is typically pointless. So you could change:

NSDictionary *dict = (NSDictionary *)obj

to:

NSDictionary *dict = obj

To take that further we could just let initWithDictionary take obj directly:

MyClass *myObj = [[MyClass initWithDictionary:obj]];

Since there is no need of a cast. The second point to make about this code snippet is that because we are dealing essentially with type annotations, we can freely change the types in method calls. That does not change the method. At runtime Objective-C does not see what objects we pass to a function or method. It only sees that a object is passed. Obviously this does not apply to primitive types such as int, float, char etc.

So we can change the code example to this:

[anObj doStuff:^(NSDictionary *dict) {
    MyClass *myObj = [[MyClass initWithDictionary:dict]];
    [myObj doMoreStuff];
    return myObj;
}];

And it compiles to exactly the same code. So what is the point of static types in Objective-C then? It is mainly for the benefit of the programmer. By using static types the compiler can warn you about possible wrong usage of on object. It also helps communicate to the reader of the code what type of object you expect to get passed to your function/method.

You will have experienced this when creating actions or outlets in interface builder. When connecting to your code you can chose the type which should be used for sender. It could be id, UISlider or whatever. It doesn’t really matter to the compiler.

Filed under objective-c programming types

0 notes

Cool things you can do in Julia

So if you are one of those cool kids meta programming in Ruby or you love all the tools, libraries and clarity of Python, why would you consider Julia. The Julia website does go into a lot of reasons why you should consider having a look at Julia, so I am not going to reiterate those here but instead give my own personal take on what I like.

A lot of this comes down to what you think about type systems. While I tend to favor dynamic languages I am not a purist. I did notice quickly how nice it was to work in Google Go comming from Python. Now I really dislike typing in languages such as C++ and Java, because I feel it always gets in my way and so much code is spent juggling around the type system.

On the other hand the downside I see with Python and Ruby is not just that they are fairly slow, but I actually quite like to specify type now and then in my programs. That is not because I think that static typing is crucial to catch all kinds of important bugs. But I believe using types helps readability.

Here is an example from C++ of copying the values of a dictionary to an array:

std::transform( 
    m_str_T.begin(), 
    m_str_T.end(), 
    std::back_inserter( v_T ), 
    boost::bind( 
        &std::map< std::string, T >::value_type::second, 
        _1 ) );

It is hard to see at a glance what is going on because there is so much noise comming from spelling out to the compiler everything it needs to know about the types of everything. If you do this in Julia (similar for Python) it would look like this:

array = [value for (key, value) in dict]

You can quickly see what is going on because the code is not riddled with type information. However we should not ignore static typing completely. Consider this Ruby code from previous blog entry:

class Facility
  def initialize
    @occupier = nil
    @duration = 0
  end
end

If I am to reason about this code I have a number of issues. I have no idea what sort of object goes into @occupier. I will have to locate some function which puts somethig into @occupier and see what the type of the object is there. Further I can’t know for sure whether duration is a continous or discrete value. At work I mostly look at Objective-C code which usually has some static type information. However collections are not annoted with what objects are stored in them. When I analyze some new code I usually look at the class definitions of the main objects used in the application to get an overview of what is going on. A common stumbling block is that Objetive-C collections don’t tell what type of objects they contain.

That is something I like about Julia. If you look at the definition below:

type Facility
   occupier::Union(Person, Nothing)
   duration::Int32

   Facility() = new(nothing, 0)
end

You can see that it tells you a lot more than the Ruby example. Duration is a discrete number and the occupier of the Facility are objects of type Person.

Multiple Dispatch

Julia is not object oriented. And I think that is a good thing. For the things I usually work with I find it constraining and unatural to model the world as if methods must always belong to one particular object. Polymorphis lets you chose what concrete method to call at runtime depending on the type of the instance you invoke the method on. But limiting yourself to deciding this based on one object is an artificial constraint. E.g. when programming a C++ game engine you might want to calculate whether two geometric objects in your scene are overlapping. You don’t know what type they are at runtime so you can’t use C++ templates. So what you end up with is some convoluted way of doing it where each of your geometric objects need to implement a collide method which gets dispatched at runtime. While you have another method overloaded for each concrete geometry type:

bool CircleShape::collide(Shape* other)
{
  if (!boundingBox().intersect(other->boundingBox()))
    return false;

  return other->intersection(this->circle);
}

Don’t worry you don’t really have to get the details of this. With Julia I could simply write a bunch of:

function collide(me::Circle, other::Rectangle)
function collide(me::Polygon, other::Circle)
function collide(me::Polygon, other::Rectangle)

And at runtime when I call:

collide(me, other)

It would dispatch to the correct implementation, because in Julia the vtable is essentially stored on each function rather than on each class. When you call a function in Julia it will lookup in a table at runtime which concrete function it should call based on the types of its arguments. I find that many design patterns such as Visitor just exist because of this flaw in how Object Oriented Programming works. While my example was C++, the problem does not go away just because you chose Ruby or Python instead.

Properly modeling your domain

In geometry a 2D point and a vector are not the same even if they both have an x and y component. You can perform many of the same operations on a point and vector but they do not always work the same. So you need to be able to distinguish them. In Julia we can model them as two distinct types:

type Point
    x::Float
    y::Float
end

type Vector2D
    x::Float
    y::Float
end

In geometry if I add a vector to a point I get a new point. But if I add a vector to a vector I get a new vector. I can model this in Julia like this:

+(p::Point,    v::Vector2D) = Point   (p.x + v.x, p.y + v.y)
+(u::Vector2D, v::Vector2D) = Vector2D(u.x + v.x, u.y + v.y)

This would not be possible in Ruby because it would not be able to distinguish between a Point and Vector2D argument. You would have to give each function a different name and keep track of when and where you are giving a point or vector as argument. One might ask if it matters. When working in C++ i often model points, vectors and unit vectors with exactly the same class. However e.g. having your own type for unit vectors gives you some neat possibilities. In code it is usually troublesome to keep apart which of your Vector2D objects represent reqular vectors and which are units vectors. Say we define a new type to represent unit vectors:

type Direction
    x::Float
    y::Float

    Direction(v::Vector2D) = (len = norm(v); new(v.x/len, v.y/len))
end

Unit vector are special because among other things their length is always 1. So while Vector2D would define the following functions as:

norm(v::Vector2D) = sqrt(v.x^2 + v.y^2)
sqrnorm(v::Vector2D) = v.x^2 + v.y^2
unit(v::Vector2D) = Direction(v)

Notice that there is only one constructor for a unit vector, which takes a Vector2D. There is thus no way to make a unit vector which does not have norm 1 (eucledian distance of 1). This allows us to define the previous functions for a unit vector as:

norm(v::Direction) = 1
sqrnorm(v::Direction) = 1
unit(v::Direction) = v  

Thus a unit vector can calculate norm really fast. Thus in code where it does not matter whether you are using a unit vector or a regular vector you can pass in any and perform calculations in the optimal way.

Type Unions

Although you might think all this seems very neat, you might already be thinking that defining 3 different types is going to cause a lot of code duplication because many of the functions you can perform on points, vectors and unit vector are the same. However Julia allows you to use type unions. E.g. the dot product works the same way for unit vectors and vectors. But you don’t need to define it twice. In the implementation you can say you don’t care whether the input is of type Vector2D or Direction:

VecOrDir = Union(Vector2D, Direction)
dot(u::VecOrDir, v::VecOrDir) = u.x*v.x + u.y*v.y

Tuning and Performance

So I covered a lot about why I think Julia is great at modeling your problem. Basically how powefull Julia’s abstraction mechanisms are. The other reason I like Julia is due to the promise of great performance and the ability to tune and reason about the performance of your code. There is a function code_native which lets you look at the assembly code the JIT compiler would generate for a function. With it you have a powerfull tool to inspect potential performance issues with the way you write your code. One thing I was curious about was e.g. whether using union types would be bad for performance because I would essentially box and unbox data into a sort of Variant data type. But checking the code that would be called when both arguments are of type Vector2D we can see that there is no boxing or conversion going on. Just raw calculations:

julia> code_native(dot, (Vector2D, Vector2D))
    .section    __TEXT,__text,regular,pure_instructions
    push    RBP         ; Store stack frame pointer
    mov RBP, RSP        ; Let framepointer point to stack

    ; Move data at address RDI + 8 to 
    ; floating point registers XMM0 and XMM1
    vmovsd  XMM0, QWORD PTR [RDI + 8]
    vmovsd  XMM1, QWORD PTR [RDI + 16]

    ; Multiply double number in XMM1 floating point 
    ; register with number at RSI + 16 store result at XMM1
    vmulsd  XMM1, XMM1, QWORD PTR [RSI + 16]
    vmulsd  XMM0, XMM0, QWORD PTR [RSI + 8]

    ; Add results from both multiplication and store at XMM0
    vaddsd  XMM0, XMM0, XMM1
    pop RBP
    ret

If you are familiar with assembly code it sould be clear that Julia has the potential to go almost as fast as C code despite being primarly a dynamically typed language. This sort of optimized code would be much harder to generate for Python or Ruby because the JIT compiler can’t know the type of arguments being passed.

Filed under julialang c++ performance assembly

1 note

Getting comfortable with the Julia programming language

The last couple of days I have been working with the programming language Julia, designed for scientific computing. I have been rewriting the ruby code found in the book “Exploring Everyday Things with R and Ruby” to Julia. Being more used to languages like Ruby and Python it took a little time to get the hang of the Julia way of doing things. Specifically working with collections was different, so here I have made some notes about how to work most conveniently with Julia collections.

Working with Arrays

Make an array with one dimension:

julia> xs = [1,2,3,4]
4-element Array{Int64,1}:
 1
 2
 3
 4

Make a new array by squaring the members, using list comperhension:

julia> sqr_xs = [x^2 for x in xs]
4-element Array{Any,1}:
  1
  4
  9
 16

Note that we got an array of type Array{Any,1}. We know however that in this case, it could more accuratly be described as Array{Int64,1}. However the JIT compiler does not know that. This happens often in Julia. Then you can help it by adding a type assertion.

julia> sqr_xs = [(x::Int64)^2 for x::Int64 in xs]
4-element Array{Int64,1}:
  1
  4
  9
 16

A type assertion is not a casting, but more like both a hint and assertion that the type is what we say it is.

Since all the elements in a Julia array have the same type unlike Ruby or Python we can’t simply write [] to create an empty array. We have to specify the type:

julia> ys = Int64[]
0-element Array{Int64,1}

# Add elements to ys
julia> for x in xs
         push!(ys, x^2)
       end

julia> ys
4-element Array{Int64,1}:
  1
  4
  9
 16

# Remove element from the back
julia> pop!(ys)
16

# Remove from front
julia> shift!(ys)
1

julia> ys
2-element Array{Int64,1}:
 4
 9

julia> empty!(ys)
0-element Array{None,1}

Removing elements

The function for removing elements from sets is the same as for splicing new elements into an array:

julia> xs = [5,4,3,2,1]
5-element Array{Int64,1}:
 5
 4
 3
 2
 1

julia> splice!(xs,2)
4

julia> xs
4-element Array{Int64,1}:
 5
 3
 2
 1

julia> splice!(xs, 2:3, [11, 12])
2-element Array{Int64,1}:
 3
 2

julia> xs
4-element Array{Int64,1}:
  5
 11
 12
  1

Working with Sets

You can create and add and remove elements to an empty set, much the same way as an array:

julia> s1 = Set{Int64}()
Set{Int64}()

julia> push!(s1,2)
Set{Int64}(2)

However sets don’t have order like arrays, so the order in which you remove elements with e.g. pop! will be random.

julia> s2 = Set{Int64}(1,2,3,4)
Set{Int64}(4,2,3,1)

julia> pop!(s2)
4

julia> pop!(s2)
2

Of course with a set you can quickly remove specific elements:

julia> s2 = Set{Int64}(1,2,3,4)
Set{Int64}(4,2,3,1)

julia> delete!(s2,3)
Set{Int64}(4,2,1)

Cheat Sheet

Empty array of type T

T[]

Empty set of type T

Set{T}()

Set with content of array xs

Set(xs...)
Set{T}(xs...) # to assert elements are of type T

Modify collections

push!(collection, element)
append!(collection, collection)
pop!(collection)
shift!(collection)
delete!(set, element)
splice!(array, index[, replacement])

Ruby to Julia

Here are some common patterns in Ruby and how the translate to Julia.

Ruby                        Julia

xs.each {|x|                for x in xs
xs << x                     push!(xs, x)
rand(N)                     rand(1:N)
(1..10).step(2).each {|i    for i in 1:2:10
10.times {|i|               for i in 1:10
{"one" => 1, "two" => 2}    {"one" => 1, "two" => 2}

Dealing with nil when there is no nil

Programming languages such as Python and Ruby will use nil a lot as markers. E.g. in this example we use nil to indicate that the facility as the restroom is empty (example from “Exploring Everyday Things with R and Ruby”).

class Facility
  def initialize
    @occupier = nil
    @duration = 0
  end
end

Unfortunatly there is not one single ready made solution to this in Julia. The best solution depends on the particular problem. One solution is this:

type Facility
   occupier::Union(Person, Nothing)
   duration::Int32

   Facility() = new(nothing, 0)
end

That is we create a Union type, meaning occupier can contain an object which is either of type Nothing or Person. Later you can do a “nil” check with:

is(occupier, nothing)

This might seem like there is no difference between this and nil. However the difference is that any field in Ruby or Python may get the nil value. But in Julia you have to explicitly allow a field to take nothing. Otherwise there is no way that field can hold nothing. If you try to assign nothing to a field of type Person you will get a type exception.

This has implications for how you structure your code, since you know that from a variable of type Person you never need to write checks for nothing to avoid getting a type exception thrown in your face. This is the sort of things that makes Java code riddled with null checks. You would only need to perform this check if the type is Union(Person, Nothing).

Sometimes a better approach than Nothing is to use Null objects. People already do this in Java, C++ etc. That basically means creating one instance of a type which you designate as the Null object. It has its fields set in such a way that you can check whether it is the Null object or you simply compare identity.

An example how you can do this:

type Node{T}
    data::T
    prev::Node{T}  # ref to previous node
    next::Node{T}  # ref to next node

    function Node()
        n = new()
        n.prev = n
        n.next = n
        n
    end
end

isvalid(n::Node) = n != n.next || n != n.prev

# We need this to avoid default display of value in REPL to get an endless recursion
function show(io::IO, n::Node)  # avoids recursion into prev and next
    if isvalid(n)
        print(io, "$(typeof(n))(data = $n.data, prev = $n.prev, next = $n.next)")
    else
        print(io, "$(typeof(n)) Null Object")
    end
end

Filed under julialang programming ruby nil null

1 note

Why do iOS Provisioning Profiles exist?

If you are doing iOS development, you might ask yourself why did Apple make the whole process of signing an app putting it on the app store so bloody complicated and confusing?

You might know all the right steps you have to do to make everything work, but you have no idea what you are doing. That was me some weeks ago. For me in order to understand how something works, it helps to know why it has to be like this.

But before we get into the whys let us look at what it is and isn’t.

At first I thought a provisioning profile was for signing your applications so that iOS would know it was safe to run. Turns out I got it exactly opposite of what it is.

What it is

A provisioning profile is NOT for signing applications. It is a plist (xml format) which is installed on your iOS device. During development applications are signed with the private key in Code Signing Identity (specified in target build settings. Remember from previous post that code signing identity is a private public key pair). When the iOS kernel encounters a signed app, it checks with the installed provisioning profiles whether:

  1. Has the app been signed with a private key, which has a corresponding certificate (public key) in one of the provisioning profiles?
  2. Is the ID of the app in the provisiong profile?
  3. Are we on a device listed in the provisoning profile containing said certificate and app id?

If all of these criteria are fullfilled we can run the application.

Provisioning profiles can be used to run apps in different ways:

  1. Run on device thru xCode (iOS Team Provisioning Profile)
  2. Ad Hoc distribution
  3. App Store
  4. Enterprise build for In-House distribution

The provisioning profile is a PKCS#7 signed plist, signed by Apple. Apple works as the Certificate Authority (CA). Look at my previous post on cryptography if Certificate Authoritydoesn’t make sense to you. A plist is a dictionary in XML form. Under the key DeveloperCertificates the developer certificate is stored in base64 encoding.

<key>DeveloperCertificates</key>
<array>
    <data>
    MIIFeTCCBGGgAwIBA...
    </data>
</array>

Copy data into a file with ending .pem like this:

-----BEGIN CERTIFICATE-----
MIIFeTCCBGGgAwIBA...
-----END CERTIFICATE-----

To the developer certificate stored in a file on PEM format (Privacy Enhanched Email). Inspect content of certificate with:

$ openssl x509 -text -in mycertificate.pem

iOS use this certificate to verify that code has been signed by an appropriate developer. If the application has not been signed with a private key associated with one of the certificates under DeveloperCertificates in the provisioning profile then the app can not run. And just to repeat from earlier: The private key and certificate together makes a so called Code Signing Identity.

Why do we need a provisiong profile?

So back to our original question. Can’t we just sign the apps? Why go through these extra hoops with the provisiong profile?

The reason is that Apple does not want us to be able to distribute the signed apps to any device. That would allow us to circumvent the appstore and Apple’s verification that the app is safe for anybody to use. So Apple wants us to restrict the number of devices we can place our apps on and which apps we can place there.

That is what the provisiong profile is for. We make updates to it through a web interface so Apple can sign it and we can download it. Because it is signed by Apple, iOS can trust it’s content.

How to find out which developers are included in a provisioning profile

Apple will list this on its developer portal. But you can also figure this out yourself.

Typically a team provisioning profile contains a list of certificates for every developer.

In this example we have extracted the XML part from the provisioning profile and stored in a file namedteamprovisioning.plist. Otherwise xpath will not work, because it can not deal with the binary data prefix in the provisioning profile.

#!/bin/sh
for (( i = 1; i <= 16; i++ )); do
    rm cert.pem
    echo -----BEGIN CERTIFICATE----- > cert.pem
    xpath teamprovisioning.plist plist/dict/array/data[$i] | tail -n 34 | head -n 33 >> cert.pem
    echo -----END CERTIFICATE----- >> cert.pem
    openssl x509 -text -in cert.pem | ack Subject
done

Where in xCode are provisioning profiles configured?

Follow these steps to find which provisioning profile a given target is using.

  1. Select blue icon representing your project in project navigator (folder icon on left most side).
  2. Select your target from the leftmost dropdown menu in view of your selected project icon.
  3. Click Build Settings tab.
  4. Scroll down to the Code Signing header
  5. Here you can see both Code Signing Identify andProvisioning Profile

Your code signing identity will have to have its corresponding certificate in the selected provisioning profile. Remember code signing identity is both public and private key while the certificate stored in the provisioning profileonly contains the public key.

Getting info about how an app has been signed

Apple Root CA, has signed Apple Worldwide Developer Relations Certification Authority which has signed certificateiPhone Distribution: REAKTOR AS, which has been used to actually sign the executable.

This is valid because the certificate signing chain is no longer than three links.

$ codesign -dvvv Systemspill.app 
Executable=/Users/erikengheim/Library/Developer/Xcode/DerivedData/Systemspill-cadtzgsdfjkesuaazsbldkfdgbua/Build/Products/Debug-iphoneos/Systemspill.app/Systemspill
Identifier=no.knowit.systemspill
Format=bundle with Mach-O universal (armv7 (12:11))
CodeDirectory v=20100 size=8950 flags=0x0(none) hashes=439+5 location=embedded
Hash type=sha1 size=20
CDHash=2ee0c12f668b02980e95eb2232944569cd8669bb
Signature size=4299
Authority=iPhone Distribution: REAKTOR AS
Authority=Apple Worldwide Developer Relations Certification Authority
Authority=Apple Root CA
Signed Time=18. sep. 2013 10:46:08
Info.plist entries=31
Sealed Resources rules=3 files=391
Internal requirements count=1 size=172

This is how the app might look when signed locally. When placed on the appstore the certificate used to sign the executable will be from Apple. So the 3 Authority lines will look like this:

Authority=Apple iPhone OS Application Signing
Authority=Apple iPhone Certification Authority
Authority=Apple Root CA

I verified this by copying the .ipa package for dragonbox, changed suffix to .zip and unpacked it. Then I ran:

codesign -dvvv DragonBox+\ 1.1.1/Payload/DragonBoxPlus.app/DragonBoxPlus 

Filed under iOS provisioningprofile xCode codesigning codesign openssl