Pseudo Associative

Named pipes in fish shell

Sun, 19 May 2013 00:47:46 +0200

Most people who have used the Unix command line should be familiar with pipes. E.g. to get the contents of a directory sorted we could pipe the output of ls in as the input of sort:

$ ls | sort

It is easy to think thought that one can only make simple chains using this principle. But in fact one can pipe the output of multiple processes into a single process. How is this possible? There is only one stdin and one stdout! But a process may read multiple files. And on unix a file does not need to represent a file on disk. It could also represent a pipe. In this case it is called a named pipe.

It is worth remember that on Unix, the file system is just a namespace. It is not the same as what is physically residing on your hard drive. By namespace I mean you a hierarchy of names, which allows you to specify a file by a file path. But the individual names in this name space could be connected to anything: one of your hard drives, a USB stick, A CDROM or a chunk of memory on your computer.

Creating named pipes

A named pipe exists in your filesystem, and looks like a regular file. Except it doesn’t exist on your hard drive but in memory in the kernel.

As an example we are going to do the diff of two sorted files. So we are pipeing in the output from two commands, into diff. So first we make two named pipes q and p for diff to read from:

$ mkfifo p q

Next we sort each of the files a.txt and b.txt and send the result into our named pipes:

$ sort a.txt > p; and sort b.txt > q

This will block while waiting for someone to read from the pipes, so we need to open another terminal window to read from p and q.

$ diff p q

Performing process substitution

But it is cumbersome to do it this way each time you need to receive input from the output of multiple commands. So the fish shell has a command psub, perform process substitution.

$ echo (psub)
/tmp/.psub.3492.14844

The stdout of psub is the name of a temporarily created named pipe. On stdin it will read into to temporarily created named pipe.

$ cat (sort a.txt | psub)

This will send a.txt sorted into a named pipe, say /tmp/.psub.3492.14844 and then what is inside () will be replaced with the name of the pipe so we get:

$ cat /tmp/.psub.3492.14844

Meaning the net effect is just to output a.txt sorted. That is just a convoluted way of sorting a text file. This get more useful when dealing with multiple input:

$ diff (sort a.txt|psub) (sort b.txt|psub)

Now we manage to pipe the output of the sorting of two files into diff, without making two named pipes in separate stages and deleting them afterwards.

Fish Shell, an awesome replacement for bash

Sat, 18 May 2013 13:25:06 +0200

Yesterday I installed fish shell. It is a shell just like bash, zsh, tcsh etc. I instantly fell in love with it. It makes using the shell so much easier and faster. The real time saver is how the command completion works. I have used lots of command completion system for the shell but nothing has worked as well as this. Just like in an IDE or in your browser address bar you get suggestions for completions coloured in grey, while what you write is red or blue depending on whether it is legal or illegal command. It also remembers what you have written before and lets you complete on that even if the match is not unique. So this makes it really fast to revisit long paths in the filesystem with just a couple of keyboard taps.

It also lets you complete git commands, because it parses the man pages and figure out what subcommands exist.

It discards a lot of the odd crufty syntax of regular shells from way back. The problem with the syntax of shells such as bash, is that most people don’t write shell programs very often. Remembering the syntax thus becomes hard because it is so odd compared to what it is in regular programming language. This is how switch-case is done in fish shell.

switch (uname)
    case Linux
        echo Hi Tux!
    case Darwin
        echo Hi Hexley!
    case FreeBSD NetBSD DragonFLy
        echo Hi Beastie!
    case '*'
        echo Hi, stranger!
end

Compare that to the oddball syntax used in bash:

case $(uname) in

Linux)  echo "Hi Tux!"
    ;;
Darwin)  echo  "Hi Hexley!"
    ;;
FreeBSD|NetBSD|DragonFLy) echo Hi Beastie!
    ;;
*) echo "Hi, stranger!"
   ;;
esac

It is also very easy to add and edit functions. You can add a function interactively and then save it all on the command line without ever opening a text editor. Editing on the command line is actually quite nice because it does completions, syntax colouring etc.

Making it easier to use third party code in your iOS/OS X apps

Sat, 18 May 2013 12:45:03 +0200

I recently learned about CocoaPods. It is sort of a package manager for Objective-C code. It works a bit similar to Gems in ruby. With it you can place a Podfile in your project and list your dependencies in a special ruby DSL (Domain Specific Language). This is very similar to how Bundler in the ruby world works. For ruby projects you specify your dependencies in a Gemfile which Bundler reads. CocoaPods works the same way.

Below is an example of a Podfile which states that we are using AFNetworking and SBJson

platform:ios

pod 'AFNetworking', '0.10.1'
pod 'SBJson', '2.2.3'

which are two widely used collections handy networking classes for iOS. A benefit of using CocoaPods over just dropping the source code into your project is that you can easily keep up to date on the latest changes to the third party code and it makes it easier for yourself and others to see what is third party code and what is your own code.

Getting it up and running is not without its problems though. It is based on ruby and ruby gems and that can be tricky on OS X. First I tried to install a newer ruby version through rbenv but it would not let me compile ruby 1.9.3 on my mac running OS X 10.8.3. Part of the problem seems to be that OS X uses clang as default compiler while ruby 1.9.3 is setup to compile with gcc.

So I installed ruby through homebrew which is another must have OS X tool. homebrew is a generic package manager for OS X, unlike CocoaPods which is really for third party Objective-C code, not whole programs. The problem with homebrew is that it installs ruby 2.0, which is often require tweaking Gemfiles because they have been setup for ruby 1.9.3. Second problem is that when you install gems such as CocoaPods, links to them are not automatically put in your path. So writing pod setup as suggested in Cocoapods homepage does not work out of the box.

Instead you have to update your links with:

brew unlink ruby && brew link ruby

As a simpler alternative you can install brewbygems which will automatically link your gems binaries to your homebrew bin directory after gem installation.

Trying out Ubuntu 13

Tue, 14 May 2013 01:30:52 +0200

I have been using OS X for a long time now, but today I decided to try out Ubuntu 13. I sort of have warm fuzzy feelings for Linux because it is what got me into Unix in the first place and ultimately OS X. I use a lot of the tools developed on Linux on my Mac.

So I today I downloaded and installed Ubuntu on a USB stick and booted it up on my iMac. It was quite an easy process and it booted surprisingly fast. As a Mac user there is a lot to like. I feel more at home than ever before in Linux. Linux desktops used to take most of the cues from Windows but now the most heavy influence at least for Ubuntu seems to be Mac OS X.

What I liked

Stuff just works now. No configuration of graphics card or X-sever. Wireless network just worked out of the box. Zero time spent on configuration.

When starting to use the OS I was pleasantly surprised by the OS X like dock (or should we say NeXT). The dock is more pragmatic than on OS X. Icons don’t get magnified by default. That was really just in the way. The expose and virtual desktop functionality work better than in OS X at the moment. Ironically they organise virtual desktops how OS X used to do it before they introduced the worthless Mission Control. I used to use Spaces a lot in OS X, until I got Mission Control which has made me almost stop using virtual desktops. I even emailed Apple and begged them to bring spaces back. So thanks Canonical for making the right choice with respect to virtual desktops.

I like how clicking an app icon in the dock triggers expose like functionality for the windows of that app. Better choice than OS X.

Ubuntu has a way of launching apps which reminds me of a combination of Launchpad and Spotlight. It gives you big display like launchpad but also lets you search for files like Spotlight. Very handy. The defaults in the file manager for searching is more sane in that you default to searching below current folder. That is what I usually expect. A global search like on OS X never made sense to me as a default.

App installation and discovery

The Ubuntu software installation program has come a long way. Now it is like a real App Store with reviews, most popular lists, screenshots. You can even buy apps, magazines and all sort of stuff. It can hold its own against Apple’s App Store with respect to navigation, user friendliness and app installation. As an added bonus you are not tied down to one app store and it is all built on top of well known established technology like deb packages.

Configuration

System preferences worked almost exactly like I was used to from OS X. Gone was all the myriad of configuration options for non essential plaguing Linux before. I remember how I could configure everything from the placement of minimise and maximise buttons on the window border. But doing something useful like selecting screen resolution was really hard. Ubuntu today is quite minimalistic and easy to navigate.

Ubuntu One

I really wish Ubuntu one was available on OS X as well like dropbox. It integrates nicer with the system than dropbox. You can select individual directories you want to sync with Ubuntu One rather than keep everything under one directory like dropbox.

What I did not like

I don’t know what it is, but fonts really look a lot better on OS X. It was one of first things I noticed when I used it. It should not be rocket science to fix. Fonts matter because most of your time in front of the screen is spent staring and fonts. I want them to look good. But I am at a loss to say what it is that makes fonts look better on OS X. I mean Ubuntu has anti aliased fonts as well. But something looks wrong somehow.

Hiding of Menu bar

Hiding of menu bar until mouse is over is IMHO really stupid. You can’t use that space for anything anyway and it hinders you from moving your mouse directly to the menu entry you want to hit. It just slows you down for no apparent benefit. In fact I think this must be my biggest complaint.

Inconsistencies

The stuff Canonical makes is generally quite nice looking and consistent. But when you start browsing the app store you quickly notice how all the applications have completely different styles. UI wise they are simply not as clean as your regular Mac OS X app. A lot of the UIs are often clunky, with way too many toolbar icons, often ugly unprofessional looking graphics etc. There are a lot of cool apps though for geeks. I saw a lot of stuff for PCB layout, physics, math etc.

Conclusion

The last couple of OS X releases have not made me that happy. A lot of it has been frivolous stuff I do not care much for. The power users seems to be losing out to an extra emphasis on the needs to the casual users. Mission Control might have simplified by merging several concepts into one, but really it just made the whole tool less flexible and efficient to use. I actually use Launchpad quite a lot but I do not consider it essential. Full screen view for Apps I never really use. I find it mostly inhibits my workflow.

Ubuntu on the other had seems to be moving in the right direction. It has a good mix of simplicity, ease of use, while not ignoring the needs of more advance users. It still has some catching up to do in a number of areas, but at its current state I would be quite happy to use it as my everyday system. I have access to most of the kind of software I would like to use.

I would much rather use Ubuntu than Windows 8, and I can imagine that for most regular users Ubuntu will be quite easy and nice to use.

German Soldiers and Unix worse is better

Sun, 07 Apr 2013 16:52:00 +0200

What does German soldiers during World War II have in common with the UNIX philosophy of “worse is better”?

Serval years ago I read a book by two Swedish authors Michael Tamelander and Niklas Zetterling about D-day. One of the things they study where the assumption that the Allied forces were superior to the Germans because the Allies won WWII. On closer examination this was not true, and neither was it true during World War I. In both wars the Axis powers were defeated due superior production capacity, manpower and resources of the allied powers. The book gives a surprising number of examples how of German forces would win battles despite surprisingly large odds against them.

To understand the superiority of the German forces on the battlefield, the authors look at how military training was done in Germany and how it was done in America and Britain. Allied soldiers were given problems to solve in the classroom. These were usually cases that involved tactics and strategy. The soldiers were given an hour to figure out the optimum solution to a problem on the battlefield. If after the hour had passed, they gave a sub optimal answer there were told to go back and think about the problem longer.

I think this is a good analogy to the MIT software engineering philosophy of doing “the right thing” as described Richard Gabriel. Getting the right solution at the battlefield is the highest priority.

When German soldiers were presented with the same kind of problems in the classroom they were given 7 minutes to solve the same problem. 7 minutes! That is orders of magnitude shorter time. How could their teachers expect them to solve these problems in such a short time?

Simple! They follow the UNIX philosophy of “worse is better”. Their solution didn’t have to be any good. It just had to be adequate. But there was one thing that is more important than anything. They came up with some solution in less than seven minutes. Spending more time than seven minutes was a failure.

The results of this difference in philosophy was catastrophic for Allied forces. By the time Allied forces had pondered the situation on the ground properly the realities had changed too much for their solution to be applicable. Germans soldiers, on the other hand were very dynamic and always adapting to the current situation.

I didn’t set out to write this post only about German soldiers and software engineering. These two examples were just outliers to get your attention. My main idea which is that these patterns emerge across a wide variety of disciplines. After reading Steve Yegge’s blog post about liberal and conservative developers. I was inspired by his idea to look for a grand arch that unites a lot of different themes.

I think my idea is even broader than what he described because it goes across so many different disciplines. I think there are better labels than liberal and conservative. I think this is better summed up in evolution versus intelligent design.

evolution vs intelligent design
market economy vs planned economy
scrum vs waterfall method
worse is better vs the right thing
the bazaar vs the cathedral
Toyota production system vs Ford production system

On each side of the table you have the same kind of ideas represented. The difference is just the difference in discipline and the language used to describe it.

Evolution vs Intelligent Design

I believe evolution presents these ideas in their purest form. It is counterintuitive that a process such as evolution is able to produce systems which are far more complex and advance than anything intelligent design has been able to do. By intelligent design I mean a intelligent human being or teams of humans creating a system or machine. Evolution is not intelligent. Evolution is not able to look ahead and see potential problems with a particular design. Essentially evolution churns out a lot of half-baked ideas, which are incremental modifications of existing solutions. Then it sees what sticks.

The reason why this works so well is because of the feedback system. People who don’t understand evolution very frequently overlooked this crucial part. If evolution was just random changes to DNA, one would never have been able to produce any organism.

The Cathedral and the Bazaar

This is a strong parallel to what’s Eric S Raymond describes as the cathedral and the bazaar. Like evolution he showed that by using a lot of mediocre software developers, coupled with frequent releases, a lot of users and lots of feedback, higher-quality software could be built than by a small team of exceptional developers. This team of intelligent and extremely capable developers represents intelligent design in the analogy.

When you designed the system, no matter how well you have thought about it you are likely to get some things wrong. If you spend a one time thinking and planning the solution there is a high chance that you veered off in the wrong direction. Evolution and the bazaar mode of development might not create very good solutions at each iteration. However each iteration is very short and feedback is given quickly so development can be adjusted before any problems compound themselves.

The human being has all kinds of design flaws which exist because evolution cannot look far ahead it’s always making small adjustments to what it has. That is why we have ended up with stupid designs like the cables going into the sensors of our eyes being connected at the front so that we get a blind spot. It is also why our spine is poorly suited for walking on two legs. However there’s little evolution can do about that because we started as four-legged creatures and were adapted from that.

We could compare this with software development by considering developing a computer game and we have the code for a four legged animal roaming around in our game world. Then we are given the task to add two legged humanoids to the game. There are two ways we could go about this. We could create the code for the two legged creature from scratch with some careful planning or just start hacking the code for the four legged creature until we get what we want. Hacking requires less sophistication, while creating the code from scratch requires developers with more skill.

We could actually see how something very much like this played out in real life with the Tanenbaum–Torvalds debate about Microkernels vs Monolithic Kernels. Or just look at HURD vs Linux. The ragtag group of hackers of varying quality managed to create a better product than elites designing the HURD microkernel.

Scrum vs Waterfall

At its most basic level this is really just about the length of one iteration in the development process. In Evolution or Scrum each iteration is really small, so in a given time frame one gets a lot of feedback to steer the development. Intelligent design or Waterfall method on the other hand, has very long iterations so that in the same time frame very little feedback is given to steer the development. It is hard to avoid going in the wrong direction. The result of this in the case of the Waterfall method one ends up making far too many features. 50% of the features delivered are never used. This increases the costs and time of development unnecessarily. With scrum on the other hand customer keeps getting new versions of the software all the time until they can say I’m satisfied with the results. This makes it very easy to avoid creating functionality that the users don’t need.

Ironically Scrum looks on paper much more wasteful than the Waterfall method. The reason is that since Scrum requires a fully working product at each iteration one has to tear up and rebuilds a lot of parts later.

Market economy vs planned economy

No person or government bureau plans how many toilet paper rolls, bread or cars should be produced in market economies. Figuring out how much should be produced of each item in a modern economy is an extremely complicated problem. And yet the problem is being solved by a surprisingly simple process, if:

more people start demanding a product: increase the price
fewer people start demanding it: decrease the price.

The first rule spurs companies to try to make more of the product because they can make more money. The second induce them to cut production. Like evolution a set of very simply rules are used to produce a very complex system.

Today we are conditioned to think that a market economy is superior to a planned economy. It is hard to appreciate how much sense planned economy makes intuitively. With a planned economy one can think about the economy and what should be produced for years ahead. For a while this actually works. Countries with planned economies were able to industrialize and grow their industrial production at amazing rates for a number of years. The reason a planned economy was able to channel much higher levels of resources into building factories and avoid duplication of effort. In a market economy lots of companies are spending resources to make almost the same product. At the face of it that seems very wasteful.

However planned economies ultimately fails to a large degree because of the information problem. You can read Joseph E. Stieglitz and Paul Krugman to learn more about this. As economies get more advanced, and there are more products to decide production levels for, the number of variables in your equations explode. Solving the equation or even getting the information needed to input into the equations become an impossible task. The result is the wrong level of production for all kinds of things. The Soviet Union was constantly plagued by overproduction of certain items and severe shortage of other random products.

This problem is perhaps not that different from the one faced by the software architects trying to gather all kinds of information about requirements of the software system that they will design. Like the Soviet planners, they will ultimately fail in getting all the information correct. Market economies and Scrum get stuff wrong all the time as well, but they get feedback quickly based on real usage to adjust quickly.

Toyota Production System vs Ford

There are a lot of differences between the Toyota and Ford production systems, but I will concentrate on those parts relevant to this discussion. The Ford approach to mass production is similar to a planned economy in that somebody at the top tries to figure out all the details of the operation. There are few local decisions as in a free market economy. Like intelligent design one makes a large upfront design where one optimizes the utilization of every machine.

This is done by e.g. placing a large buffer of parts at each machine in case there is a failure at one of the preceding machines at the assembly line. Thus the assembly line can keep going even when other machines fail. Like Waterfall we try to avoid failure by trying to anticipate what might go wrong.

The Toyota production system is more similar to Scrum, evolution and the market economy. Failure is celebrated because it gives valuable feedback which will help us improve. Batch sizes between machines are made small so that if any machine fails, big chunks of the assembly line is brought to a halt. That brings focus to problems. This is coupled with encouragement to regular workers to help make things work again. This is continuos improvement. So essentially you get a manufacturing system which is not set in stone but which improves through multiple iterations with lots of feedback.

This is analog to the Cathedral vs the Bazaar in the mediocre developers are used actively not just the elite programmers. Workers are helping making the assembly line work better. It is not just about the specialists and the engineers like with the Ford production line.

The Marshmallow Challenge

Peter Skillman came up with a design exercise called the Marshmallow Challenge. The goal was simple:

in eighteen minutes, teams of four must build the tallest freestanding structure out of 20 sticks of spaghetti, one yard of tape, one yard of string, and one marshmallow. The marshmallow has to be on top.

It is harder than it sounds. Among the interesting results from this was: the recent MBA graduates did worst at this exercise. Among the best performers were kindergarden graduates. The reason for this was that the kids used the principle of iterative development, while the MBA students focused on developing and executing the one true solution. So when they finally put the marshmallow on, the top the structure would collapse because they had not properly understood how to create the structure. The kids however started putting on the marshmallow very early on and learned from mistakes.

In other words the MBA students were using “intelligent design” or acting like Allied soldiers, while the kids were using an evolutionary strategy and acting more like the German soldiers.

User interface design

As a final comparison, I would like to share my experience with designing user interfaces for computer software. I find that developers often think they have a much better idea of how users think than they actually do. When some thought is being put into designing user interfaces it often involves people who are experts in the field arguing over what is best. These discussions can draw out for very long time, and are IMHO a total waste of time. After a lot of discussions back and forth they agree and build a system and then release it to the users. It turns out the users do not understand it. By that time it is too late. It was very expensive to build the system and too expensive to start changing it now.

I find it better to start testing with users using mockups. Instead of arguing over 2 solutions it would often be faster to just test each solution on a few users. By having lots of iterations and testing one gets the benefits of evolution over intelligent design in that the massive amounts of feedback allows one to quickly home in on the right approach.

Another important aspect of evolution, which I have not mentioned thus far, is the importance of large populations and a lot of variance. Natural selection is of no use if all individuals in the population are genetically equal. Nor is it useful if there are only a couple of individuals. If one only has one design to test it’s much harder to evolve it than if you have multiple designs. With multiple designs you can pick out the ones that work best and have them “mate” with each other.

This is not very different from how I have read that the design process at Apple works. They start with 10 prototypes for products and where one of the goals is that there should be a large variations in those prototypes. Then these prototypes are evolved and the bad ones are discarded through several iterations. At each iteration the remaining prototypes are developed further.

Unfortunately this way working does not seem to appeal to managers. Ironically they seem to think the same way as Soviets planners: that by throwing out the free market of ideas you avoid duplication of effort and can thus progress faster. They think it is wasteful that a lot of engineers are working on different solutions to the same problem. However, this is exactly why open source software has been so successful because there is a free market of ideas, which are constantly competing against each other. Ironically the open source world is much more like a market economy with respect to product development than commercial software development, despite not being driven forward by a profit motive.

Liberal vs Conservative developers

To piggyback on Steve Yegge’s ideas about conservative and liberal developers, I would say that Conservative developers are essentially developers with a much stronger belief in intelligent design. By having lots of advance tools, elaborate static type checkers and guidelines for software development one believes all the wasteful duplication of effort done by the developers following the evolutionary approach (Liberal developers) can be avoided. The Liberal developers follow a style of development that is more organic. Code is written, changed and thrown away more frequently. There are a lot more errors being made. Dynamic typing allows you to write the software faster and get feedback from the real world faster. Static typing on the other hand slows you down but makes it more likely that your program is correct each time it compiles and runs.

Conclusion

Evolution, market economy, German soldiers, “worse is better” - all derive their advantage by using small iterations, each guided feedback from the “real world” of the previous iteration. Intelligent design, water fall method, “the right way” etc on the other hand, have long iterations, and thus little feedback.

I don’t really aim with this to say that intelligent design is always worse or inferior. This is more to caution against the thought that there is a always a shortcut that avoids duplicated effort. The thought that if we just think hard about it we can just build the product right away without creating prototypes that we have to throw away later.

Understanding Unsigned Numbers

Sun, 07 Apr 2013 16:21:05 +0200

When programming higher level languages I typically try to avoid using unsigned numbers. As a programmer you typically don’t want to have to think too much, and unsigned numbers requires some thinking to not screw up. But when doing assembly programming, especially on a 8 bit processor you better understand unsigned numbers.

For simplicity we are going to consider 4 bits (a nibble) when talking about unsigned and signed numbers. 4 bits can represent numbers in the range 0 to 15 or signed as -8 to 7. When we write a binary number such as 1001. Each of the bit positions have a value (beginning from left most): 1, 2, 4, 8. Basically the value is calculated by 2^bit_position, where the first bit is numbered 0. To calculate the value of 1001 we multiply the digit at each bit position with its corresponding value and add up. So 1001 becomes 1*8 + 0*4 + 0*2 + 1*1 = 9. Except this is when we consider those 4 bits unsigned. If we consider this to represent a signed number, then having the right most bit set (MSB) indicates that we have a negative number. Which one?

Modulo arithmetic

Do understand why we have chosen MSB to represent negative numbers, we have to understand modulo arithmetic. In elementary school we learn about the number line. Negative numbers are at the left side and positive on the right side. The number line expands into infinite in both directions. When have limited numbers of bits it doesn’t work like that. Instead the number line becomes a circle. Think about the 12 hour clock. The 12 position can represent both 0 and 12.

Likewise with a 4 bit number the 0 position of the number line (which is now a circle) can represent both 0 and 16. 15 (binary 1111) is right before the 0/16 position where we get a wrap around. If we add 2 to 15 we move clockwise and get 17. But because we are on a circle and not a line, we wrap around just like a clock and get to 1. You can also see the logic of this by looking at the binary representation. 17 is represented as 1 0001. It requires 5 bits. Only look at the least significant (left most) 4 bits and you get the value 1.

If you have 1 and subtract 2, you should move from 1 two places counter clockwise. That lands us on 15 again. So it is natural to let 15 represent -1. 14 would represent -2 etc. So to find the corresponding negative number you subtract from 16. 16 - 15 = 1, 16 - 9 = 7. So positive 9 is the same as -7.

So if you got a positive number and want to get the corresponding negative number, how do you do it? You could subtract from 0. Another way is to get the two’s complement of the number. You get two’s complement by first inverting all the bits (0 becomes 1, 1 becomes 0). Inverting the bits is the same as getting, what is called the one’s complement. So to change the sign of a number you can perform:

~number + 1

In AVR assembly, if you have say a 24 bit number (3 bytes) in the registers r19, r18 and r17 and want to change the sign, you can write this code:


;can't add one directly. Need to use a register

ldi r16, 1


;invert all the bits   
com r17

com r18

com r19


;add 1 to the 24bit number

add r17, r16

clr r16 ;only want to add the carry to the next bytes

adc r18, r16

adc r19, r17

It might have been easier to just subtract from zero. You can do that with the NEG mnemonic, but it does not work with carry, so you can’t use it to flip the sign of numbers with more than 8 bits. You can do it with the SUB mnemonic, but that is not much easier because result of the subtraction is stored in the left operand.


;lots of moving around because result gets put in first operand
mov 16, r17

clr r17

sub r17, r16


mov r16, r18

clr r18

sbc r18, r16


mov r16, r19
clr r19

sbc r19, r16

Branching with signed and unsigned numbers

It is easy to mess up your code if you do not take into consideration whether the numbers you are using are supposed to be signed or unsigned when you are branching. Say you have to registers r1 and r2, which are comparing and then branching.


if (r1 < r2) goto label

if (r1 >= r2) goto label

Use BRLT and BRGE if r1 and r2 are supposed to be signed. If they are not use BRLO and BRSH instead. Consider this:


 ldi r16, 128

 cpi r16, 127

 brge label

You might think this will branch to label since 128 > 127, but since BRGE interpret the numbers as signed, we get the comparison -128 > 127 instead.

Getting started with AVR assembly programming on OS X

Wed, 27 Mar 2013 12:04:41 +0100

This is not about the usual stuff I write about. But recently I have been playing around with the AVR microcontroller. It is like a tiny 8bit computer you can use to controll hardware. My main interest from a software development point of view is that it is great for learning assembly programming and actually make little programs which are worth making.

I am e.g. writing software to controll the speed of a DC motor using pulse-width modulation (basically how a light dimmer works, but for a motor). And I am also creating a funny box with buttons and lights for the kids to play with. Common with all off these applications is that the programs you write are really small, but need tight integration with hardware. It is a place were you can actually use assembly programming without it being a complete pain in the neck.

So if you want understand learn assembly programming, AVR is a great place to start. It has a clean easy to learn instructionset, and the hardware is pretty simple. E.g. you can easily figure out how many clock cycles a set of instructions require to finnish. With a fancy intel or AMD processor on a modern PC or Mac, this is very complicated because of things such as branch prediction, high latency to memory, cache hits and misses etc. AVR doesn’t really have cache, or rather all its memory is cache so everything is a lot easier to understand.

What software to get

My motivation for writing this is that it took me a long time to get started, because the information about all the software you need is scattered around. Here is a list of the software I think you should download to make AVR assembly and C programming easy:

CrossPack (package program uploader etc)
AVRA (assembler)
vAVRdiasm (diassembler)
avr-assembly.tmbundle (for editing code in TextMate)
AVRFuses (GUI app for setting configuration bits on chip)
MacSim (AVR simulator)

CrossPack is great to get started writing C code for AVR chips and possible use some inline assembly. CrossPack install the AVR GNU assembler avr-as described here. The assembly files typically end with .S. You can invoke it through gcc with e.g.

avr-gcc -Wall -Os -DF_CPU=8000000 -mmcu=attiny13 -x assembler-with-cpp -c ledflash.S

This will assemble the file ledflash.S for the AVR microcontroller unit (MCU) ATtiny13. The problem with this assembler is that it does not use the same directives as the the official ATMEL assembler. Most example code and tutorials you can find online is written according to the ATMEL assembler.

Fortunatly if you look around you can find the open source avra assembler. A problem when first googling is that an older assembler tavrasm will more likely pop up first in your search. Ignore this one. It does not seem to be maintained.

Avra is newer and is very easy to compile yourself. There are no dependencies. You just have to give a list of files to gcc or clang to compile it. To check that everything has been assembled correctly you can use the vAVRdiasm to diassemble the .hex file and see that you get back something that looks like what you put in ;-)

avr-assembly is a TextMate bundle. That means it is a plugin for the TextMate editor which allows you to get syntax highlighting for AVR assembly code and gives you hotkeys to assembly the program and upload it to your AVR microcontroller.

AVRFuses is a GUI interface to AVRdude (software in CrossPack which sends your programs from your Mac to your AVR microcontroller chip). It makes it easy to set or read fuses or upload programs. Fuses is simply bits you set in a special memory location of your AVR chip which affects how it operates. E.g. its clock frequency, what powersaving features are turned on etc. Fuses are set separatly from your program.

MacSim is a simulator for AVR microcontrollers. It is alpha and lacks a lot of features, so don’t expect to run full programs on this one. You might wonder why bother? Well unless you have an expensive developer board from ATMEL (makers of AVR chips), it will be difficult to debug your code. Usually I use a LED (light emitting diode) to flash when certain parts of my program have been reached. But you can’t use that to check the values of numbers and see if you have done all calculations correctly. Often you have sections of your code which are really just algorithms, which you want to test. MacSim is great for this, because you can see the values in registers and memory as you step through.

As a final note, be aware that you have to fiddle a little bit with all of this software. A lot of stuff did not work exactly as the manual said. It is usually just very minor adjustments to make it work. E.g. the TextMate bundle instructions gave the wrong path to bundles for TextMate2.

Part 1 - Intro to the Nu programming language

Sun, 26 Aug 2012 22:51:28 +0200

Main building blocks

The building blocks of a Nu program are atoms and cells. Some examples of atoms are:

Example	type
42	number
42.5	number
foo	symbol
+	symbol

Strings and numbers should be familiar with anybody who has done some programming before. Symbols are used to give names to variables and functions. However they do not have to. They can exist by themselves. The “+” symbol e.g. holds the plus operator (which is a function in Nu). We will look more at symbols later. First lets look at a simple expression in Nu. We write the calculation 1 + 3 * 5 + 7 * 11 + 13 in Nu:

(+ 1 (* 3 5) (* 7 11) 13)

As you can see the operators comes first and there is no implicit operator precedence. You need parentheses to say what is evaluated first. The numbers and the operators +, * are the atoms. These are in lists which are made up of cells. In memory the program will look like this:

Each of the blocks you see is a cell. Cells can be strung together to form lists and lists can contain other lists to effectively create trees. A cell consists of a head and the tail. Both the head and the tail can contain either an atom or point to another cell.

A Nu expression is evaluated by pseudocode similar to the one found below:

eval(node)
    if (isAtom(node))
        return node.value
    else if (isCell(node))
        switch (eval(node.head))
            case "+":
                sum = 0
                for (next = node.tail; next != nil; next = node.tail)
                    sum += eval(next.head)

            case "*"
                for (next = node.tail; next != nil; next = node.tail)
                     sum *= eval(next.head)

eval is basically called recursively on all the lists and their contents. Some practice on expression is evaluated by first evaluating the inner most expressions and substitute.

Let us look at how our example expression is evaluated:

Step 1

(+ 1 (* 3 5) (* 7 11) 13)

Step 2

(+ 1 15 (* 7 11) 13)

Step 3

(+ 15 77 13)

Step 4

Working with lists and cells

Nu expressions do not need to be arithmetic. We can also perform operations on the data structures to a logical operations and so on. Manipulating lists is very important in and you because we can use lists to store data. Basically like a regular data structure such as an array or set. But just as important as we have just seen, and Nu programs themselves are made up of lists.

(list 1 2 3 4)

This creates a list of the numbers 1, 2, 3 and 4. As said before lists are just a bunch of cells linked together.

(cons head tail)

A cell is created by using the cons function. So (list 1 2 3 4) is actually syntax sugar for (cons 1 (cons 2 (cons 3 cons 4 nil))). Which in memory it looks like:

Let us look at another example. We can represent a point with the X, Y coordinates (20, 30) as (const 20 30). Again in memory this will look like:

(head cell), (tail cell)

To get a hold of the first element of a cell we use the head function. To get the second we use the tail function. We can use the Nu command line nush to try this out:

[nutour] nush
Nu Shell.
% (head (cons "hello" "world"))
"hello"
% (tail (cons "hello" "world"))
"world"

This works for lists as well because they’re made of cells.

% (head (list 1 2 3 4))
1
% (tail (list 1 2 3 4))
(2 3 4)

With head, tail and cons you can combine old lists to make new ones:

% (cons (head (list 1 3 5)) (tail (list 2 4 6)))
(1 4 6)

Lists are immutable so what happens in memory is that:

Becomes:

(append list1 list2 list3 … headN)

With cons you can only add one element at a time. If you want concatenate two or more lists you can use append:

% (append (list 1 2) (list 3 4) (list 5 6))
(1 2 3 4 5 6)

Symbols and variables

A variable is basically a bucket we can use to put results of evaluations. E.g. we can put cells or atoms into the bucket. The label on the bucket is a symbol.

A more computer science way of looking at it is that for every scope we have an associated symbol table. A scope exists within a ( and ) parentheses. We can bind a symbol to a value in the symbol table to create variables.

(set symbol value)

Here is an example of the previous example using symbols:

(1 2 3 4 5 6)
% (set a (list 1 2))
(1 2)
% (set b (list 3 4))
(3 4)
% (set c (list 5 6))
(5 6)
% (append a b c)
(1 2 3 4 5 6)
%

The reason for all this focus on lists and cells, is because Nu programs themselves are lists. You can put the list representing a Nu program in a variable and manipulate the program with the cons, tail, head and append functions. When done you can “run” the program by evaluating it with eval function. Remember a Nu program is just an expression. Let see how we can use this. Say we want to be able to write mathematical expressions like this:

(5 + 3)
(42 - 22)

You can not do that in Nu, out of the box, because operators have to come first. But being able to manipulate the structure of programs allows us to achieve it anyway. Let us start up nush and try:

[nutour] nush
Nu Shell.
% (set str "(5 + 3)")
"(5 + 3)"

(parse string)

Now we got our program in string form. However Nu can not evaluate strings directly. A string is not a chain of cells. But the parse function will take a string and turn it into lists of lists:

% (set prog (parse str))
(progn (5 + 3))

(eval list)

As you can see Nu has inserted something extra progn. That is always inserted when you parse. It is simply a function which evaluates all its arguments and returns the last one. We can try to evaluate the program now:

% (eval prog)
Nu uncaught exception: NSRangeException: -[__NSCFConstantString substringWithRange:]: Range or index out of bounds
  from <TopLevel>:1: in 5
  from <TopLevel>:1: in progn
  from <TopLevel>:1: in eval

Do not bother with the cryptic error message. The point is that it doesn’t work. So lets write the code that would rearrange the operator and arguments. First let us figure out how to get hold of the operator:

% (tail prog)
((5 + 3))
% (head (tail prog))
(5 + 3)
% (tail (head (tail prog)))
(+ 3)
% (head (tail (head (tail prog))))
+

Okay now we have the code for getting the operator. We can modify the second expression to get the first argument:

% (head (tail prog))
(5 + 3)
% (head (head (tail prog)))
5

We can modify second last line to get the second argument:

% (tail (head (tail prog)))
(+ 3)
% (tail (tail (head (tail prog))))
(3)
% (head (tail (tail (head (tail prog)))))
3

Putting this all together we get:

% (list (head (tail (head (tail prog))))
- (head (head (tail prog)))
- (head (tail (tail (head (tail prog)))))
- )
(+ 5 3)

This is not very easy to read, but thrust me it can be done simpler and more readable, but that requires more knowledge of Nu. The point of this intro was to show as early as possible the purpose of the extremely regular syntax of Nu. The regularity is what makes it less nice to read than say a Java or Python program. However it is this simplicity which gives you the power to manipulate the structure of programs easily. Later when you are more familiar with Nu I will introduce Macros, which is basically a much terser and readable way to do what I just did.

When I was introduced to macros the first time however I thought they were quite magical, so I think it helps to understand that below the surface they are just using head, tail and cons to rearrange program code.

People matter more than software the development process

Sat, 18 Aug 2012 09:33:32 +0200

I just read an article by Alistair Cockburn on people as non-linear components in software development. That made me try to think of my own experience being part of a team and having technical lead as an application architect.

Whiteboards

Can not agree more with Alistair Cockburn about whiteboards. Being with other developers and discussing solutions while drawing is very effective in creating common understanding. I can also vouch for the point about number of people. Most efficient to me is two people. Four people works as well. Above that the communication starts to become regimented. For instance in big Scrum team meeting there is little discussion going on. It is more like a “report to the leader” kind of style. But the idea was that we were supposed to inform each other about what is going on.

My style was usually to off the bat call in to short meetings with whiteboards. Generally I did not plan meetings ahead of time. Part of the reason for that is that I just wanted it to feel like a discussion, rather than a formal meeting with a big M.

Back to whiteboards. Despite the fact that whiteboards are such a cheap and effective way of communicating and exchanging ideas few companies fail to see this. Most places I have worked have either lacked whiteboards or have only gotten them after a strong push by a few dedicated developers or managers.

Video Conferencing

I absolutely hate them Video Conferencing systems. In general I think they are a waste of time. So much of non verbal communication is lost. Often you do not see facial mimic or catch subtlety in voice. You can’t as easily gesture and draw. I always found that projects relying on Video conferencing did not run very smooth. Sometimes it sort of works, but only because you have had a lot of face to face with those at the other end of the video conference screen before.

Personality and chemistry

Everybody knows people are different. Pair programming will work great with some people others not. Assign people the wrong tasks and they zone out and waste time. To successfully run a team you have to really know people. It is hard to make that work if you get just randomly assigned some people. Personally when I knew people well and knew who would be on the team I would think a lot about who would be suited for which tasks. Who could potentially work together etc. Positive feedback really helps people go along with suggestions and stay positive and committed. A key thing here is to not make the mistake of thinking about what is good by your standards but what is good by the standards of the person you give feedback to. It is really the same as with kids. By my standards it is not an achievement to be able to brush your teeth. But to my 4 year old son it is, and it should be complemented. Another important thing is you can’t dole out complements uncritically. You can’t say people do great whatever they do. Then they will not place value in what you say and they might get and inflated sense of their capabilities.

So what this really comes down to is knowing people well. Only if you know people well do you know if they have done better than usual. Complements buys you credit to allow you to criticize people’s work. People are not very open to criticism by someone they don’t respect or like. If they feel you are on their side, they will be more open to critique from you. If someone is not open to criticism by you, then the critique has no value, because it will largely be ignored. Quite likely it will make matters worse because that person now has gotten a negative attitude towards you.

Informal networks

At any job people have all kinds of roles and networks between each other which is not spelled out in an organizational chart. People do stuff that they were not assigned to do by a manager. You might know of a guy that knows about all kinds of linux software versioning problems or another guy that is really good at the build system and can help you with that. To get the job done really requires knowing a lot of people and who to turn to for what. To be able to know that you have to socialize and get to know people. If your workplace doesn’t give a natural way to socialize then a lot of the necessary networks wont be formed.

Pick your fights

Nobody is perfect and most people have some annoying habits or bad personality traits. As Cockburn says changing peoples habits is really hard and I would say changing peoples personality is almost impossible. You got to decide which traits and habit you have a chance of changing and which you have a poor chance of changing. And of course decide what is most important. A lot of stuff that annoys you, you just have to slide, otherwise you will just end up delaying the project by picking unnecessary fights. This comes down to figuring out how important something is to a person vs how important it is to you. Again you got to know people well to make this decision. It can often be surprising what is important to some people. Don’t make too many assumptions based on your own values. Simple stuff like coding guidelines can create extremely heated discussions.

Use small teams

For some reason it seems like management prefers to have fairly large teams and few projects. My guess is because they find it confusing and mentally difficult to keep track of lots of projects and small teams. However I’d say people don’t really work well together in a team beyond 4-5 people. If there are more people it will end up splitting into small informal teams and projects.

Things that are needed but which doesn’t get officially sanctioned will usually happen anyway. If there are .e.g. nobody with an architecture role in a team, there will end up being a person who is the defacto architect. Just as with groups with no official leader will get a defacto leader after a while.

Objective-C versus Java

Sun, 29 Jul 2012 14:20:03 +0200

There are a lot of new people who have jumped into Objective-C programming due to the popularity of iOS. This has caused a large amount of blogs from people who have just started with Objective-C about how horrible it is and how much better Java is, and that Apple has made a really stupid mistake basing their development on this archaic programming language which would be best left on history’s garbage heap.

I think a lot of the criticism is quit unfair, and I would like to use the opportunity to argue why I think Objective-C is a good choice for Apple and why I prefer it over Java. I am not claiming a balanced comparison here since this is mainly about emphasizing the good parts of Objective-C.

Concurrency

By using blocks and Grand Central dispatch is easy to use extra cores and avoid blocking when doing IO. Since Grand Central dispatch exists at the operating system level is possible to do load-balancing across the whole operating system. This is possible because one can optimize the number of threads being used for the number cores available. In Java are you won’t know how many threads are being used by other programs so it’s hard to know the optimal number of threads for your program.

Blocks and queues also makes it easy to avoid using critical sections to handle access to share resources by multiple threads. Java on the other hand you will be more dependent on Mutexes and locks which are harder to work with and scale does not scale as well.

Constructors

A problem I have encountering number times when working with big software is that virtual methods could called and constructors. In languages such as Java, C# and C++ this will cause a number of problems. Those programs are really hard to do to debug and figure out. The problem arise because the virtual method gets called before the instance has been fully constructed. That means in Java’s case that variables have not been initialized.

Another problem with constructors is that they often lead to a lot of duplication of code. You can not reuse or call a constructor in the same way as you can call another method. All these problems are nonexistent in Objective-C because objective-C does not use constructors. In Objective-C instances are initialize with regular methods. That means calling virtual methods during initialization is not a problem. It also means that it’s easy to reuse initializer’s defined in the same class. This facilitates code reuse.

Method invocation

In Objective-C methods are invoked through message passing. In a nutshell that means sending a text string to an object and the object will look at the string and decide which method should be called.

This allows you to do a number things which are difficult to achieve in a statically typed language such as Java. Core data is a framework that utilizes this behavior. It is a framework that manages objects graphs and serialization. Objects are lazy loaded off disk as they are needed. The initial object loaded will point to proxy objects. As soon as one tries to do something with them Core Data will load from disk and replace the proxy objects with concrete objects.

Another example of where dynamic typing shines is constructing user interfaces. In user interfaces objects need to be loosely coupled. To see makes it easy to connect the actions of a user interface element to a method in an arbitrary class. This gets cumbersome in Java because the user interface elements requires the objects they interact with to have well defined static interfaces. It also allows you to do a number of other things which are very helpful when designing user interfaces, such as bindings and undo. With bindings you can tie a value of one objects to a user interface element. Undo is easy to achieve because method indications can easily be stored.

The delegate pattern also becomes more powerful because the class can check if the delegate implements a method or not if it doesn’t the class can choose to do the job itself. You can’t check in Java if the delegates implements a method or not.

Automatic reference counting versus garbage collection

No doubt garbage collection has its merits, but it’s not a silver bullet. Automatic reference counted a supported in Objective-C has a number of advantages. It requires less memory and is deterministic. These are important things in mobile devices which often have little memory and where you want responsive and fluid interfaces. A garbage collector can kick in and make your user-interface jerky. With automatic reference counting I find that I have to think very little about memory management. In that respect is not very different from garbage collection.

Performance

There have been a number of performance comparisons between Java and Objective-C. Most of these are extremely unrealistic. Often they have been micro benchmarks and this crucially testing the message passing performance of Objective-C versus a message invocation in Java. I would claim that in a real-world setting Objective-C it will beat Java performance wise. The reason for this is that in most of the code you write the slow Objective-C message passing is not a bottleneck. Despite the fact that all the user-interface code an iPhone application is written in Objective-C one cannot see any clear performance problems from that. Quite the contrary these interfaces will usually be smoother than on an Android device.

Usually performance problems will come from specific sections of your code. The important thing then is to be able to optimize and and tune those parts. Objective-C is very strong in this respect because it allows you to effortlessly integrate with low level C code. You also have fine-tuned control over memory allocation and memory layout. This is one of the reasons why the Java version of Git despite heavy optimization never reach more than half the performance of the C implementation. Is mainly came down to the inability to control memory layout in Java. Cache misses have severe effects on performance in today’s computer architectures. To avoid them one has to place data which is used together, close in memory.

I think a good analogy here is computer games. I think you will dispute the computer games are some of the most high-performance Software that exists. Yet most of the behavior into games is written in script languages. Often very slow script languages. High performance is achieved by writing the critical parts of the game, the game engine in C or C++. The beauty of Objective-C is that is essentially puts a script language and C into one language and make them interop very easily.

Verbosity

I often see comparisons with Java where the author is complaining about the verbosity of Objective-C. however this often comes down to the fact that Cocoa has long descriptive method names. One could easily write new methods with short names. The language does not prevent you from doing it. However a lot of the verbosity of java is baked into the Java syntax, there is very little you can do about it.

I’m thinking of such things as inner classes and adapters. Private and public has to be specified for every class and method. Problems which in Objective-C could easily have been solved by using a function pointer or storing a selector requires a lot of machinery in Java because everything always has to be a class or object.

Numerics

From my understanding Objective-C has more proper handling of floating point arithmetic and you don’t have the problem in Java that all types are signed. This cause problems when doing bit manipulation. Most of the time this is not a problem but specialized software or performance tuning you might need to do it.

Value types

I think it is natural that things like points, rectangles and vectors are value types. Meaning whenever you assign, you make a copy. In Java this is not the case. Even a simple data structure like a point is allocated on the head and passed around as a reference. There are other languages which use references for this kind of data as well, but then they typically make the data structures immutable so for practical purposes they act like value types. For this has been a common source of bugs since one easily ends up modifying an input parameter. In Objective-C you can easily use C structs to get value types. For more complex data types which makes sense to treat as value types it is common practice to have immutable versions.

Robustness and security

Objective-C gives you full access to operating system services. You can use all types of interprocess communication offered and gives full access to utilize functions that deal with processes such as fork. This is important for robustness and security because it allows you to split a program into several different processes. If one process dies it doesn’t take down the whole application. It also allows you use different privileges to different process to limit security holes. In the Java world it seems like the emphasis is on threads which offer no protection from bringing each other down.

The process abstraction is very important in providing robustness. That is why a language such as Erlang, which despite lacking static typing is used to create some of the most robust software that exists. Erlang is designed around creating lots of processes that communicate and can restart each other if one crashes.

Final word

Objective-C compares well with Java. There is no reason why you could not be as productive in Objective-C as in Java. Apple is doing the right thing in sticking with Objective-C. Going with Java would be a stupid choice. It would negate a lot of their advantages that they have in Cocoa which is in my opinion one of the best designed frameworks for Application development. Apple has already shown that it is possible to modernize Objective-C. If one should need more productive programming languages I think it would be better with MacRuby, F-Script and Nu because those languages are built on the Objective-C runtime and mesh well with Cocoa, because they are dynamic languages like Objective-C. Java would not fit in well, and we already saw have Java was a failure for Mac development. I don’t see why things would be different on iOS.

A somewhat valid argument is why should Apple go with an almost unknown language instead of Java which everybody already knows. That would make sense if they had started from scratch. But Apple didn’t. They had a framework (Cocoa) that had been built and polished over very many years. It would have been foolish to throw away that for the current fad.

Objective-C versus Java

Sun, 29 Jul 2012 00:18:59 +0200

Concurrency

Blocks and queues also makes it easy to avoid using critical sections to handle access to share resources by multiple threads. And job on the other hand you will be more dependent on Mutexes and locks which are harder to work with and scale does not scale as well.

Constructors

Method invocation

Automatic reference counting versus garbage collection

Performance

Verbosity

Numerics

Value types

Robustness and security

Final word

Nu for Ruby Developers

Mon, 26 Mar 2012 00:58:31 +0200

Nu is a programming language developed by Tim Burks. It is sort of an alternative to MacRuby. A short way of describing it is that it is Ruby with LISP syntax. programming.nu is a bit short on explaining the language so I thought I’d add some examples that might help Ruby developers get into it. I borrowed examples from the “Programming Ruby” book by David Thomas and Andrew Hunt.

I will show the Ruby and Nu version alternating. Iterate over an array in Ruby:


["cat" "dog" "horse"].each do |animal|
	print animal " -- "
end

And in Nu:


((array "cat" "dog" "horse") each: (do (animal)
	(print animal " -- ")))

Working with arrays in Ruby using interactive shell irb:


[nu] irb
>> a = [1, "cat", 3.14] # array with three elements
=> [1, "cat", 3.14]
>> # access the first element
?> a[0]
=> 1
>> # access second element
?> a[1]
=> "cat"
>> a << "dog" # append element
=> [1, "cat", 3.14, "dog"]

Arrays in Nu using nush shell:


[nu] nush
Nu Shell.
% (set a (array 1 "cat" 3.14)) # array with three elements

% # access the first element
% (a 0)
1
% # access second element
% (a 1)
"cat"
% # append element
% (a << "dog")
% ()
% (a description)
"(\n    1,\n    cat,\n    \"3.14\",\n    dog,\n)"

Print * 5 times in Ruby:


5.times do
	print "*"
end

In Nu:


(5 times: (do (x) # x is not used. Will contain 0, 1, 2, 3, 4
  (print "*")))

Working with dictionaries in Ruby:


instSection = {  
  "cello"    =>  "string",
  "clarinet" => "woodwind",
  "drum"     => "percussion"
}
  
instSection["clarinet"] # -> "woodwind"

Working with dictionaries in Nu:


(set instSection (dict  
  "cello"    "string"
  "clarinet" "woodwind"
  "drum"     "percussion"))
  
(instSection "clarinet") # -> "woodwind"

Not from the book, but put it here to illustrate a tricky problem in Nu, I couldn’t find documented on the web page but had to find out from the source code. Consider the case where you are searching through a list or array until you find an element. When you found it you don’t want to continue. The normal way to do this in Ruby would be:


def findValueBelow(lst, threshold)
	lst.each do |x|
		if x return x
		end
	end
	nil	
end

puts findValueBelow([8, 6, 7, 4, 2, 3],4) # gives 2

But in Nu we have to use the return-from function for this:


(function findValueBelow (lst threshold)
  (lst each: (do (x)
    (if (return-from findValueBelow x))))
  nil)
  
(puts (findValueBelow '(8 9 4 2 3) 4)) # gives 2

I will be writing more on my experience in using Nu in the future.

Kindle Fire experience by iPad owner

Mon, 09 Jan 2012 22:57:25 +0100

I have had a Kindle Fire now for a few days. I am an Apple guy and an iPad owner but that doesn’t stop me from liking this little device. Lets first start of with some of the things I like about it.

Size. It is so easy to hold in the hand and carry around. Typing on it is in many ways easier than iPad if you don’t have it on your lap. You can type with two thumbs which isn’t possible with iPad.
Integration with Amazon. Lets face it, while iBooks has a great interface, it simply does not have the same selection as Amazon and you can’t read the books on as many devices. I can read my kindle books on iPad, Kindle, iPhone, my Mac etc. On the Kindle Fire I can buy books conveniently through a special application. No need to browse the Amazon web interface like on iPad.
The Android back button. I love this concept. On iOS you can only browse back and forth within an app. On Android back means back to whatever was you previous screen, even if that was another application. A nice use for this is when you click links to web pages in your email application. You look at the linked web page but then you want to go back to the email and click another link. Easy, just click the back button. On iPad this would require either double click home and select email application among running apps or single click home and select email among all the apps.
Price. It is a really good price for what you get and this makes tablets accessible to a lot of other people.

Now for the things I didn’t like so much. The carousel thing was mainly annoying. I found it quite messy to have applications, books, movies and everything all in one. It is also about as useless as the Windows 7 exposé copy. It doesn’t give any quick overview over what you have. There was no obvious way to see what applications were running and quitting them. There was a sort of process manager but that was not very accessible and felt more like a geek tool.

But the most annoying thing was probably that there was no physical buttons for frequent operations like going to the home screen or going back. I found it really awkward and slow to have to tap my screen once to get the toolbar at the bottom with the home button each time I wanted to click the home button. A button used that often should be super quick and easy to click.

Apart from that as a user of Apple products I am used to a lot of polish on the products. These are things which are hard to explain but comes down to getting very minute details right. On Kindle I feel there are a lot of these small details which are not done properly as on an iPad. When you move an icon on the home screen of iOS you see the other icons move to the side. On Kindle Fire moving icons on the favorites shelf looks more like they get teleported from one place to the other. This is confusing because you don’t get the same feedback and feel for where things are moving from and too. If you scroll to far in a list there is just some sort of flash to indicate that you went to far. In iOS you actually continue to move past the list but the list snaps back as soon as you let go. I think it is a much clearer feedback. This is common issue I have with Android. You just don’t get the same quality feedback on everything you do. I often click, scroll etc and don’t know if the system registered my action or not.

There is also a lot of indirect manipulation instead of direct manipulation. E.g. I expected to be able to move an item from the carousel down into the favorites. You can’t. Instead you have to press and hold. This will get you a menu where you can chose to add to favorites, which exists visually right below you. This feels unnatural. A lot of things are like that on the Kindle Fire.

It is all these little details which add up to create the smooth experience on iOS, but which makes Android not feel quite right. But more importantly I prefer iPad overall because it is much more versatile. The bigger screen opens up the possibilities for more kinds of applications. While I loved the back button and think iOS should steal that idea there are simply too many other areas where iOS interaction works better. Application switching, management and organization is done more elegant IMHO.

Details like physical placement of buttons is done much better. E.g there are no volume control buttons on it. Volume is something you usually have to change very quickly. So it should be very quick to access. It is infuriating using e.g. phones where you suddenly realize you got too low volume while speaking to someone and you have to put them on hold while you try to navigate the system settings to find volume control.

Finding a string in a string list O(N) or O(N^2) ?

Fri, 16 Sep 2011 22:19:00 +0200

This ought to be simple right?

My very first thought is that we are potentially looking at every string to find the right one. So it is O(N). But wait. For each string we loop through each character of that string. We got two loops. So maybe it is O(N^2) ?

But lets think about this more carefully. Each loop is looping over different kinds of data. The outer loop are strings while the inner loop are characters. So why no use the same units everywhere? Meaning let’s look only at the characters. How many character comparisons do we have to do?

Say the string list contains a total of N characters. The string we are searching for is made up of M characters.
Consider a worst case. Every string in list are the same as the search string except the last character. The string list then contains N/M strings. That is, each string is M characters long. How many comparisons do we do? For each of the N/M strings we do M comparisons. That totals (N/M)*M = N comparisons. So it is linear. Really? How can that be, we have to go trough every character in the search word multiple times. We got two loops. How can it not be O(N^2)?

The key observation is that in the string list we are never looking at a character twice. If we only look at the number of comparisons we do in total. We don’t do more than N comparisons between a character in the string list and a character in the search string.

Consider some of the other extremities. N = M, that is the search string contains as many characters as there are characters in the string list in total. And say the string list only contains one string. Obviously its is O(N) comparisons. At the opposite extremety each string in the string list is only 1 character. We compare with N strings, but for each of these comparisons M - 1 characters in the search string will never be visited.

This example illustrates that when considering big O it is not enough to merely look at the number of loops nested inside each other. One has to think about what are exactly the elements which are being processed.

The old struggle: native vs cross platform

Wed, 07 Sep 2011 21:20:34 +0200

Today there is a lot of talk about iPhone and Android apps. They were not supposed to have happened because the future belonged to the web. It happened anyway.

At a Java conference today I talked to some software managers at different consulting companies. They all seemed convinced that the app was a fad and HTML5 would take over. I think they are going to be wrong again. It is understandable why they think so, because the web is really ITs wet dream:

You can do all updates, configuration and maintenance in one place
Just one platform to support
Develop once and run everywhere
Easy deployment. All you the clients need is a web browser.

But wanting something doesn’t make it so. Consumers have shown that they don’t care about how great web apps are for IT people. Consumers want slick, smooth and rich applications. And unfortunately the people in the IT departments has no idea of what this really is.

Fundamentally this isn’t really about the web at all. Instead it is the age old struggle between native applications and cross platform applications. Naturally IT likes cross platform. It has always been like this and they are always going to want it that way. They want to write their stuff once and use it everywhere.

But this is not what the consumer wants. The consumer choose a platform because they think that platform is better. This goes for personal computers, phones and game consoles. Each platform will have unique strengths and abilities to make themselves attractive to the consumer. The consoles want to offer unique interactions that the other platform can’t afford or be able to produce richer graphics than the other platform. Phones want to be easier to use, offer a unique eco system, have unique ways for interaction and so on.

It is never going to be a game purely about the amount of memory, speed of microprocessor and number of extension ports.

Cross platform toolkits like Swing, Qt and HTML5 will always have to play catchup to the native toolkits. In the software managers dream HTML5 is just about to catch up with the native kits. Just you wait. In short time all she need HTML5 to present users with the same experience as the native kids. Of course this is all wishful thinking. They assume a static world were each platform stays still why cross platform tools rapidly advance. This is not the case. Neither Qt nor Swing can provide all the bells and whistles of the native toolkits. They never have. Each time they caught up new native functionality was added.

Just as it has for decades cross platform and native apps will continue to exist side by side. One side is not going to kill off the other. It is simple really. We live in a market economy. As soon as someone makes some great application in a cross platform toolkit be it Qt or HTML5, there are going to be someone who makes the same app in Cocoa, win32 or whatever. That app will run faster, have tighter integration with the host OS, offer nice UI elements that don’t exist in the cross platform toolkits. Users are going to buy this app because to them it is better they don’t care that it is not how IT wants it to be.

You might ask what more can native kits offer? Computers are already fast enough to run anything you want to run through HTML5. Before the iPhone came out they probably said the same thing. But suddenly your UIs had to be fluid, animated and highly interactive. By the time phones are fast enough and HTML has gotten enough features to do what the first iPhone did, there will be something new. And then it is the catchup game all over again.

Managing coupling in a data oriented way

Wed, 17 Aug 2011 23:51:37 +0200

When your code is tightly coupled it is hard to change one part of the code without affecting other parts. When your modules are loosely coupled they can be modified or removed independently with ease.

I just came across a blog run by the guys behind BitSquid, a new high end game engine. They have some great entries about numerous things related to game development. But today I thought I’d highlight the tree blog entries by Niklas about managing coupling:

This is not your regular advice on how to manage coupling in an OOP setting, where you would talk about usage of abstract interfaces, composition, which classes should know about each other etc. The kind of stuff Java developers would be used to. Rather this is a data oriented design, which is very C-style and low level.

Using IDs to reduce coupling

Niklas details why you should refer to objects most of the time using IDs rather than pointers. An ID is typically just a 32 bit integer. Unlike pointers IDs can be moved to different CPUs or cores using different address space, serialized to disk etc. The naive approach would be to create a hash table or STL map to lookup pointers based on integer ID. But the blog entry shows how we can do very quick lookup by storing offsets inside the ID. Noel at Games from Within also talks about using IDs instead of pointers in Managing Data Relationships. I think you should read these before starting to use GUIDs e.g.

Event management

In the second blog post Niklas has the novel idea that instead of you pushing events to an event handling system where stuff happens, each subsystem just stores events as they happen and the higher level system then polls the lower level system for events. This way you don’t couple any low level system to a specific events notification system.

I have worked with these kind of problems before. You might have global loggers or message objects, so that in your algorithms you might write something like:

if (wrong) Msg::Error("Something wrong happened!")

Or you might have some logger object or singleton class:

if (wrong) Logger::GetLogger("error")->Log("Something wrong happened");

The problem with this approach is that your algorithms gets strongly coupled to your framework or particular way of sending messages. You also lose control over how and when to display messages. Niklas doesn’t go into details but one way I have found useful is to pass a message handling object to functions or objects and have them fill this message handling object with messages or events:

void process(MsgObj *msg) { if (wrong) msg->addError("Something wrong happened!");

Apple’s Cocoa API often uses a similar approach for dealing with errors. In the Visualization Toolkit, VTK, they extensively use an observer pattern. A lot of software has classes that represents algorithms. When the algorithms want to report on progress to the main UI or report errors they would typically use a central message object. Often using a global function or singleton. In VTK they make algorithm objects observable. That way progress and errors are reported by sending messages to observers rather than to a centralized object. These approaches should not be confused with what Niklas talks about because these are OO solutions to similar problems that he outlines.

Duck Typing in C++

In the duck typing in C++ blogpost Niklas talk about stuff that might seem a bit similar to QVariant class found in the Qt application and UI framework. With QVariant you can also create arbitrary data structures by combining simple QVariant types with QVariantMap and QVariantList. But Niklas show an interesting twist in how you can in a way define something that can best be described to a Qt developer as a QVariantMap template for QVariantMap objects with fixed number of members. This allows very efficient storage of each variant map. Niklas doesn’t talk about Qt at all, but that is a way you might think about it. You have similar mechanism in Clojure and other LISPs and Scheme versions.

Wrapping Lua/C++

Wed, 17 Aug 2011 22:25:00 +0200

Hey I was looking at your wrapping c++ classes in lua article, I must say its been the biggest help in understanding some of the details to me.

The first part I got working no problem, the second part gets me a little stuck. Basically the extra argument isn’t getting passed (I think) Calling Sprite.new(..) only seems to pass the 4 arguments and not the table, what have I missed.

Thanks,

Phil.

That is easy, you have simply used “.” instead of “:”.

Sprite:new(...)

is syntatic sugar in lua for

Sprite.new(Sprite, ...)

That is how you get the table in as first argument.

Hey,

I was using "Wrapping C++ classes in Lua" article, as a guide to expose my C++ classes to Lua but I've encountered an error. I can't seem to get Lua -> C++ calls working. Checking the self argument isn't working; the check function failed. It expects the 'global' metatable (as in the registration of the class) but, gets a table. I know you commented that the code my not be correct however, I've been trying to figure out this issue for a day now. Some help would be greatly appreciated.

Regards,
Mike

Tue, 12 Jul 2011 11:38:23 +0200

Which approach are you using? The one without support for inheritance or the one with support for inheritance which uses the “__self” field on the table?

In the latter case, there might be a bug in the checkSprite() function. You could try to add this function:

/*! Pops user data of the stack and checks that it is of correct type */

void *popUserData (lua_State *L, int argnum, const char *tname) {

void *p = lua_touserdata(L, -1);

if (p != 0) { /* value is a userdata? */

if (lua_getmetatable(L, -1)) { /* does it have a metatable? */

lua_getfield(L, LUA_REGISTRYINDEX, tname); /* get correct metatable */

if (lua_rawequal(L, -1, -2)) { /* does it have the correct mt? */

lua_pop(L, 3); /* remove both metatables and user data*/

return p;

}

luaL_typerror(L, argnum, tname); /* else error */

return 0; /* to avoid warnings */

}

And replace the call

ud = luaL_checkudata(L, index, “Lusion.Sprite”);

with

ud = popUserData(L, index, “Lusion.Sprite”);

Why so complicated you might ask. The problem is that the regular checkudata() function doesn’t deal with the fact that we are using the “__self” variable.

But as mentioned in an answer on the loadcode blog, I think this approach with “__self” got too complicated relative to the benefit. I recommend doing the implementation without inheritance. You will find that given Luas duck typing it is less of a restriction than you think. When I got more time I will write more about this.

Some thoughts on Clojure vs Go

Thu, 10 Feb 2011 15:12:21 +0100

Neither Go nor Clojure are object oriented in the sense that they provide implementation inheritance. With Go you can create what it essentially classes without inheritance. Created with keyword “struct” in Go. These “classes” can used wherever an interface is required if a subset of the classes methods match all the methods defined in the interface.

In similar fashion Clojure has what is called a data type created with keyword “defrecord”, which is similar to a Go “class” in that it does not support inheritance. But while a Go “class” automatically implement any interface which is a subset of its methods, in Clojure you have to specifically list which interfaces a Clojure “class” implements:


(defprotocol Payroll
  (paycheck [emp hrs]))

(defrecord HourlyEmployee [name rate]
  Payroll
  (paycheck [emp hrs] (* rate hrs)))

(defrecord SalariedEmployee [name salary]
  Payroll
  (paycheck [emp hrs] (/ salary 12.0)))

In Go listing implemented interfaces is neither possible nor needed:


type Payroll interface {
  paycheck(hrs int)
}

type HourlyEmployee struct {
  name string
  rate float
}

func (emp HourlyEmployee) paycheck(hrs int) float {
  return emp.rate * hrs
}

type SalariedEmployee struct {
  name string
  salary float
}

func (emp SalariedEmployee) paycheck(hrs int) float {
  return emp.salary / 12.0
}

Since Clojure doesn’t really have methods per say, you can’t define other protocols with the same function names. That would just override the meaning of that function. This should not necessarily be interpreted as Clojure being less flexible then Go here. Clojure has a number of different ways to dispatch a function call based on the type or value of the arguments.


(defmulti paycheck :type)
(defmethod paycheck ::hourly-employee [emp]
  (paycheck [emp hrs] (* rate hrs)))

(defmethod paycheck ::salaried-employee [emp]
  (paycheck [emp hrs] (/ salary 12.0)))

I am not going into the details but we assume here that emp is hash table with a :type key. Multimethods allow dispatch on multiple arguments. However given two existing unrelated classes A and B, you can define an interface C in Go which covers a subset of methods in both A and B and then you can define a function f(obj C) which can do similar processing of some instances of A or B. This approach is a bit more involved in Clojure. You would probably have to define a new set of multimethods which in effect would represent interface C. These functions would have to forward to the correct function for A or B.

Beyond this Go starts falling short of the flexibility of Clojure for building abstractions since Clojure is a LISP and thus supports powerful means of abstraction like macros. There is nothing remotely similar in Go.

Go on the other hand is more capable when it comes to low level tuning of your application. Memory layout of datatypes can be controlled in detail which is not possible in either Java or Clojure. This is quite important given that cache is increasingly important to the performance of applications. In Go it is easy to place elements accessed together close in memory so that they are likely to be pulled into the cache at the same time.

IMHO that means Go is more poised to take over areas traditionally dominated by C/C++, while Clojure will be fighting in areas currently dominated by Java, C# or script languages, like enterprise systems. E.g. Go wasn’t designed to be used for game engines but I can imagine if a game engine developer had to chose they’d chose Go over Glojure, simply because they want much more low level control over their resources. In other respects they are overlapping in appeal. At Google there seems to be C++, Java and Python mainly used. For most tasks Python will simply be much quicker to use. When more performance is needed Java or C++ is used, which requires longer development time. Go appeals to those who want Python like ease of programming with C++ like performance. But Clojure gives Java like performance for those who want Python like ease of programming. Well, not entirely true but it can be tuned to get close and it is in perhaps most respect more powerful and faster to develop in than Python.

Who will succeed? Clojure advantages:

Part of the huge Java ecosystem.
Support more powerful abstraction mechanism.
Functional programming is gaining momentum and developers might see it as crucial that their next programming language supports this paradigm.
Support for REPL based development. Basically this means you can interactively develop a program. Clojure like any LISP gives you a sort of command line shell were you can write clojure code and test calling individual functions. You can load a file with clojure code into the shell and call individual functions to test them or redefine some of the function while the program is running.

Go advantages:

Syntax which is much more familiar to most developers. And we know this has been very important for the success of languages in the past.
More options for performance fine tuning (memory layout).
Imperative programming paradigm is better known and understood by the majority of developers.
Huge organization behind it (Google).
A very easy to understand concurrency model. Clojure also has great support for concurrency. Probably better in many ways, because it gives more options and is a functional language. However goroutines used for concurrency in Go is just dead simple to get started with.

I love LISP’s but my prediction is that both Go will become bigger than Clojure eventually. Simply because I don’t know of any time where a language with unusual syntax became a big success. Smalltalk, LISP, Objective-C etc never became big successes, mainly due to their unfamiliar syntax IMHO. I am guilty of this myself. I looked at Objective-C very early on when I had started learning C++ and rejected it almost immediately because I thought the syntax looked so weird. As a young developer I confused complexity of language with syntax. I had the crazy idea that C++ was somehow a simpler language to understand than Objective-C, because it looked more like C. This seems to be a common fallacy.

When pointers are better than iterators

Thu, 11 Nov 2010 00:08:20 +0100

The STL is a very well designed library, and iterators is a powerful concept. Unfortunately you can not fully utilize the power of iterators unless you write generic code:

template<typename Iterator>
myfunc(Iterator begin, Iterator end) {
  // Do stuff
}

Now the function does not depend on the type of the iterator being used in the call. However C++, being a multi-paradigm language means you might be writing OO code and not generic code. E.g. you might have defined an interface on one of your classes:

class MyInterface {
  virtual void GetStuff(int count, vector<int>::iterator out) const = 0;
};

I can’t define interface methods to be template based. The problem with the interface above is that I force the receiver to use a particular kind of iterator. E.g. I have to write code like this:

vector<int> buffer(count);
myvar->GetStuff(count, buffer.begin());

But what if I need buffer to be of any of these alternative types?

 int * 
 QVector<int> 
 vector<int, myallocator>

Then I would be screwed, because they all have a different concrete iterator type. However if we change the interface to:

class MyInterface {
  virtual void GetStuff(int count, int *out) const = 0;
};

Then we can use any kind of vector type:

vector<int> buffer(count);
myvar->GetStuff(count, &buffer[0]);

This code does not care which type buffer is as long as it represents an array of data in contiguous memory.

These observations give a few rules of thumb for when to use iterators and when to use pointers. * Never use iterators to receive data. Use pointers. * Let your internal data be accessed only through STL style iterators, but only ever use these iterators with generic functions. This will allow you to change internal container type without breaking code. If you use non-template functions you will break code.
* Never return pointers to your internal data. Instead push or pull data to/from external buffers through pointers. Non-generic functions should interface with pointer based methods.

So say I have a polygon class I should be able to do stuff like:

// generic function so use iterators
std::sort(poly.beginVertices(), poly.endVertices());
std::vector<Vertex> buffer(10);

// setting and getting values, so use pointers
poly.getVertices(&buffer[0], poly.size());
poly.setVertices(&buffer[0], poly.size());

// Non-template based class, so use pointers
vertexProcessor->doStuff(&buffer[0], poly.size());

// If class was template based, then we could to it like this
vertexProcessor.doStuff(poly.beginVertices(), poly.endVertices());

I know the first objection you probably have is that the pointer solution leads to a lot of unnecessary copying of data. That is potentially a performance bottleneck, right? Not necessarily. Typically you don’t need to try to optimize this. If it proves to be a problem the solution is fairly simple. Copy out only small chunks at a time. E.g. fetch 10 elements, do some processing, fetch the next 10 elements. Rinse repeat.

If you have a small fixed size buffer you keep pull data into, it will most likely stay in cache and it will be very fast to copy the data into it.