This recent post caught my attention. Amix, a developer over at Plurk, decided to write a fast polling server. Instead of reaching for off-the-shelf software he identified his situation as unique enough to warrant rolling his own solution. It was a fun read and I suspect a great exercise for him. In sharing and in the ensuing conversation I think I learned something.
Programmers don’t want you to program.
There is a certain maxim most programmers come to learn in time: do not re-invent the wheel. Where there is an existing piece of software available that is well tested and supported by the community, you should use that instead of rolling your own. The comments in Amix’s post deplore him for writing his own polling server and harass him with suggestions to use Valgrind and other pre-existing software. On the surface it is sound advice, but one reply from Amix struck a chord.
Varnish is a good cache, but it’s not really suited where I am headed. An example, how would you change the Varnish solution to support long polling? I can explore this option pretty easily with a basic understanding of how to build a custom server that can perform really good and that can be customized for my needs. I personally want to learn how to build things myself (also things that are complex and that require some thinking and engineering). I know I’ll need this skill longer down the road and practice makes a master - so the more practice I have, the better I’ll be at doing this.
He is absolutely right. One does not learn these sorts of skills from simply reading about them. You have to write the solution and actually use it and watch it fail miserably. Smart people will know what to do from there: test, learn, reiterate, try again. Wash, rinse, repeat. It’s how all great software gets written. Yet there are far too many who would hold on to dogma and tell him to forget about reinventing the wheel. They’ll tell him he cannot write security-hardened software and “why should he?” they’ll say: “it’s already been written.”
I think Zed Shaw had the most constructive thing to say in that whole discussion. You have to read between the lines to understand it. He says:
Zed Shaw 25. Mar 2009
Pretty cool, the more I find out about the tech at Plurk the more I start to admire you guys, especially compared to the crap that twitter was running.
Here’s a quick code review of your C code for security/safety reasons:
add_mserver – Should have asserts protecting against NULL on the host param and that NUM_OF_SERVERS isn’t out of range.
init_memcache_servers – You don’t validate the return of memcached_create.
parse_path – You don’t validate the return of malloc. Using strlen is a buffer overflow. Pass in the length of the string or use a lib like bstring. For loop uses instead of a predetermined length, which you already have from strlen. This function is externally accessible so these are real problems.
memcache_handler – Arbitrary fixed size for key. Using strcpy rather than a string with a predefined size. Ironically, your strlen checking for no buffer overflows has a buffer overflow. Use strnlen, actually just avoid all str functions that also don’t have a length param. You don’t check the return value of parse_path. You pass in key to
memcached_get but the str functions that make key are notorious for not honoring the . You should know the size of key after building and add your own to double check it.main – You don’t check return values of any function in here.
Finally, run this under valgrind and assume valgrind is always right. It sometimes isn’t, but it’s usually more right than a programmer so it’s your job to prove why valgrind is wrong. Usually if you tricked valgrind then your code is too tricky. Quit doing that.
Cool, hope that helps clean it up. Feel free to shoot me edits for feedback.
He offers constructive feedback. Without a hint of the “wheel” maxim or insult of any kind, he offers Amix some clear ways he can improve his code. He doesn’t tell him once that he shouldn’t use this code or shouldn’t bother writing it.
The maxim isn’t without merit, but I think what I’m seeing here is an overly-liberal application. If you want to master a skill you have to practice it. You can’t do that in software just by fixing bugs and sending in feature patches. Sometimes you have to just write it yourself. I’m glad Amix and Zed are cool with that. Most programmers I know and talk to aren’t. They stick to dogma like a warm blanket. Yet if nobody reinvented the wheel once in a while we might run out of wheel-makers and be really screwed. Break with tradition once in a while, you might discover something.
[tags]programming, dogma, plurk, zedshaw, amix, c, libevent, memcached,