Geek on Two Wheels

Context: What if Lisp was invented by the Japanese.

This is an IRC conversation from this morning reproduced in full with minor edits to formatting, spelling and removal of system messages.

09:49 < n0ob> so then it would seem that the summary is that the article is wrong, but postfix notation is inherently superior except that it requires learning
09:49 < lumy> did the post fix thing come from the article?
09:50 < lumy> (the being superiour part)
09:50 < n0ob> no
09:50 < lumy> ok. then we agree about the article.
09:50 < yangman> I wouldn't say the article is wrong, but starts on a good path then misses the point
09:51 < yangman> ... and mostly starts on that path from the wrong point, but, really, only linguists would pick up on that
09:51 < yangman> the actual neat part is how language has changed to better suit the hardware model
09:52 < n0ob> how so?
09:52 < yangman> FORTH is reverse polish because arguments and return values go on the stack, and you need to be very explicitly aware of what's on the stack
09:52 < yangman> (hardware model is perhaps wrong. I'll corret that later)
09:52 < n0ob> well, hardware model may not be incorrect, but there are more than one model to choose from
09:53 < yangman> with a lot of languages that followed after, the stack is basically abstracted away
09:53 < yangman> with FORTH, "what's on the stack?" is very key
09:53 < yangman> with stuff like lisp, "what's my result?" is what's important
09:53 < yangman> NOW
09:53 < yangman> what I initially wanted, but didn't have time nor energy for, is to point this out and make the connection back to Japanese
09:54 < yangman> there's a phenomenon in Japanese, and actually mandarin as well, where the verb is promoted to earlier positions in an utterance
09:54 < yangman> English has this also, but to a lesser extent
09:54 < yangman> so, instead of "(I) fruit(obj) ate"
09:55 < yangman> it's instead "(I) ate, fruit(obj)"
09:55 < yangman> making the connection back to "what's important", you can say that while the fact that a fruit was eaten is important in the first, in the second it was the action
09:56 < yangman> making the connection back to FORTH vs Lisp, you have "what object did I manipulate" vs "what manipulation did I perform"
09:57 < yangman> anyways, that's my outline for a blog post
09:57 < yangman> I got 10 minutes into trying to find linguistic papers on the phenomenon I was describing, but gave up
09:57 < n0ob> that is an interesting point
09:57 < lumy> heh.
09:58 < yangman> but, since I just typed all of that out, it's going on my blog

(I graduated from university with a joint degree in Computing Science and Linguistics. While no longer fluent in Japanese, I lived and went to school in the Tokyo area for 4 continuous years, leaving the country for Canada at age 12.)

11:14 < krichter|work> yup, perforce
11:14 < yangman> p4 is like a sleigh dog with some weird mental condition
11:15 < yangman> everything is fine and peachy most of the time, but occasionally, for no reason, it’ll tear up all your socks or run your sleigh into that tree
11:15 < yangman> that specific tree, around the corner, by the shed
11:15 < yangman> you don’t know why
11:15 < yangman> but, at the end of the day, you’re just glad it’s there at all, because those 400kg crates don’t haul themselves
11:16 < n0ob> also, branches seems to require brain surgery
11:16 < cdemwell> lol
11:16 < yangman> yeah, it’s a trick you’ve heard that the dog can do, but you’re always afraid to try because you notice people that say they do them are often missing limbs

With 12GB of RAM in my desktop, occasionally hibernating Linux in order to reboot into Windows would take, literally, minutes. With 2.6.37, however, there is finally compressed hibernation image support in mainline, significantly improving shutdown and boot-up time.

Joy of joys.

Now, if I actually had time to actually play games…

I present to you, Exhibit A:  Bug #1339—lighttpd doesn’t set empty QUERY_STRING in cgi environment

Now, please direct your attention to Exhibit B: RFC 3875, section 4.1.7. Specifically, the below quoted paragraph:

   The server MUST set this variable; if the Script-URI does not include
   a query component, the QUERY_STRING MUST be defined as an empty
   string ("").

A potential  poit user recently encountered this problem.   While I would like to update the lighttpd bug entry alerting them to this crucial violation of specification, I’m also unwilling to sign up to yet another bug tracker just to fire off a single comment; avoiding such overhead is precisely the reason why I support OpenID.

Update: Mere minutes later, said user has pinged the bug.  Hopefully this leads to a resolution more correct than the first.

Planting the Seed

My earliest memory of using a computer is way back in the late 80′s, playing Pacman on a who-knows-what in a computer lab at my father’s university.  I wasn’t very good at it, but I was also five. There is a photo from even earlier of baby-me, wearing pants with a butt flap (hanging open; no diapers), back-side to the camera, again in a lab of one of my parents’. I’m standing on a chair in the photo because it would be impossible to reach the keys otherwise.

The whereabouts of the photo is unknown, but I have seen it, and I am damn proud of its existence.

I inherited most of my technical-mindedness from my father, who is a very capable hacker of his own right.  The story goes that although we couldn’t afford a TV of our own in China, we would almost always have one to watch because he was constantly fixing them for other people. To this day, if any sort of electronics break in our household, it will often be disassembled, then repaired or scavenged for parts.  It’s getting harder and harder, of course, but I can’t remember the last time we actually disposed of an appliance because it was deemed mechanically unrepairable.

There’s no being humble about it: I have The Knack.

Sprouting

“I got my start early” would probably be the expected opening to this paragraph. But, I didn’t. Owning a personal computer in China in those days was unthinkable.  It was less so years later in Japan, but still something very much unattainable on my parents modest income.  Less than a year before leaving Japan for Canada, my family received a used computer as a gift. I managed to find a few programming books from my elementary school’s library that teaches game programming, but only to copy the final code samples so I could play a few games. Although the interest was there, I was simply unwilling to put in the effort to learn programming.

My first serious dive into software development happened in the late 90s, towards the beginning of high school, when I got my hands on a copy of Macromedia Flash.  Not unlike most other youth with similar dispositions, the want to become a video game developer was a significant driving factor.  With my previous attempts at Java Applets and C++ failing to yield results, having a built-in vector graphics engine was an instant win.  My Flash development, programming wise, peaked when I produced a discrete 2-body gravity simulator for my Physics 11 final project. I was, no doubt, the biggest nerd in my class.

From there, I expanded to Delphi Pascal (to build a MUD client plug-in), took a brief detour in x86 assembly (Delphi’s substring replace function was too slow), picked up Java when I became a developer for the above mentioned MUD, finally learned enough C++ to hash out a few simple CLI applications, then dropped way down on the abstraction scale to PIC assembly for a self-directed project in electronics for my final year in high school.

A couple months later, I started formal studies in computing science at Simon Fraser University.  This would also be the first time since elementary school that I am taught programming in a classroom.

Growing Tall

The rest, as they say, is history. Theories were learnt, jargon was absorbed, and line after line of code were written.  I got involved in open source development, worked a handful of semesters at an actual software company, and, eventually, got a piece of paper declaring me a Bachelor of Computing Science.  There are even people using the code I’ve released to the world.

As a hacker, life was good.

A Drought Approaches?

Life, though, wasn’t good.

I’m not enjoying this.

Somewhere along the way, the side of me that hungered for intellectual knowledge and challenge had decided that it’s had enough.  What was once a hobby of novelty and intrigue has, over time, turned into the repetitive and mundane. While I still enjoy the act of problem solving, hacking for its own sake began to feel unfulfilling, banal, and, worst of all, pointless.

So, where to go from here? My long-term plans, as of the last few years anyway, have always been to make a graceful exist from the software industry after earning what I feel has been enough from it.  My hope is that once I am no longer programming for a living, I may be able to enjoy it again.  On the other hand, I am becoming more and more comfortable with the idea of no longer doing this at all. The knack and geekiness will always be there, of course, but, at least for the moment, my future as a hacker isn’t looking too bright.

Then again, who knows. I never planned to become a computing scientist, either.

The longer it takes to fix a bug, the smaller the fix will be.

When I first read this little gem of wisdom on DadHacker, I literally laughed out loud.  It was the same month that we finally resolved the once-infamous fd.o bug #13405.

#13405 was filed against xf86-video-radeonhd towards the end of November, 2007.  xf86-video-ati has a counterpart as bug #19215, but filed almost a year later.   Despite a handful of very bright core developers and a few dedicated newbies (including yours truly) looking very closely at the problem, the fix was not to be committed until early May of 2009.

That’s over seventeen months.

To be fair, I don’t think there were any serious attempts at actually resolving the bug until I came along a year after its initial filing, eager to prove to the open source world that I have what it takes.  Even accounting for that, however, this was a process that took roughly five months.

In that time, a couple other commits fixing cursor corruption were made, but they turned out to be additional bugs that were only affecting certain hardware.  While both happened under equally peculiar circumstances, the original bug remained unresolved, despite multiple pairs of eyes and man hour investment in the double digits.

Finally, in the first week of May, 2009, the ever-awesome Alex Deucher made commit da58e351 to xf86-video-ati, which I promptly ported to radeonhd, finally closing a year-and-a-half old bug.

So, what was the problem that plagued us for so long?

-	RHDRegWrite(Cursor, Cursor->RegOffset + D1CUR_CONTROL, 0);
+	RHDRegWrite(Cursor, Cursor->RegOffset + D1CUR_CONTROL, 0x00000200);

One bit.

Laugh out loud, indeed.

While job hunting last year, the need to constantly write about my programming background prompted me to start writing an autobiography specifically about my coding history.  It’s a project I’ve been working on sporadically for the past several months, and has helped me remember some fairly obscure details of my past that, frankly, would have been useful during some interviews.

One of the most prominent parts of my software development history is my involvement with Sharune.  Today, while attempting—and so far failing—to locate some old Delphi code written for this little MUD, I decided to do a search for the thing that I had built to see if it had become yet another artifact of the web.

No such luck.

What I did find, however, was this old forum thread of me seeking help with what was the precursor to that project.

This, essentially, is the moment I stopped being just a video game nerd and started becoming a “fucking prolific” hacker.

We get this question a lot about radeonhd: “When is 3D going to be supported?”

The short answer: never; 3D acceleration is the domain of Mesa.

But, of course, it’s not as simple as that.  For the long answer, read on…

More than Meets the Eye

Your Linux “video card driver” is not a single blob of code.  As with any sufficiently large piece of software, things need to be broken up into discrete components for both technical and maintainability reasons.  And, there is no doubt a complete graphics stack is large: the AMD Catalyst driver is a 93MB download.  That’s over one hundred Mega bytes of executable code and supporting data.  In that humongous pile of bytes are roughly half a dozen different components working together to deliver your polygonal needs, without counting the configuration tools.  This, unfortunately, is not common knowledge.

In the open source world, our drivers are much more svelte, partly because less features are supported, and also because application-specific optimizations are omitted.  Unlike the proprietary drivers, however, the boundaries of separation are clearly visible.  In respect to “driver” code that is specific to a particular hardware, there are, more or less, three separate components involved:

• X display driver
• Mesa DRI driver
• Kernel DRM driver

Indeed, in true UNIX fashion, all of these are actually worked on as completely separate projects, albeit with significant overlap in the personnels involved. (Proprietary drivers from AMD and nVidia provide equivalents to all three in a single package.)

DDX: People’s Favourite Barking Tree

When the words “Linux video card driver” are invoked, the DDX is what people typically think of.  And, for good reason: historically, when configuring X11, the DDX was one of the few things that had to be configured by the user, and could be selected. This is your radeonhd, radeon, nv, and intel.

The main function of these drivers is to provide mode setting: that is, selecting the screen resolution, colour depth, turning outputs on and off, etc. The X server uses the DDX to perform basic, essential hardware manipulation so that you can have a graphical display at all.  Each DDX is hardware-specific: indeed, it stands for Device Dependent X (but this trivia is not very important.)

That’s not all.

A DDX also provides hooks that allow the X server to offload certain drawing operations to the GPU so that they can be accelerated.  XAA, EXA, and UXA all fall under the DDX’s domain, in addition to Xv.  The former group is responsible for things like moving windows around your desktop, scrolling a website in your favourite browser, and compositing desktop effects.  The latter handles video rendering.

To summarize, everything 2D is the domain of the DDX.  And, in simple terms, it is its only domain.  If you are looking for 3D acceleration, this is not the tree to be barking up at.

Mesa DRI Driver: The Invisible Hero

OpenGL is the method with which applications achieve 3D rendering under Linux.  Unfortunately, but for good reasons, it is not possible to go from an OpenGL function call to manipulating hardware states in a single step.  Multiple translations must be done under the hood to convert those calls into something that the GPU can understand.

Mesa is largely divided into two parts: an OpenGL compatible library stack that applications can call into, and DRI drivers that translate Mesa-internal instructions into forms that the video hardware can understand and execute.

After a call is made to the Mesa library and it has been translated into a GPU-specific language, the resulting operations need to be sent to the hardware for execution.  Mesa, however, cannot do this without help as it lives in user space:  only kernel space code is allowed to touch hardware directly.  Note that DDX is one of very few exceptions to this rule: it is technically user space code but allowed direct hardware access, for better or worse.

Unfortunately, sending these instructions through the DDX, and hence X, is not fast enough.  This point finally brings us to…

DRM: Bridge to the Hardware

The final piece of the driver stack is the Direct Rendering Manager.  Oversimplified, it exposes certain features of the hardware with an abstraction layer to user space such that applications like Mesa can make use of a GPU more efficiently.

It is also the job of the DRM layer to provide measures that ensure multiple applications that require working closely with the hardware to not step on each other’s toes.

DRM drivers are Linux kernel modules, which are accessed by user space through libdrm.

All Together Now

So, say you have a shiny new RadeonHD 4870.  What driver features would it take to have it “working under Linux”?

First of all, you need a DDX that can properly mode set the GPU. radeonhd and radeon have both been capable of this for some time.

For fast 2D, again, support needs to be added to the appropriate DDX.  Mostly done, as above.

In terms of accelerated 3D, support from both Mesa and the kernel are required.  As of this writing, work on both are progressing nicely but remain in experimental states.  And, in this particular case, changes were required to libdrm as well.

It should now be clear that, for newer video hardware, there is rarely a trivial answer to the question “Does it work in Linux?”  In fact, it can be downright confusing.

Now, Lets Make It Complicated

Video hardware rarely change completely from generation to generation.  This means that the same code is often times used to operate GPUs from multiple generations.  Understanding which version of which component you need can occasionally become a messy affair.

To draw an example from the Radeon world, R3xx (Radeon 9xxx) and R5xx (Radeon X1xxx) GPUs share a similar 3D engine, but the mode setting engines are drastically different.  Therefore, while you can use the same Mesa DRI driver on both (r300_dri.so), radeonhd will not work on the former.  As for DRM, radeon (note that there is a kernel module and a DDX bearing this name) takes care of everything. (There are, of course, oddities from those generations that do not fit in to this generalization.)

Furthermore, there is currently a push in Linux to move video mode setting into the kernel (Intel is already there).  Kernel Mode Setting, more commonly referred to as KMS, brings the user- and kernel-space separation back to X, relieving the responsibility of hardware manipulation from the DDXes and putting it solely in the DRM drivers.  The details of this topic will—perhaps—be explored another time.

So, Now You Know

Armed with this knowledge, your task is to now educate your fellow Linux users regarding the mistaken concept of “The Linux Graphics Driver”.  Your efforts will, hopefully, help reduce the need for developers to repeat this explanation, again and again.

Whether you’re just learning git or want to learn a new trick or two, git ready is a pretty good place to go.  The listing of articles by task goal instead of command explanation should be especially helpful to the newbies.

Of my three Gentoo machines, two of them are running a rather scary combination of stable and unstable packages and an overlay or two via layman, sprinkled with ebuilds gathered from Gentoo bugzilla along with a few of my own.  It’s become a maintenance nightmare, especially since the non-layman ebuilds are not under version control. I’ve finally fixed that today with a portage overlay of my own.

My server is set up as the master, serving the overlay to the desktop, laptop (which is set up to emerge packages only at home, where I have a local distfiles mirror and distcc to the Core i7 machine is availble) as well as itself.  Since all of my git interactions with the server is done over ssh, the layman URL is pointed to a cgit location so that it can be pulled without special access.

Finally, the whole shebang is mirrored on yangman.ca so that anyone can access it.  The public overlay list is described at http://yangman.ca/gentoo-overlays.xml.

It’s a rather convoluted set up, but it allows the widest access to my overlays without pointing the Internet at my home server.

The first packages to make into the overlay are gnome-mplayer and gecko-mediaplayer, giving them the proper version bump love.

Hopefully, this will bring some sanity to my various /usr/local/portage.