Project Aardvark

Intern vs. Compiler

Tuesday, May 31, 2005 posted by Benjamin Pollack

OK, in Tyler’s last post, he hinted that there were some segfaults involved getting a certain GPL program to run properly in VS.NET 2003. I want to elaborate on what exactly happened, because it's happened enough hours ago now to be funny instead of humiliating. Note that this is going to get a bit technical and I probably will come off looking like an idiot, in which case all I can say is a) sorry and b) I’m still learning Windows and VisualStudio, so cut me some slack. :)

On Friday afternoon, having pushed as far ahead with the reflector as I could until we had some of the other stuff that needs it actually running, I downloaded the GPL program we’ll be extending, discovered that it had Visual C++ project files, and opened those in VS.NET. I checked out the README, which said things ought to build just fine on VisualStudio.NET 2003, so I thought it would be a piece of cake to get running. Bzzzt! I hit F5, the build starts, and then dies with my least favorite error message in the entire world:

“Internal Compiler Error”

Just in case you have never had the pleasure of getting this error message (and given that we’re currently getting a lot of Joel on Software readers here, that’s probably unlikely), it means that the compiler not only cannot handle whatever code you gave it, but cannot even give you a meaningful error message, because the compiler itself has a bug. Worse, I didn’t get this error in just one lone file; I got it in almost a hundred different files in the project. So, I began spelunking around the source code. After about 45 minutes of tweaking code, sometimes getting further, other times segfaulting the compiler rather spectacularly, I finally realized that all of the errors were coming from C++ header files. On a hunch, I went to the build settings disabled precompiled headers. I started the build cycle again, and, about 25 seconds later (I love these computers) had...missing symbol definitions. Well, fine, but at least missing symbol definitions I can handle. A quick check revealed that the missing symbol definitions happened because the projects were simply missing a dozen or so .cpp files. I quickly threw those into the project and rebuilt, and then got something that genuinely did not make any sense to me.

“Symbol deflateEnd@4 not found.”

The symbol I recognized; the project has a dependency on zlib, and deflateEnd looked like (and is) part of zlib’s API. Yet I could trivially verify that zlib was building properly and getting linked properly. The problem, I decided, was that deflateEnd@4 was clearly a mangled name (i.e., the name that was compiled into the library did not match its actual name), and the only time I’d seen names mangled like that was from C++ projects, so VS.NET must be compiling zlib as C, but the main program thinks it’s C++. (Experienced Windows programmers are now laughing maniacally; the rest of you can just follow along.) So I spent another 30 minutes trying to figure out why zlib’s extern “C” command in the header file wasn’t taking hold until I finally gave up and asked for Joel’s help. Joel comes over, identifies that this is not C++ name mangling, but rather Windows __fastcall mangling, and quickly generates a nice printout of the symbol table of zlib.lib where we verify that there are (surprise) only normal C names. Thankfully, I now had enough information to fix the problem. I changed zlib to be compiled with __fastcall, then (when that failed) as __stdcall, and finally, finally had something that compiled.

Well, almost. Turned out I needed to link against some parts of Windows that the project didn’t specify. But after that it compiled. Cleanly.

It just segfaulted immediately on launch.

At that point, I was too worn out to continue and decided to just leave the whole mess for later. I checked it into our Subversion repository with a note that it crashed instantly on startup and went home for the weekend.

So today, while we were trying to get VS.NET and friends installed on Michael and Tyler’s computers, Michael very kindly came over and started attacking the program with fresh eyes. He succeeded in recreating the full call stack at the time of the crash (which was actually surprisingly difficult due to threading issues), and with that boost I was able to determine that the crash was happening due to an instance variable getting trampled, which was happening due to a call to a string copy that seem to be overwriting most of a class’ data.

So Michael and I are sitting there, hunched over the monitor, trying and failing with idea after idea, when Tyler wanders over and asks, “Is this the same as what’s in Subversion?” I tell him yes. He goes over, opens the project, builds it, and hits the Debug button.

And it runs.

Perfectly.

We still have absolutely no idea what the problem was. I ended up blasting my entire source tree and checking out again from our repository, which ought to have been exactly the same broken code that I was working with before (I’m the one who checked it in, after all), but which now inexplicably works just fine. So now I’m finally able to hack on actually extending and modifying this stuff instead of waging a war with the compiler. I just feel annoyed, because I wasted several hours and I caused Michael to waste several hours when we could simply have switched to another machine. It wasn’t a complete waste of time—my initial project fixes were necessary (I double-checked), and I learned an awful lot about the VS.NET debugger—but I still feel like I could have gotten a lot more done if I’d just tried things on a different computer.

There’s probably a lesson to be learned here, but I’m not sure what it is, so instead I’ll just close by promising to keep you up-to-date if I discover my machine is possessed by demons. If it’s not, of course, you’ll have to make due with normal Project Aardvark updates. But here’s to hoping.

Reflector: Feature-Complete

Thursday, May 26, 2005 posted by Benjamin Pollack

For the past two and a half days, I’ve been methodically chopping away at one of the four major components in Aardvark. The experience was a bit exhausting; I went from knowing no C# and only a limited amount of .NET at midmorning on Tuesday to basically hitting feature-complete on the component called the reflector at 1:00 PM Eastern. Now, I’ll be the first to admit “feature complete” is a code word for “still has a ton of small yet annoying bugs that you ‘forgot’ about for the time being, and at least one major bug that, if properly triggered, will cause your computer to explode” (see antonym: “finished”), but the fact that I was able even to get that far that fast attests to two things: the power of a good development environment and the value of a good product specification.

As you have all probably inferred from Joel’s photo of all the books we have to read, Aardvark will largely be implemented in C#. I initially approached C# with a bit of trepidation. Although my bread and butter (and grades) depend on C++ and Java, anyone who knows me well knows that I’d much rather be working in one of the dynamic languages—especially Squeak Smalltalk or Ruby—than in any of the C descendents. My previous experiences with Microsoft languages (VisualBasic in all of its many varieties) did not exactly raise my expectations, and I was prepared for the worst. After a few days, though, I can honestly say that C# basically feels like Java with many of the problems removed and surprisingly few new issues of its own. The .NET library, too, has been very solid, and several components whose counterparts only just arrived in Java 5 (such as relatively easy-to-use non-blocking I/O) greatly expedited my work. I’ve also begun playing with Mono—an open-source/free-software implementation of .NET for Windows, Linux, and Mac OS X—on my PowerBook, and have been fairly impressed. .NET, at least in its non-graphical variety, seems likely to become genuinely multi-platform. The fact that .NET is so similar to Java in so many ways does mean that it is not revolutionary by any stretch of the imagination, but even an incremental improvement can greatly aid productivity.

Yet VS.NET was not the tool that most aided my coding. That honor goes to the Aardvark product specification. Last semester, I served as an undergraduate teaching assistant in a course that is supposed to teach advanced program design. As part of the course projects, students had to write detailed specifications describing how their project would work. A lot of the students loathed writing these specifications, and generally wrote as little as possible. They then plowed in with half-finished designs, changed everything as they discovered they had failed to account for basic functionality, and ended up with a product that had little relation to their original proposal. The end result usually worked; it was even frequently well-designed. But the students went through a lot of needless pain by not planning ahead, and then having to go back and rewrite large amounts of code.

The Aardvark spec, in contrast, is exceedingly detailed, and makes crystal-clear how each discrete component should interact with the others. Quick sketches of screenshots show how a user will interact. Protocol specifications detail precisely how various components will communicate. Questions that cannot be easily answered without running prototypes are prominently flagged so that we know where we will experience potential problems and can prepare for them with flexible code. That does not mean that all of the decisions are already made for us at all; the reflector clocks in at 950 lines of code, and designed its internals on my own. We’ll all have to do likewise on the other components as well. What it does mean is that we can write the components in isolation. Once I finish writing a solid testing framework (probably using NUnit), use that to iron out the remaining bugs, and do one last empirical verification that everything is working properly according to spec, I can know that the reflector will work when it’s dropped into the rest of Aardvark—even though none of those components currently exist.

That’s power.

Next week: details on New York City (short version: it rocks) and a discussion of bringing the REPL to .NET.

Getting Ready to Start

Tuesday, May 24, 2005 posted by Benjamin Pollack

Well, Project Aardvark may not officially start until May 31st, but I’ve already arrived in New York City and am working at Fog Creek to ensure that we can actually start on the 31st. I’m the first one here, which means that I have a lot of unique opportunities: I get first pick for where I’ll sit, which potted flower will grace my desk, and which chair I get. The flip side of that is that I also had the opportunity to unpack the 30+ boxes of computer equipment you saw in the photo in Joel’s last post Yesterday, I set up five computers, ten LCDs, a router, lamps, flowers, tables, chairs, networking cables, extension cables, and more, all under blazingly hot lights setup by Lerone so he could tape the whole thing. Oy.

The good news is that that’s all done now (this morning I even finished getting the last of the boxes collapsed and shoved into the garbage area), and the reward is that I’ve got an insanely fast computer and a pretty decent view of western Manhattan in an awesome office. I even got enough of my computer’s development tools set up to accidentally publish an article to our live server when I was testing out CityDesk. It’s truly a great work environment.

I’ll be spending the rest of the day (and probably the rest of the week) doing some extremely preliminary research and development of Aardvark and actually getting back up-to-speed on VisualStudio.NET and C#, neither of which I’ve worked with very much in the past, as well as reading in and out of the pile of books sitting on my desk. Once I’ve gotten a little bit more settled in, I’ll post on how .NET development compares to Eclipse in Java and Xcode in C/C++/Objective-C.

Meanwhile, it’s time to get back to work!