Project Aardvark

CFUNITED and a Request for Comments

Monday, June 27, 2005 posted by Benjamin Pollack

I want to encourage all of you who are in the DC/Baltimore area to come and check out Project Aardvark at CFUNITED. You can talk to us, check out our software, and just chill. Instructions for how to get a free pass to the Fog Creek booth are available at Joel on Software.

Also, rather than do more long entries, I thought I'd try something a little bit different. Quite a few of you have tried to get in touch with me through various means, including posting in various forums, filing reports with Fog Creek Customer Service, and emailing other poor saps who happen to be named Benjamin Pollack. (Incidentally: my name is Benjamin, not Ben.) I don't really get any of those, except in rare circumstances, which is frustrating for both of us. Rather than force you to go through all of that effort just to have your email lost, I'm inviting all of you to email me at my actual address, bpollack.fogcreek@com (flip @ and .). I'll read through your mail (or at least what I have time for; if I get deluged, I'm going to have throw out what I can't get to) and post an article after CFUNITED responding to some of the more interesting messages. If it goes well, I'll consider having mailbag days on a more regular basis.

Just to be clear: "going well" does not mean I get all nice emails (although I certainly welcome them!). It means that the ones that are criticism are well-written and informative. I certainly don't have any misconception that I've had anywhere near the experience of Joel or Eric or the other big bloggers; my entries are about how a senior in college with eight years of hobbyist experience views the world. If you feel that I'm clearly off in left field, email and explain why! The worst-case scenario is that I still disagree, and either way, you've got a chance to make your case in front of 30,000 readers.

So send me your feedback! I look forward to reading it.

A Thousand Eyes

Thursday, June 23, 2005 posted by Benjamin Pollack

FIX AND CONTINUE

Firstly, thanks to the numerous people (including Joel) who pointed out that VisualStudio.NET does support fix-and-continue for C/C++ and has since Visual C++ 6. Oops. The reason I didn't know that was that incremental linking was turned off, since the project was created by importing a Visual C++ 4 project, and fix-and-continue requires incremental linking in order to function.

My new question, though: when incremental linking is disabled, and you modify a file in a running program, the dialog box that pops up just says something along the lines of, "VisualStudio needs to restart the program for your changes to take effect." Since there aren't a ton of compilers that support fix-and-continue, it never struck me as particularly odd that VS.NET would be incapable of it. Why can't the dialog box say, "This program was not compiled with incremental linking, so VisualStudio must restart the program if you want your changes to take effect immediately" with the options "Restart With Changes," "Stop," and "Continue Without Changes"? That would at least give enough information to know that there is this thing called incremental linking that means I don't have to restart the program, and my new message is no longer than their existing one. Using verb-based buttons instead of Yes/No/Cancel would also mean that you could skip the text if you really wanted and still figure out what you need to do based on buttons. Just my two cents.

BRANCH FOR CFUNITED

And in other news, Tyler and I are about ready to tag one of our builds as the official CFUNITED build. We're hoping to do that about lunchtime. Yesterday made me very confident about the build we'll be using; I managed to shut almost ten bugs that have been nagging me for a couple of weeks and get things like user-friendly error messages up that contribute to users thinking a program just "feels right." I'm pretty confident that everything will be working well for CFUNITED.

After the conference, I'll be turning my attention back to the UI of one of the client programs. One of the programs is basically done at this point; it's been refactored, looks nice, has internationalized error messages, works pretty well, and as of yesterday is basically feature-complete. (It's missing one feature that I'm adding today, but that's it.) The other one works OK, but needs a heavy refactoring in order to make some superficially trivial UI changes, and then I need to just shred both executables until they get smaller (they're 500 kB and 300 kB; I want to get them both under 200 kB). It's nice to finally see everything pulling together into a cohesive product that people will actually be able to use.

The REPL in .NET

Wednesday, June 22, 2005 posted by Benjamin Pollack

For the past few weeks, I’ve been working almost all of the time in C++. Every once and awhile, someone files a new bug against the Reflector, and I go in and work in C# for an hour, but by and large it’s all C++. Not only that, but the C++ code I’m working with is was fairly poorly designed; it’s very hard for me to check in more than a couple hundred lines per day on a very good day, simply because so much of my time is spent trying to figure out how exactly to modify the code we’ve got to do what I want.

Just as a random example about what I mean, one of the applications currently does not display its main window until after a lengthy connection procedure. I want it to display its main window immediately. This ends up being a very, very complicated change due to a bunch of assumptions in the code about when the window gets displayed, but I’m not even focusing on that. I’m focusing on figuring out what code to modify. There are variables called m_hwnd and m_hwnd1 and WNDPROCs named WndProc and WndProc1 that represent entirely different things. Moreover, just for fun, though m_hwnd and m_hwnd1 are different window classes and represent radically different functionality, their respective WNDCLASSes are given the same WNDPROC. To fix that problem, the programmer changes the WNDPROC on a per-window basis eight lines later by calling SetWindowLong and passing in the actual WNDPROC. Why? I don’t know. There are no comments in that method. Even if there were, this obfuscation makes it needlessly difficult to figure out which window is which. After carefully reading through a printout of the code for 40 minutes yesterday evening and scribbling notes all over, I renamed the variables m_hwndViewport and m_hwndMain. But really, why weren’t the variables just given sensible names to begin with?

That’s actually not what drives me most insane about working in C++, though. What drives me nuts is the lack of a REPL.

REPL stands for read-evaluate-print loop, and refers to an interface that takes a line of programming code, executes it, prints a result, and immediately returns. It’s basically just a fancy name for a command line. Most dynamic languages have a REPL, or something similar. Ruby has irb, the interactive Ruby interpreter, Python fires up in interactive mode by default, and so on.

The thing is that a language based around a REPL actually changes how you go about developing code. When you work on C++ (or any language without a good interpreter), generally your development procedure goes like this:

1. Write a core piece of functionality
2. Litter it with print statements if that’s your thing
3. Compile (this still takes 30 seconds for our unoptimized 400 kB program)
4. Fire it up under a debugger
5. If the program starts behaving oddly, put in some breakpoints
6. Start stepping through code when you hit one, inspecting variable contents and paying careful attention to the call stack
7. Discover suddenly what you did wrong
8. Quit the running program and make some trivial modifications
9. Recompile (another 30 seconds) and try again

There are three problems here: firstly, you keep having these breaks where you need to recompile the program to integrate your change. Sure, they’re “short,” but it’s enough to break your flow for a few moments and make you lose track of exactly where you were going with your train of thought.

Secondly, there’s no quick check for ensuring that your fix actually works; you have to make your best attempt to patch the code based on your understanding of the error and start the process over again. If you analyzed the problem correctly, your chance of getting it right is very, very high, but you still may have made a fairly bad mistake if you even slightly misunderstood the error domain. (Tyler and I, for example, had an incredibly taxing couple of hours yesterday simply trying to figure out which program was causing one of our bugs, and we misdiagnosed the issue several times in the process.)

Thirdly, you can’t easily write and test just a small amount of code. In most languages, your only option if you want to do this is to write a test suite for your class protocol, which takes a lot of time during development and probably isn’t necessary for a lot of smaller stuff.

Developing with a language centered on a REPL is a totally different experience. Here’s basically what developing in Smalltalk looks like, for example (though Common Lisp and Scheme  work basically the same way):

1. Add or modify a few methods in a couple of classes
2. Open up a Workspace (Smalltalk’s REPL) and print out the results of some arbitrary code that tests what you just wrote
3. If you hit a bug, you can easily inspect any value in the entire system to find the error
4. Once you find it, make the change, massage any “damaged” data back to pristine state by hand using the Workspace, and then resume execution where the breakpoint happened to see whether your fix worked
5. Repeat

Notice: there’s no compile phase, it’s trivial to work on small pieces of code, and hitting a bug does not mean rapidly switching gears, but instead means you to try your fix in the running code right at this very second, and you get immediate feedback on whether your fix was correct. Step 4 has been rechristened fix-and-continue and has appeared in a more limited way in some more traditional languages, such as Java 1.4, C# in .NET 2.0, and C and C++ in Apple’s versions of GCC, but the other parts of the process simply don’t really exist.

The good news is that you can approach a REPL style of development in .NET by using an interpreter. Although I don’t quite buy that .NET is language-neutral (write in any language, as long as it’s object-oriented and has no more or fewer features than C#!), that mantra has brought forth tools like IronPython, a Python implementation that runs on .NET and takes full advantage of .NET’s toolchain. When I was working on the Reflector, the fast compile speed of C#, combined with IronPython’s ability to load assemblies on-the-fly, meant I got very, very close to REPL-style development using traditional tools. I would write some code in C#, hit compile, jump to IronPython, reload the assembly, and check out the code I’d just written. When I found bugs, I’d run some code against the problem classes, try out a fix or two in Python, then move that fix to C# when I was convinced I had properly diagnosed the problem. The constant shift between Python and C# was a little bizarre, but eventually my brain just got used to the concept that debugging was in Python and new code was in C#. The ability to so rapidly test and diagnose errors played a big part in letting me get the Reflector done as quickly as I did and actually having the result work.

If you’ve not tried REPL-style development before, but are afraid of or don’t have time for languages like Smalltalk and Common Lisp that are built around them, I’d still strongly encourage you to give a tool like IronPython a try. IronPython 0.7.5 runs on .NET 2.0 beta and is available from Microsoft. IronPython 0.6 runs on .NET 1.1 and is what I’m using here. And, if you like it, consider trying out one of the REPL-based languages. I have a hunch you’ll find it highly addicting.

RANDOM THOUGHTS

Thursday, June 16, 2005 posted by Benjamin Pollack

More miscellaneous updates (I'll post a coherent thing tomorrow maybe; I just don't have the attention span for it right now):

PREVIEW OF HEROES OR ZEROS

Not for you; for us. Lerone sent us the first few minutes of the documentary he's working on. It's extremely weird to see yourself in a movie, but it's actually looking pretty good.

The clip Michael posted, incidentally, is not from the documentary. He's been videotaping like nuts on our DV, so that's what that's from. (Lerone's filming in HD on a very nice Sony that he nevertheless feels perfectly happy sticking out taxi windows for good shots.)

ALMOST...THERE...

Today, we demoed a complete end-to-end run of Aardvark for Joel. It crashed in a way it's never done before, and we can't replicate it. So that stank. I'm spending the rest of the day now systematically trying to reduce the number of bugs assigned to me in FogBugz to zero, which, according to my best estimates, is about 18 hours of work. Hopefully, that estimate is more liberal than it needed to be. I'm also doing really hard things, like adding a Fog Creek logo to the about boxes. Man, some days...

LEARNING WINDOWS

I finally concluded today that I like the current Windows API over the old classic Macintosh API that Apple had in Mac OS 9 and earlier. So if I ever develop a time machine, go back to the late nineties, and get into programming on Windows NT and 2000, this would be my platform of choice.

GETTING OUT

I think I've honestly gone out more evenings over the past week than I went out last semester at Duke. New York is just so much better for that kind of thing (go New York subway!). Friday was Broadway; Saturday was a movie; Monday was something I don't remember right now, but it was good; Tuesday was hanging out with a friend from Duke and his 7 closest high-school pals; and last night I had a spectacular dinner with a good friend of mine I hadn't seen in a long time. In case that's not enough, tonight Fog Creek's all going out to the Yankee's game, Saturday all of us interns are going to a comedy club, and there's a very high chance I'll get stuff lined up for Friday or Sunday too. That I'm still coherent at work is somewhat amazing to me sometimes.

ON SPECS

So, I rambled off earlier on how specs make life a lot easier, and I still stand by it. We're keeping the spec up-to-date religiously, so at any given point it's a superb reference to see how things ought to work. Occasionally, though, the spec can lead to a lot of confusion if you don't use it right. For example, we added some challenge/response stuff to the protocol. Tyler and Joel worked out how that would go, and added it to the spec. I then modified the Reflector to obey the spec. In the process, I found some slight ambiguity in exactly how things should be implemented, so I picked a way, modified the spec appropriately, and promptly forgot about it. The next day, I forgot that I'd tweaked the spec, remembered only the original passage, and implemented the challenge/response in the client programs in exactly the wrong way. It was very quick to fix, but also a bit annoying to realize what I'd done. So: if you're going to keep the spec religiously up-to-date, make sure you also read it, or it won't do much good.

WEATHER

Why is New York insanely hot for a week, then, when it finally cools down for a baseball game (perfect!), it rains. This is not fair.

AND FINALLY, A SHAMELESS PLUG

To any comp sci majors or minors at Duke who are reading this blog regularly and looking for a half-credit course, I would strongly encourage you to sign up for my house course (HOUSECS 79.04: Open-Source Design and Methodology). There are still 15 seats left, and Drew and I would love to have you there. We won't have iPods for ya, but I promise it'll be a tremendous amount of fun. I'm hopeful we'll actually build a real open-source product and "ship" it before Winter break (that part will be optional though if we do). Best of all, it's pass/fail, so you can't go wrong.

Random Musings

Tuesday, June 14, 2005 posted by Benjamin Pollack

REPLICATABLE HEISENBUGS

Tyler and I are getting into a routine whenever we have inexplicable crashes in our C++ code. First, check to see whether an instance variable is getting trampled. If it is, copy files modified since you last checked in somewhere else; blast the entire directory; do a fresh checkout from Subversion; and then copy the files back. Simply clicking "Clean Solution" is not sufficient. Any ideas on what's going on here? (And yes, we were smart enough to check for invisible files.)

GOOD CODE DESIGN

For the love of...something, if you're going to take the time to write classes, then please make at least some attempt to split your user interface code from your implementation code. This class I've been working on for the last two days is 4000+ lines long and is basically the entire program. I wish they'd just used procedural C instead; it'd be a lot easier to work with.

On a similar note, if you have two procedures literally named "Proc" and "Proc1", with no comments about what they do, you may need to get out more.

BROADWAY

Over the weekend, my family was in town, and we had a chance to see Wicked. Watching a Broadway show truly on Broadway, instead of on tour, is a truly awesome experience; I'm trying to figure out now which to go to next.

THE ZIEGFELD

I also saw a movie with a friend of mine at this theater called the Ziegfeld. The Ziegfeld is a digital movie theater that looks like a concert hall with a screen instead of a stage. The theater is truly beautiful, the seats are spacious and comfortable, and the sound system is awesome.

OUR WORK ENVIRONMENT

So I thought you all might like some photos of our corner of the office. Also, Tyler challenged me to make the final executable smaller than the total size of this website, including pictures, so I'm going to skew things in my favor.

All of us chilling in the main office. I think this was about two weeks ago when we were just getting started. New Jersey is out the window.

Michael working away on the e-commerce engine.

Yaron deep in thought about naming possibilities.

Tyler testing the latest security enhancements while listening to the company iTunes repository.

Tyler and Michael. To the back right you can see the main office.

Me working on making GPL code do what we want.

Fixing Other People's Code

Monday, June 13, 2005 posted by Benjamin Pollack

It’s almost a truism that programmers would rather write new code than modify existing code. The truth of that adage is doubled when talking about modifying someone else’s existing code. Unfortunately, that’s exactly what Tyler and I have been doing nonstop for the past two weeks, and let me tell you, it has not always been pleasant.

After we got the source code the GPL component to compile in VS.NET, the next few days were spent, more than anything else, simply getting a feel for the code base. Different programmers have different conception of what orthogonal design entails; the degree of separation, the function of different components, the interfaces between classes, all vary dramatically from coder to coder and company to company, and invariably your conception is not quite the same as the fellow next to you. Understanding the design of the existing base, then, becomes very important to ensure that when you start making extensions, you do it right.

The problem is that when you’re first getting started, the chance of getting it truly right the first time—especially if you’re making large-scale modifications—is very low. In Aardvark, for example, I initially attached some new connection-specific handshaking code inside a class that controlled the windows taskbar icon for our program. That probably sounds a bit silly, but it actually made a reasonable amount of sense. The taskbar class was where connections were created and destroyed. Since that code was written for multiple connections, and I was restructuring it to allow only one connection, adding in some handshaking code into the new connection code there seemed as if it made sense at the time. It also had the benefit of keeping all Aardvark-centric modifications in just one class.

Needless to say, that was a bad call and caused problems down the road when Tyler started adding encryption code and needed to hook it in after the connection had been started and after Aardvark handshaking, but before any other communications took place. So that had to be refactored heavily. Thank heaven that Subversion lets you move files while preserving history.

It’s doubly a problem when two coders are trying to come to grips with the code base at the same time. While I was on one end of the room, adding handshaking code and modifying the interface, Tyler was on the other side of the room trying to make the connections use secure sockets. That meant we were each trying to modify the same section of code at once, and because we each had different concepts of what the original programmers had in mind from a style perspective, we stepped on each other’s toes a lot even with Subversion’s assistance to keep things straight. Despite our best efforts, checking in code that worked fine on one machine but that broke the other’s code base became a frequent occurrence.

(Thankfully, Subversion has numerous features—such as repository-wide versioning that lets you revert the entire source tree to the way it was at a specific point in time, powerful change tracking tools to figure out quickly which lines of code broke the program, and solid merge support—that made this far less painful than it could have been. And, of course, Subversion integrates very well with FogBUGZ, letting us track easily which check-ins fixed which bugs. Without Subversion, it would have taken us much longer to get here.)

Even if we had instantly made all the right decisions, though, stylistic variations would have hit us in another way. Large projects tend to have multiple standard styles for different components. The TightVNC project is a great example of this. vncviewer, which is the part of VNC that lets you control other machines, codes directly against the Winsock API in its ClientConnection class, whereas the sibling WinVNC server has a VSocket class that abstracts all socket communication and uses that class in vncServer and vncClient classes. WinVNC likewise makes extensive use of Windows’ Unicode string manipulation functions, whereas vncviewer codes against the ANSI functions. Classes in WinVNC are normally called vncFooBar or VNCFooBar, whereas classes in vncviewer are called just FooBar with no prefix. WinVNC makes extensive use of a pattern for dialog boxes that relies on threads; vncviewer lacks such a pattern.

This is not a problem in any real way; both code bases are solid, fairly consistent internally, and most importantly, as anyone who’s used TightVNC can tell you, very reliable. It’s just a bit confusing if you’re working in a project that has two fairly different styles to keep your coding appropriate, and the different high-level designs increases the chance that you’ll make a bad design decision by applying one project’s architecture to the other.

Would it be worth cleaning up? Almost certainly not. It would take a lot of time to get everything refactored to have a consistent style, and would come at no benefit whatsoever to the end-user. It does perhaps emphasize, though, that large projects (including Aardvark, thankfully) can benefit by having standard coding conventions to ease moving around the code base.

Ethereal: Life Saver

Thursday, June 9, 2005 posted by Benjamin Pollack

I've been sick and not sleeping well for the past few days, and it's finally catching up to me, so this will be short, but I want to take a moment to give a hat-tip to the Ethereal project. Ethereal is an open-source tool that lets you monitor all Ethernet communications. It has a decent GUI, provides robust filtering, runs very efficiently, and has been invaluable over the last couple of days in finding bugs in our code. I fixed three or four major crashers today, including one that's been nagging me for over a week but that I haven't been able to troubleshoot due to lack of data. Getting rid of that bug made me feel much better (and also solved some superficially unrelated problems as a side-effect).

The good news is that Tyler's making great progress on his security code, the e-commerce extensions are looking good, the Reflector's getting increasingly robust, and with these major crashers in the clients gone, I'm free to focus on the GUIs and enhancing usability. We've had various levels of prototypes running with increasing robustness over the past week, and I've got great confidence that we'll have a solid product ready for CF United. We'll undoubtedly have lots of bugs to fix in the beta cycle, but it's nice to be getting to the point where we could almost demo Aardvark all the way through and have everything work.

Impressions of NYC

Tuesday, June 7, 2005 posted by Benjamin Pollack

Last week, I promised that I would post about how you can get a REPL in .NET and why you care. I failed to actually follow through on that promise, largely because I forgot, but also because I have been spending an awful lot of time attacking heisenbugs that result from my still rather gap-strewn conception of how Windows message processing works in multithreaded applications. So I’m sure that everyone will be very excited to see that I remember my promise, and today will happily follow through by writing instead some random musings about New York City. This is the fourth in a six-part series. I will be writing entries five and six next, and then go back and write the first three entries at some later point when I feel like it.

(Hey, I can’t blog on technology all the time, but if you’d at least like something computer-related, I’ll point you to Maciej Ceglowski’s response to Paul Graham’s famous “Hackers and Painters” essay. (Note: essay is not for the humor impaired or people who like to show how “mature” they are by getting offended.) Otherwise, stay for the chat.)

I come from a small town near Indianapolis, Indiana. I have always liked Indianapolis. The city, especially in the last ten years, has grown tremendously, yet it’s still very easy to drive for ten minutes and get to rural suburbia. Another ten minutes can usually put you in at least fields and usually small forests, if you know where you’re going.

Duke is in an even more rural setting. The Triangle actually has a fairly sizeable population, but it’s spread over such a wide area that the overall feel is that you’re never really in an urban area. Downtown Durham, Raleigh, and Chapel Hill are far less developed even than Indianapolis, which has a small but surprisingly vibrant city center.

So it shouldn’t be much of a surprise that trying to live in New York at first was like having a massive caffeine IV jammed into your carotid artery 24/7.

Joel gave us each a book, Time Out in New York, that lists lots of stuff to do in the city. The book is as thick as AAA guides for some of the states I’ve visited. In the book’s introduction, it states something along the lines of, “New York has a reputation for being this insanely packed metropolis where everyone is always in a hurry and small children end up being shoved off the sidewalk into sewers where they are devoured by rats the size of some of the smaller models of Toyota. This is in fact not the case anywhere, except maybe on Fifth Avenue, and even then only during peak business hours, and besides, many New Yorkers have domesticated sewer rats as exotic pets.”

This is a lie. Not only have sewer rats not been domesticated; the sidewalks really are packed everywhere. At least half of the city generally seems to be on the same sidewalk as I, going in exactly the opposite direction with such rapid speed and with such concerned expressions that I have to assume they have severe problems containing their bowel movements and are in urgent search of a restroom. That, combined with New Yorkers’ love for using their horns to convey such messages as, “We are stopped,” “we are moving,” “my car is turned on,” and “I have a horn” meant that I was pretty much on sensory overload for a few days.

Once you get past that, though, and get used to the fact that the city is always moving, New York is awesome. The same fact that makes the sidewalks packed also means that there are a tremendous number of people to meet and that anyone who complains of nothing to do is just being lazy. Within one block of where we live, there are probably two dozen restaurants, numerous bars, a large shopping complex, and a posh gym. All the theater you could ever want is just a short subway ride away, and the subway runs efficiently and generally is easy to use (except at 3 AM, when the subways switch over to Random Track Mode, but that’s another story).

The buildings, too, are kind of incredible. The first week I was here, I was living on 33rd St. and First Avenue, and I actually commuted to Fog Creek on foot. Part of the fun of doing that was that I got to walk under the Empire State Building every day, which, at least if you haven’t grown up here, is truly an awesome experience. Indeed, height is everywhere; the apartment in which we’re staying is taller than any building on Duke’s campus, and would probably leave a decent mark on Indianapolis’ skyline. The city is one giant mechanical marvel.

New Yorkers have also been very friendly; everyone I’ve talked to has been more than happy to answer questions about the city, and some people, especially on street corners and on the subway at night, are even willing to provide me advice without my asking. Seriously, I know that New Yorkers have a reputation for being rude, but my experience has led me to think that that stereotype probably dates from a different decade. People have universally been helpful and curteous. The city also feels surprisingly safe, and it should; crime in NYC is among the lowest of the big cities in the United States.

What do I miss? For one thing, grass. There’s not nearly enough of it. Central Park is nice, but it feels (and is) artificial, a kind of oasis in the middle of a world defined by brick, mortar, concrete, steel, and a faint scent that smells suspiciously like pee and oil. There are no forests. I can frequently look up and see trees in penthouses, Brett maintains a tomato plant in the office, and I have a nice orchid on my desk, but it’s just not the same as being able to wander around real wooded trails a mile from your house. I also kind of miss silence when outdoors: although Fog Creek’s offices and our apartments are quiet except for the air conditioning (and when I’m cooking, the fire alarm), it is very, very hard to find any place outdoors where you cannot hear music, cars, or both.

Despite that, New York has been incredibly exciting, and I honestly think it makes a great place to live. I’m really looking forward to being able to explore more of what New York has to offer over the coming nine weeks, and have little doubt that I will be very sad to leave NYC in the fall and anxious to return.

Until then, sadly, it’s me versus heisenbugs. If you’ll excuse me...