Monday, October 26, 2009

Bash vs. Which

The Unix which command is useful for searching your $PATH to find which version of an executable is going to run. But did you know that it doesn't always tell you the truth? For instance, it doesn't tell you when the command you're asking about is hidden by a shell builtin (or function, keyword, or alias):


$ which echo
/usr/bin/echo


which found /usr/bin/echo on your path, but echo is a shell builtin (at least in bash), so that's what will get run, not the one on the path. The GNU man page for which presents workarounds that will pick up functions and aliases, but shell builtins still slip through the cracks. The situation is even weirder on Solaris, where which is a csh shell-script, that sources your .cshrc file as part of its operation -- very strange if your shell is bash.

Clearly, which is not to be trusted. The easiest workaround, if you use bash, is the type builtin:


$ type -a echo
echo is a shell builtin
echo is /bin/echo


A slightly nicer workaround is to make a bash function for which, that uses type to do the right thing. I also like which to do an ls -l where applicable. Here's what I have in my .bashrc:


if alias which > /dev/null 2>&1; then unalias which; fi

function which {
hash -r
t=`type -t $@`

if [ -z "${t%%file*}" ]; then
for p in `type -p $@`; do
ls -l ${p};
done
else
type $@;
fi
}


Now which gives you much more information:


$ which -a wish
lrwxrwxrwx 1 root root 13 Dec 23 2008 /usr/local/bin/wish -> /usr/bin/wish*
lrwxrwxrwx 1 root root 7 Jan 8 2007 /usr/bin/wish -> wish8.3*


The unalias protects you in case some systemwide .bashrc or .profile aliases which in some way (unlikely, but good hygiene anyway). The call to hash clears the hashed function list, so that any changes to your PATH are reflected. The reason for the if statement in the function, is that we certainly want to report builtins, functions, etc., but if there are none, we can use type -p to make it easier to pass the filenames to ls -l. If there are builtins or such, we just punt and give the unaltered type output.

You can pass -a to this which if you want to see all the matches for the command, in order of precedence.

Note that this has to be a function, not a script, because running the script will source your .bashrc, which might change the PATH (all PATH changes should be in .profile, but sometimes people do it in .bashrc).

Friday, October 2, 2009

Csh Stupidity

One of the great things about the Linux era, is that bash seems to be taking over as the preferred shell to use, instead of csh/tcsh. The projects at my current job began during the SunOS era, so when I started there a couple years ago I went with the flow and let the sysadmin set up my default as tcsh just like everyone else there uses. When I was getting started, no one had a reasonable .profile or .bashrc I could use, so I went with tcsh so I could use the tribal .cshrc.

In terminals, the first command I run is "bash"; most of my work is in an emacs shell window, so my .emacs says (setq shell-file-name "/bin/bash") -- actually there's an (if) in there to choose cygwin's bash if I happen to be on Windows. I do a lot of shell programming at the command line, and I simply can't live with csh-style programming.

A few days ago I was working on my .cshrc file, and I noticed how dim-witted csh's if-else handling is. The classic anti-csh diatribe is Csh Programming Considered Harmful, which is an entertaining read, but there's some insanity that isn't even covered in that worthy rant. Let me start by presenting a correct csh script:
setenv FOO hello
if ($FOO == hello) then
echo hi
else if ($FOO == goodbye) then
echo see ya
endif


Source that script, and the output is:
hi


Various typos can have effects on this script that range from annoying to silent but deadly. Here's an annoying one: put the second if on a different line from the else. The output is:
hi
else: endif not found.


Well, at least you got an error message telling you something is up, although requiring two keywords to be on the same line is a level of stupidity worthy of Tcl. What if you do the opposite, and have too many endifs? Well, the output is just "hi", with no error message. I suppose that doesn't bite too much, until later when you nest another if into the script, and it suddenly gets closed by one of your stealth extra endifs.

But here's a bad one. Take the original script. Add an else at the beginning, and an endif at the end. Seems like an else without an accompanying if should get a syntax error, but it doesn't. The extra endif -- which might have been there because we always had an extra one, but never got an error about it -- closes the weird else. And guess what? The script runs with no errors, and no output. The else is kind of a "if not true". How can this be a good thing? 

[Update 2013/11/21: Ha! Now this has actually happened to me, in almost the same way described here.  I had to add some functionality to an existing script at work.  It was a branch added near the top of an if with several branches. I mistakenly added an endif at the end of my new branch. The new functionality (and the branch above it) tested fine, no error reported, so I checked it in. A few days later someone noticed that the later branches weren't running (this script gets launched from a cron job, so the problem wasn't something that would get immediately noticed). The later branches -- starting with an else for which there was no active if -- were silently skipped. Grrrr...]

I did these experiments with csh and tcsh on RedHat. Avoid csh. Long live bash!

Monday, September 21, 2009

Makedepend: Error: Failed to Read ...

I complained a few months ago about makedepend not picking up dependencies, and gave a recipe for avoiding makedepend.

I'm stuck with it for a project, and a couple of times I've gotten a makedepend failure, with only this useless warning:


makedepend: error: failed to read [some directory name]


The problem ended up being an #include somewhere, where the opening double-quote (") was missing. One time I accidentally had a single-quote at the beginning, closed with a double-quote; the other time a co-worker left off the opening quote, but did have the closing quote.

No help from makedepend on which file has the faulty include: it just gives up on the whole directory. Thanks, makedepend. Notice that if you leave off both quotes, or have both as single-quotes, makedepend is kind enough to give you a meaningful message. Or, if you open with a double-quote, and forget the closing quote, it exits without an error! Though I'm not sure if it gets the dependencies right in that case.

Do yourself a favor, use gcc's dependency-generating scheme, and never use makedepend again.

Wednesday, July 15, 2009

Perl Cwd Performance

The Camel book says -- in the Efficiency section of Chapter 24 -- to use the Cwd module instead of calling pwd repeatedly.

Boy, it's not kidding. I wrote a test script that does almost nothing but change directories and print the current directory, repeating that about 7500 times. Using `pwd` to get the directory took an average wall-clock time of 11.7 seconds. If instead you import Cwd's version of chdir(), the script can use $ENV{PWD}, which took an average wall-clock time of 0.3 seconds.

Oddly, using Cwd::cwd() took an average of 15.4 seconds. This wasn't a very scientific test, but those numbers bore up under repeated tries (doing several runs in a row, throwing out the time for the first run). I used wall-clock time, because the sys/user time reported by Unix time was quite a bit lower, even though it was an unloaded system. Maybe because the directories were NFS mounts.

So here's a good strategy for using Cwd:

use Cwd qw(chdir);
sub cwd {
return $ENV{PWD};
}


Note: the Cwd documentation points out that $ENV{PWD} is only kept up to date if every module used in the script uses Cwd::chdir to change directories.

Monday, July 6, 2009

gcc Warning Options

A few years ago I was in a job interview, and the development manager was gloating because he had squashed all compiler warnings in the code, and then added the gcc -Werror switch to the build, so that warnings now caused the build to fail. At the time, I thought "That's annoying overkill". My feeling was that compiler warnings didn't live very long -- after someone saw one a few times, they would fix it.

But I've changed my mind. Where I work, the team had even been doing without -Wall for many years. They had been using -pedantic. In other words, the warnings were all noise, and no signal, since the pedantic warnings don't tell you about potential bugs in your code, just about adherence to language standards. Moreover, they were completely happy to ignore all compiler warnings -- even those for things so dangerous that gcc warns about them without any warning switches, like 64-bit size issues. I now understand why someone would insist on -Werror.

Really, the best set of warnings for g++ is -Wall -Wextra -Werror. [If you have an older gcc, like the 3.2.3 that's stock on RedHat 3, use -W instead of -Wextra.] When we switched these on recently, there were thousands of unique warnings. Many of them were harmless in the end of the day, but there were dozens that indicated serious code bugs and/or crashes waiting to happen.

Of course, you can disable some of the -Wextra warnings. On our RedHat 3 build, we find it convenient to add these disabling flags:
  • -Wno-parentheses: too picky
  • -Wno-unused-parameter: provoked by <fstream>
  • -Wno-deprecated: provoked by <strstream>

In the best possible world, we would use <sstream> and wouldn't need -Wno-deprecated. But that's a battle for another day. For today, I'm just very happy that we have real warnings turned on, and that they get attention due to -Werror.

Monday, May 18, 2009

Emacs Registers

A few weeks ago when I was talking about tricks to get back to your editing location in emacs, I neglected to mention the jump-to-register command, by default bound to C-x r j. Actually, the first step is to save a location to a register: C-x r SPC (point-to-register). You'll be prompted to enter a character to refer to the location.

The nice thing is, when you jump-to-register, you don't have to be in the same buffer or file as the saved location. So you can mark places that are interesting, go explore some other stuff, and then immediately leap back to the buffer and line that you need.

Do you use emacs registers much? There are a few things you can do with them, the most common being point-to-register/jump-to-register and copy-to-register/insert-register. Whereas the kill-ring is like a text clipboard vector, copy-to- and insert-register are like a clipboard associative array.

I find it useful to keep register identifiers (the single character that names a register) partitioned into namespaces:
  • punctuation marks: canned useful strings defined with set-register that my .emacs loads at startup
  • numbers: locations for jump-to-register
  • letters: temporary copy-to-register strings for ad hoc cut/paste operations
It was only a few weeks ago that I realized that control-keystrokes or meta-keystrokes can be register names. It probably makes more sense to let the letter namespace represent canned strings along with punctuation marks, and use control characters for the quotidian things that letters represent for me right now. I might gradually shift to that way of doing business.

Sunday, May 3, 2009

Handy C++ Visitor Macros

This is the fifth post on the subject of the Visitor design pattern. First I tried to explain the motivation for Visitor, then presented the textbook implementation, what I consider to be some improvements on that implementation, and finally an example of the power of the Default Visitor variant of the design pattern. In this post, I want to show some preprocessor macros that make Visitor even handier. Brace yourself, this is going to be a long post. There's a lot to discuss.

The C preprocessor is a dangerous weapon when used indiscriminately. In the case of our C++ Visitors, however, some preprocessor magic will provide us with the following benefits:
  1. Compiler errors (not warnings) when a concrete Visitor class has not been defined correctly.
  2. Automatic generation of tedious parts of the Default Visitor implementation.
  3. Linker errors when Visitors which must not implement Default Visitor are not updated to visit a newly-added class.
We need to add the Statement hierarchy described in the last post to the Visitor class we declared earlier. Instead of hand-coding everything, we are going to write a header file that uses some preprocessor macros to describe the hierarchy of visited classes. If we also add a header file containing #include directives for each concrete visited class, then we won't have to edit Visitor.hpp or Visitor.cpp when we add classes to the hierarchy in the future. Nifty!

Let's call the first header file VisitorSpecs.hpp. It looks like this:

// NOTE: No include guard for this header!

#ifndef VISITED_ABSTRACT_CLASS
#define VISITED_ABSTRACT_CLASS(cls,parent)
#endif

#ifndef VISITED_ROOT_CLASS
#define VISITED_ROOT_CLASS(cls) \
VISITED_ABSTRACT_CLASS(cls,0)
#endif

VISITED_ROOT_CLASS(Expression)
VISITED_CLASS(Variable, Expression)
VISITED_CLASS(Number, Expression)
VISITED_CLASS(Operation, Expression)

VISITED_ROOT_CLASS(Statement)
VISITED_CLASS(Assignment, Statement)
VISITED_ABSTRACT_CLASS(MultiStatement, Statement)
VISITED_CLASS(Block, MultiStatement)
VISITED_CLASS(IfElse, MultiStatement)

#undef VISITED_ROOT_CLASS
#undef VISITED_ABSTRACT_CLASS
#undef VISITED_CLASS

Hang on a second... This defines a couple of macros as no-ops, and leaves another macro undefined. What gives? The answer is that Visitor.hpp defines the macros one way before including VisitorSpecs.hpp, and Visitor.cpp defines the macros another way, and also includes VisitorSpecs.hpp. That's why the header must not have an include guard: it will get included into Visitor.cpp multiple times. You may even find other consumers for VisitorSpecs.hpp, as you'll see a little later.

The macros are going to end up a little bit complicated, to provide some of the wonderful compiler fail-safes I've been hyping. But to make things clear, let's present the basic implementation first. Visitor.hpp has to include VisitorSpecs.hpp two separate times. First to declare the class names:

#define VISITED_CLASS(cls,parent) class cls;
#define VISITED_ABSTRACT_CLASS(cls, parent) \
VISITED_CLASS(cls,parent)
#include "VisitorSpecs.hpp"


Then to declare the member functions, Visitor.hpp defines the macros as follows:

#define VISITED_CLASS(cls,parent) \
virtual bool enter(const cls &); \
virtual void exit(const cls &);
#define VISITED_ABSTRACT_CLASS(cls, parent) \
VISITED_CLASS(cls,parent)
#include "VisitorSpecs.hpp"

Remember, the "root" classes (those with no parent) are defined in terms of abstract classes. So in fact, Visitor.hpp declares enter() and exit() for every visited class.

Visitor.cpp defines default implementations for the member functions by defining the macros like this:

// No parent default: don't recurse, do nothing.
#define VISITED_ROOT_CLASS(cls) \
bool Visitor::enter(const cls &) \
{ \
VISITOR_CATCH_MISSING_HEADER(cls); \
return false; \
} \
void Visitor::exit(const cls &) \
{ \
}

// Child classes: do what the parent would do.
#define VISITED_CLASS(cls,parent) \
bool Visitor::enter(const cls &c) \
{ \
VISITOR_CATCH_MISSING_HEADER(cls); \
const parent &p = c; \
return enter(p); \
} \
void Visitor::exit(const cls &c) \
{ \
const parent &p = c; \
exit(p); \
}

// Abstract classes: same as for concrete classes.
#define VISITED_ABSTRACT_CLASS(cls, parent) \
VISITED_CLASS(cls,parent)

#include "VisitorSpecs.hpp"

The macro definitions for Visitor.cpp implement the Default Visitor scheme: if a concrete Visitor doesn't define enter() for a given type, the default is picked up from the base class, and looks in the virtual table for the enter() belonging to the parent class. We have Visitor.cpp include the header file VisitedClasses.hpp, which includes the .hpp for every concrete visited class. Thus, when a new class is added to the hierarchy, Visitor.hpp and Visitor.cpp do not require any change whatsoever.

Notice the lines in the function definitions that say VISITOR_CATCH_MISSING_HEADER(cls). The reason for these lines, is that if you add a new class to the hierarchy, but forget to add an include to VisitedClasses.hpp, most compilers give you a confusing error message about initializing the variable p. On the versions of g++ and Solaris CC I've looked at, the message says nothing about an undefined class, let alone a missing header file. So, in Visitor.cpp we define a macro which will give you an error message that points in the direction of the problem, if you have forgotten that include. The macro is:

#ifndef NDEBUG
#define VISITOR_CATCH_MISSING_HEADER(cls) \
typedef cls INCLUDE_ ## cls ## \
_hpp_IN_VisitedClasses_hpp; \
struct Dummy : \
public INCLUDE_ ## cls ## \
_hpp_IN_VisitedClasses_hpp { \
Dummy(); \
}
#else
#define VISITOR_CATCH_MISSING_HEADER(cls) /* nothing */
#endif

We make the macro a no-op for non-debug builds, though it shouldn't really affect the code even if you enable it for release builds. Compare the messages in g++ 4.1.2. Suppose you omitted #include "IfElse.hpp" from VisitedClasses.hpp. Without the macro, the message says VisitorSpecs.hpp:38: error: invalid initialization of reference of type 'const MultiStatement&' from expression of type 'const IfElse'; with the macro, you still get that message, but before it you get: VisitorSpecs.hpp:38: error: invalid use of undefined type 'struct Visitor::enter(const IfElse&)::INCLUDE_IfElse_hpp_IN_VisitedClasses_hpp'. A strange message, but it gets the point across better.

Speaking of compiler errors, we will beef up the macro definitions in Visitor.hpp to catch a common coding error at compile time instead of having to debug strange behavior at runtime. The issue is this: since the enter()/exit() functions in our Visitor are not pure virtual, if you mistakenly add a concrete Visitor with one or more incorrect prototypes for those functions, the code will compile, link, and run -- silently skipping the functions with the incorrect prototypes, because the abstract base Visitor implementations will be found in the virtual table.

For example, if your concrete Visitor declared bool enter(const MultiStatement *) -- with a pointer argument instead of a reference -- then you wouldn't know anything was wrong until you ran the program. In the worst case, you would not even notice that whatever behavior was supposed to happen in that MultiStatement enter() didn't happen; even if you did notice it, you would have to step through in the debugger to watch the visit go into the wrong function. To preclude some of the more common silent-but-deadly typos, we bulk up the Visitor.hpp macros like so:

// The "real" prototypes.
#define VISITED_CLASS_ENTER_EXIT(cls,parent) \
virtual bool enter(const cls &); \
virtual void exit(const cls &);

// Fakes that turn typos into compile errors.
#define FORBID_PTR_ARG(cls,parent) \
virtual _USE_REFERENCE_ARG *enter(const cls *) \
{ return 0; } \
virtual _USE_REFERENCE_ARG *exit(const cls *) \
{ return 0; }
#define FORBID_NON_CONST_ARG(cls,parent) \
virtual _USE_CONST_REF *enter(cls &) \
{ return 0; } \
virtual _USE_CONST_REF *exit(cls &) \
{ return 0; }
#define FORBID_CONST_FUNC(cls,parent) \
virtual _USE_NON_CONST_FUNC *enter(const cls &) \
const { return 0; } \
virtual _USE_NON_CONST_FUNC *exit(const cls &) \
const { return 0; }

// Final macro definitions.
#ifdef NDEBUG
#define VISITED_CLASS(cls,parent) \
VISITED_CLASS_ENTER_EXIT(cls,parent)
#else
class _USE_REFERENCE_ARG;
class _USE_CONST_REF;
class _USE_NON_CONST_FUNC;
#define VISITED_CLASS(cls,parent) \
VISITED_CLASS_ENTER_EXIT(cls,parent) \
FORBID_PTR_ARG(cls,parent) \
FORBID_NON_CONST_ARG(cls,parent) \
FORBID_CONST_FUNC(cls,parent)
#endif

#define VISITED_ABSTRACT_CLASS(cls, parent) \
VISITED_CLASS(cls,parent)

#include "VisitorSpecs.hpp"

Without the FORBID* macros, the incorrect pointer-arg prototype above would prevent our FindLhsVars visitor in our previous example from finding assigned-to variables in complex statements such as blocks or if-else statements. With the macros in place, we get the following compile-time error:
VisitorSpecs.hpp:36: error: overriding 'virtual Visitor::_USE_REFERENCE_ARG* Visitor::enter(const MultiStatement*)'. That is money in the bank.

Are you still with me? Then there's one more feature of this setup that I'd like to point out. Notice that VISITED_CLASS and its related macros can be defined in any file that includes VisitorSpecs.hpp. As useful as Default Visitor is, there are some concrete Visitors that must be updated whenever a new class is added to the hierarchy -- for example, the pretty-printer Visitor from the earlier post. In that case, instead of hard-coding the enter() function declarations in the concrete Visitor class definition, do this:

#define VISITED_CLASS(cls,parent) \
virtual bool enter(const cls &);
#include "VisitorSpecs.hpp"

Now, if you forget to define the enter function for a newly-added class, you will get a link error when the linker tries to build the virtual table. From g++ 4.1.2: Print.cpp:61: undefined reference to `Print::enter(IfElse const&)'. Again, money in the bank. You might also find that there is some boilerplate code you would like to generate for every visited class -- a thought that comes to mind is an enum representing class types in the hierarchy. Simply extend the macro to include a third argument for that extra information -- if it can't be created with a '##' concatenation -- and include VisitorSpecs.hpp in the appropriate place.

Whew! This is a huge post, but when you apply these macro definitions to the improved Visitor in the earlier post, you have a very powerful idiom to work with. I can see a couple more posts coming up in this series: 1. a listing of the final Statement, Expression, and Visitor code presented here (I've been compiling it as I wrote these posts, to make sure it all worked), and an example main() to exercise it; and 2. discussion of a few Visitor issues I've glossed over to get this thing out.