Installing, Running and Maintaining Large Linux Clusters

Found via SlashDork is this piece on buiding up Linux clusters to more than 1000 nodes... experience confronting some of the LHC scale computing challenges: scalability, automation, hardware diversity, security, and rolling OS upgrades. Looks like a must read (must try to understand!). 1K nodes would be a good start in SPTRTW