I'm working on setting up a stress test for our perl-based web spider. I recently implemented some support for things like tie-ing the spider's history database to a DBM file, and I want to see how well it behaves when it has to deal with sites with lots of pages.
So, I wrote a short little perl script to recursively generate a directory tree of arbitrary depth, and with an arbitrary number of files per leaf directory. It's easy to generate millions of pages this way, and I had the spider spit out around 2 million pages worth.
Then, I had to set up permissions on them to allow the IIS process to read them.
Ever try to set permissions recursively on 2 million files in windows explorer? Be prepared to wait awhile... like, a long while. And this is on a relatively fast pentium-4, on a file system that's striped over two IDE drives. Not the fastest possible configuration, but it shouldn't be all that pokey, either. But it sure was. Until I fired up the command prompt. Bypassing the windows explorer shell sped things up quite a bit.
So, I'm learning the joys of the windows command-line interface. Sort of an oxymoron, that... I know how to do everything I want to do on solaris or on mac os/x, but not so on windows. Some things are easy, like rmdir, which is the same as on unix (other than the command-line arguments). rm is del, so that's not too hard either. However, there are some inconsistencies. Some of them are just different arguments for the same idea, some of them come from the fact that the NTFS file system has much richer semantics for defining access control lists than the unix filesystem does.
Instead of saying "rm -rf foo", you might try to say "del /Q /S /F foo". If foo is a directory tree, this leaves you with the full tree of directories, but with no files in them. You then have to say "rmdir /Q /S foo" to actually blow away the directory tree itself. Saying "rmdir /Q /S foo" at the beginning would have done what you wanted - blow away the whole tree, lock stock and two smoking barrels.
For copying a tree, instead of saying "cp -r foo bar", you have to say "xcopy /Q /E /O foo bar". The /Q means quiet, the /E means the same as -r, and the /O means to copy the NTFS Access Control List and ownership info as well.
Lastly, the NTFS ACLs themselves. There's a command-line tool titled 'cacls' that is supposed to let you edit ACLs, but it's less comprehensive than the Security tab in Windows Explorer. In particular, it doesn't seem to talk at all about inherited permissions, which is what I want to use. Still have more research to do on this one...
