Monday, February 04, 2008

File system change notification on Linux

So I mentioned a few days ago that I had a file system change monitor API going on Windows. Now I've implemented a Linux backend and will be looking at Mac OS X next.

The API is straightforward. First, you create a monitor, passing it a directory and whether the monitor should recurse to subdirectories or not -- non-recursive monitors are supported on Windows, but on Linux all monitors are recursive;
"/home/slava/factor/" t <monitor>

Monitors must be disposed by using dispose or with-disposal since they wrap operating system resources. They function essentially as streams; you read from a monitor, which blocks until some file in the directory has changed. The file's path name is returned along with an array of descriptors indicating what has changed:
dup next-change . .

A sample piece of output might be:
{ +modified-file+ }
"/home/slava/factor/test.txt"

That is all! If you need to continuously monitor a directory for changes, all you have to do is spawn a new thread with in-thread, then call next-change in a loop and take appropriate action.

On Linux and Windows, we use asynchronous notifications behind the hood, so the block API presented to the user is actually implemented with continuations and a low-level multiplexer behind the scenes, just like all non-blocking I/O in Factor.

I'm going to use the io.monitor API to speed up the vocabulary browser and refresh-all. Right now they have to scan the whole Factor source tree but instead they will monitor it for changes. Since not all OSes support change notification, and not all file systems on supported OSes support it either, there will be graceful fallbacks, of course.

I'm not aware of any other language which attempts to implement advanced I/O functionality in a cross-platform manner. Java's process launcher is limited compared to io.launcher, and it doesn't support a cross-platform file system notification API at all; Python, Ruby and Perl implement advanced features but present OS-specific APIs to the programmer. Furthermore Factor attempts to integrate as much as possible with the I/O event loop, and of course, everything is consistently documented.

6 comments:

Anonymous said...

have a look at http://greenearthcommons.org/rian/gfslogger/ for OS X. the binary does not work on leopard, you need to compile it yourself.

David R. MacIver said...

The upcoming NIO.2 API for Java 7 ( http://jcp.org/en/jsr/detail?id=203 ) seems to intend to support something like this.

Anonymous said...

Not to undermine your efforts, but if you are using .NET to develop for Windows, that kinda API sounds kinda like wasted effort and reinvented wheels.

Full details on the .NET-class implementing this:
http://www.c-sharpcorner.com/UploadFile/mokhtarb2005/FSWatcherMB12052005063103AM/FSWatcherMB.aspx

Anonymous said...

libevent does callback based i/o polling using the host OS's best interface: epoll, kqueue, /dev/poll, poll, or select as a last resort.


For file system monitoring, there's also SGI's libfam, which is cross-platform across various unixes. Don't know if it works on Windows.

Slava Pestov said...

joestein: Factor's io.monitor library is just a cross-platform wrapper around the Windows and Linux APIs, so there's no wheel reinvention here.

Slava Pestov said...

Anonymous: last I checked, libevent didn't support Windows, also the code was kind of ugly. Since I'd have to make extensive changes to it to make it usable, I'd rather just write the whole thing in Factor and save time.