Arieh's Weblog

     
 
Monitoring large numbers of files, /dev/poll and others ...

I am in the process of investigating the ability to monitor large numbers of files in a system.

A lot of material has been written about this topic here (Kegel), here (Erenkranz), as well as BSD's Kqueue.

Most of the discussion on those centers on monitoring sockets, pipes, streams, etc.

In that context, there is an open source project File Alteration Monitor that tries to deal with the topic of monitoring files and directories.

I begun my investigation with using the /dev/poll device.

In general, the idea is to open the files being monitored and register their file descriptors with the /dev/poll device. What I wanted is that if any of those files are written to, that the DP_POLL will 'fire' and I will get the event that the file has indeed been written to.

I have not been able to observe the expected behavior. Instead, it seems as 'regular' files operate with poll(7d) in a manner similar to what poll(2) state, namely, that regular files trigger for read, write and error always, therefore not allowing to trigger on writes to the file, for example.

Having read Bart Smaalders blog article on event ports I decided to try using that new feature of Solaris 10.

In my program, I proceeded to open the files I wanted to monitor, get the file descriptors and associate them with the port that I created. I was unable to associate the 'regular' file with the port, with the library call returning a EBADFD for the file descriptor.

I am wondering if I am doing something wrong (probably) or that what I want to do is not possible with the /dev/poll device or the event ports subsystem.

Later at night, from home, I tried to use the kqueue functionality available on BSD systems. I was successful in doing so, getting events on file appends, truncates, etc.

I liked the flexibility that kqueue provides and wonder whether there is similar richness of functionality on event ports or /dev/poll.

Posted by arieh @ 09:46 AM MST [ Comments [2] ]
 
 
 
 
Comments:

I did some investigatin of this myself a while back (on BSD). I've always wanted to write something relatively small that monitors a large number (thousands) of files and initiates meta-data (re)indexing operations on file changes.

Out of curiosity, what's your definition of a "large number of files"? I think kqueues can be problematic with really large number of files because the OS runs out of inodes. Not sure though. I also know there's a patch floating around to make FAM use kqueues where it's supported (it uses polling on the kqueue-capable BSDs now). I hope it will be incorporated into a future FAM release.

<em> Best regards,
Aron Stansvik </em>

Posted by Aron Stansvik on November 14, 2004 at 08:32 AM MST #

Sorry, I mean file handles of course, not inodes.

<em>Aron</em>

Posted by Aron Stansvik on November 14, 2004 at 08:41 AM MST #

Post a Comment:

Comments are closed for this entry.
 
« November 2009
SunMonTueWedThuFriSat
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
     
       
Today

[RSS Newsfeed]

Valid XHTML or CSS?

[This is a Roller site]
Theme by Rowell Sotto.
 
© Arieh's Weblog