Puppy's InterProcess Communication

Page updated June 29, 2013 
pup_event is the generic name that I have given to the underlying ...how to describe it... code that does everything "under the bonnet", in the background, that makes your pup run nice.

This page describes an implementation of InterProcess Communication in Puppy, that falls under the pup_event umbrella. So, this can be known as the "pup_event IPC" technique, to give it a name when referenced or discussed elsewhere.

The method chosen is server-less, that is, works peer-to-peer. Applications send and receive messages between themselves directly, without an intermediate server or daemon. However, some applications can provide information that other applications request, and I use the word server in reference to them. I also refer to applications as clients of the IPC mechanism.

Puppy's IPC mechanism offers various services to client applications:
  1. IPC by synchronous message swapping.
  2. IPC by mailbox (asynchronous) message passing.
  3. Block hotplug events from the kernel.
  4. ...more coming...
There is a CLI (commandline) utility supporting IPC in Puppy, named 'pup_event_ipc', that can be used in shell scripts. Documentation is provided below to show how binary (compiled) applications can use IPC by means of library calls.


pup_event_ipc

This is a CLI (commandline) application named 'pup_event_ipc'. It is designed to be used in shell scripts, acting as an intermediary to the pup_event IPC mechanism, enabling scripts to pass messages between each other. It is located at /sbin/pup_event_ipc.

Usage of pup_event_ipc is shown by examples in the rest of this document. You can also run it like this to see some usage information:
# pup_event_ipc --help
The source code is at /usr/local/pup_event/pup_event_ipc.bac in the latest Puppy, also reproduced below. It is also available online in the Woof Fossil VCS repository (also this document) after anonymous login, at: http://bkhome.org/fossil/woof2.cgi/index

The code can be examined to see how any other application can access the IPC mechanism.

The commandline format of pup_event_ipc is this:
# pup_event_ipc "request:client[:message]" [-t <n>]
-t <n>: Specifies timeout in milliseconds. Timeout will return with value=1.
request: Name of another application to send the message to*. Or,
request: Keyword to request information from the server.
                  Supported keywords:   block mailbox waitmail
client:  Any single-word to identify client (usually sender) application*.
message: Any text string (that other side understands), up to 4000 bytes.
*The names for clients need to be unique. Choose anything you want, perhaps prefixing all names with your initials, for example John Smith could prefix with "js_".


Message formats

This section describes message formats. This is already introduced above, as:
"request:client[:message]"
The various permutations of this are described below.

Synchronous message-swapping

Demonstrating synchronous message-swap between two different applications.
In one terminal window I type this:
# pup_event_ipc "js_app1:js_app2:some stuff from app2"
In another terminal window I type this:
# pup_event_ipc "js_app2:js_app1:data from app1"
After launching both of them:
First window:
# pup_event_ipc "js_app1:js_app2:some stuff from app2"
data from app1
Second window:
# pup_event_ipc "js_app2:js_app1:data from app1"
some stuff from app2
This is a very simple mechanism that can be use by two applications to perform a synchronization, or to pass messages between each other.

It is a simple swap, and both sides must post to the other. If only one side posts, the message will remain in the "mailbox" until the other side posts.
The utility 'pup_event_ipc' blocks, that is, waits, until the swap is completed, before exiting.

Note, the CLI utility 'pup_event_ipc' does have a timeout option (-t n), so either side can terminate before the swap has completed.

Mailbox (asynchronous) message passing

When a client posts a message to another client, it goes first into the IPC "mailbox", later it gets delivered to the other client.

With asynchronous message passing, you don't send the message to the other client, you send it to the mailbox only, then the sending-client is free to do other stuff.
The other client, to which you would like your message to go, can at any time query "is there a message waiting for me?" and if so, fetch it.

The format of messages is:
"mailbox:client[:message]"
If a client posts a message with this format, with the third-field message-string, it gets posted, and the server acknowledges. For example:
# pup_event_ipc "mailbox:js_app1:message string posted to app1"
Mailbox acknowledge
A client, any client, can ask if there is mail waiting for "js_app1". This is done by having no third field. For example:
# pup_event_ipc "mailbox:js_app1"
message string posted to app1
However, if this is done again:
# pup_event_ipc "mailbox:js_app1"
Mailbox empty
That last example returns immediately, but if you want to wait until a message arrives at the mailbox, use this variant:
# pup_event_ipc "waitmail:js_app1"
The utility will not exit until there is a message delivered.

But of course, the pup_event_ipc utility does support a timeout:
# pup_event_ipc "waitmail:js_app1:" -t 4000
Synchronous message-swapping and asynchronous mailbox message-passing, are the two peer-to-peer messaging and synchronization mechanisms.
There is another group, that is a bit more like a server-client arrangement, in which the 'request' field of the message requests something from another application. That other application will be serving a certain kind of information. Although I don't really want to consider these as servers, as they are just equal clients in the IPC mailbox system, for the sake of putting a name on them, I am referring to this group of applications as "server applications".

Server applications are documented below. Currently only block server is offered.

Block hotplug events

A client application may query the "kernel-event server application" for various information, such as detecting plugin/removal of USB or card drives.

For now, only block is supported. For example:
# pup_event_ipc "block:js_app1"
add:sdc add:sdc1
The utility waits until a drive hotplug event occurs, such as a USB Flash stick plugged in, then returns with the information.

The 'pup-event_ipc' utility is only designed as a one-shot query, so exits after detecting the events.
However, hotplug events are not lost. The first "block" request has registered "js_app1" (from above example) and hotplug events will accumulate.

So, by having a loop, to receive and process the hotplug events, no events will be missed. Ex shell script:
while true; do
 EVENTS="`pup_event_ipc "block:js_app1"`"
 ...process...
done
What needs to be understood about the above, is how simple the implementation is. The client application just creates a file (if it does not exist) named "/tmp/pup_event_ipc/block_js_app1" and waits for the server-application to put something into it. This inotify IPC technique is described in the next section.


The inotify backend

The IPC technique used is more than a little unusual. The established techniques such as D-Bus, named pipes, message queues and Unix Domain Sockets, were investigated and rejected for one reason or another.

The inotify mechanism supported by the Linux kernel is not intended to be used for IPC, at least not to the extent that I have done. As it has turned out, however, it "ticks all the boxes" for what I want for Puppy -- it is very fast, low resources, extremely simple, and remarkably amenable to the particular uses that I wanted to put it to.

The References Section below has links that introduce inotify, read those if you need to understand the fundamentals.

There is a directory /tmp/pup_event_ipc, the "mailbox", that contains the files that have the messages to be transferred. When a file pertaining to a particular client changes, the client "wakes up" and handles it. Basically the steps are:
1
2
3
ClientA is waiting on fileA
ClientB writes to fileA ClientA wakes up and reads file

That is really all it is. From those three steps, a complete IPC system can be devised.

Of course, there all kinds of caveats and gotchas in this mechanism, such as guaranteeing a complete message is written to a file before it is read, and that files do not contain garbled messages if two or more clients write to them simultaneously.

The way that I have implemented it, does, I think, avoid these pitfalls.
Atomicity of read and write is important for this to work, and from reading online I got two different figures for this for Linux: 1KB or 4KB (kilobytes). I was optimistic and used the higher figure.

A communication is between two clients, using two files, for each direction of transfer. As long as they choose unique names, there is not going to be a clash with other transfers. There will not occur the problem of two different clients simultaneously writing to the same file.

Anyway, even if two or more clients were allowed to write to the same file "simultaneously", atomicity and the O_APPEND write mode (see below), will ensure that individual messages remain intact.

The CLI utility is the first implementation of inotify-based IPC for Puppy Linux, though, it is generic and will work for all Linux distributions. The easiest way I think to explain how it all "hangs together" is to show the code for this utility.

The utility 'pup_event_ipc' is written in BaCon (see: http://bkhome.org/bacon) with embedded C, so is a binary executable, small and fast. This is the code as at June 29, 2013:
REM (c) Copyright Barry Kauler, June 2013, bkhome.org
REM License GPL3 (refer: /usr/share/doc/legal)
REM InterProcess Communication for Puppy Linux, CLI utility using inotify.
REM 130628 first version.

DECLARE helpflg,milliseconds TYPE int

'parse the commandline...
helpflg=0
milliseconds=0
sendstr$=""
IF argc==1 THEN helpflg=1
FOR x=1 TO argc-1
arg$=argv[x]
IF arg$=="--help" THEN
helpflg=1
CONTINUE
ENDIF
IF arg$=="-h" THEN
helpflg=1
CONTINUE
ENDIF
IF arg$=="-t" THEN
INCR x
milliseconds=VAL(argv[x])
CONTINUE
END IF
sendstr$=arg$
NEXT

IF helpflg==1 THEN
PRINT "(c) Copyright Barry Kauler, June 2013, license GPL3 (/usr/share/doc/legal)"
PRINT "This is a CLI utility that may be used in scripts. Run like this:"
PRINT " pup_event_ipc \"request:client[:message]\""
PRINT " request: name of another application to send the message to*. Or,"
PRINT " request: keyword to request information. Supported keywords:"
PRINT " block mailbox waitmail"
PRINT " client: any single-word to identify client (usually sender) application*."
PRINT " message: any text string (that other side understands), up to 4000 bytes."
PRINT " any characters allowed, including \":\""
PRINT "ex: pup_event_ipc \"otherapp:myapp:some data for otherapp\""
PRINT "ex: pup_event_ipc \"mailbox:otherapp:some data for otherapp\""
PRINT "ex: pup_event_ipc \"mailbox:myapp\""
PRINT "ex: pup_event_ipc \"waitmail:myapp\""
PRINT "ex: pup_event_ipc \"block:myapp\""
PRINT "pup_event_ipc has an optional timeout (milliseconds),"
PRINT "ex: pup_event_ipc -t 1000 \"otherapp:myapp:data for you\""
PRINT "Will exit with non-zero value if an error:"
PRINT " 1=timeout. 2=wrong msg format. 3-7=message-passing failure. "
PRINT "*Any unique names that both clients agree on."
PRINT "PLEASE SEE /usr/local/pup_event/pup_event_ipc-README.htm for more details."
END 0
END IF

PROTO close, ftruncate, inotify_init, inotify_add_watch, inotify_rm_watch
PROTO lseek, open, poll, read, sizeof, strlen, write
DECLARE fd1,len1,leni,size_buf,thiswatchfile,otherwatchfile TYPE int
DECLARE arr1,thisapp,otherapp TYPE char*
DECLARE thisdescr,otherdescr TYPE int
size_char=sizeof(char)
size_inbuf=size_char*4096
DECLARE inbuf,bufinotify TYPE char ARRAY size_inbuf

SUB exitsub(NUMBER i)
IF watchdir>0 THEN inotify_rm_watch(fd1,watchdir)
IF fd1>0 THEN close(fd1)
IF otherdescr>0 THEN close(otherdescr)
IF thisdescr>0 THEN close(thisdescr)
END i
END SUB

'struct pollfd, defined in asm-generic/poll.h...
RECORD pfd
LOCAL fd TYPE int
LOCAL events TYPE short
LOCAL revents TYPE short
END RECORD
'linux/poll.h, only includes asm/poll.h, which only includes asm-generic/poll.h...
CONST POLLIN=1
pfd.events = POLLIN

SUB waitsub()
'there is very slight possibility of deadlocking, if reply between lseek & read/poll...
offt=lseek(thisdescr,0,SEEK_END)
IF offt==0 THEN
IF milliseconds==0 THEN
'this will block until the file is modified...
leni=read(fd1,bufinotify,size_inbuf)
ELSE
pfd.fd=fd1
eventstatus=poll(&pfd.fd,1,milliseconds)
IF eventstatus==0 THEN exitsub(1)
IF eventstatus<0 THEN exitsub(7)
ENDIF
ENDIF
offt=lseek(thisdescr,0,SEEK_SET)
numr=read(thisdescr,inbuf,size_inbuf)
ft=ftruncate(thisdescr,0)
IF numr>1 THEN
'get rid of carriage-return char...
inbuf[numr-1]=0
inbuf$=inbuf
PRINT inbuf$
ENDIF
END SUB

'parse the message...
field1$=""
field2$=""
field3$=""
field4$=""
message$=""
arr1=sendstr$
off1=INSTR(arr1,":")
IF off1=0 THEN exitsub(2)
arr1[off1-1]=0
field1$=arr1
off2=INSTR(arr1+off1,":")
arr1[off1+off2-1]=0
field2$=arr1+off1
IF off2>0 THEN
off3=INSTR(arr1+off1+off2,":")
'the message-part can have ":" chars in it...
message$=arr1+off1+off2
arr1[off1+off2+off3-1]=0
field3$=arr1+off1+off2
IF off3>0 THEN
off4=INSTR(arr1+off1+off2+off3,":")
arr1[off1+off2+off3+off4-1]=0
field4$=arr1+off1+off2+off3
ENDIF
ENDIF
'need to extract names of this-app and other-app...
otherapp=0
thisapp=0
thiscreateflag=1
SELECT field1$
CASE "mailbox"
IF message$!="" THEN
otherapp=field2$
ELSE
thisapp=field2$
ENDIF
CASE "waitmail"
thisapp=field2$
CASE "block"
thisapp$=CONCAT$("block_",field2$)
thisapp=thisapp$
'thisfile$ is not created here, it is created by daemon that monitors kernel uevents...
thiscreateflag=0
DEFAULT
otherapp=field1$
thisapp=field2$
END SELECT

'create the files...
otherdescr=0
thisdescr=0
watchdir=0
IF FILEEXISTS("/tmp/pup_event_ipc")==0 THEN MAKEDIR "/tmp/pup_event_ipc"
IF otherapp!=0 THEN
otherfile$=CONCAT$("/tmp/pup_event_ipc/",otherapp)
USEC
/*defined in /usr/include/asm-generic/fcntl.h & i386-linux-gnu/bits/fcntl-linux.h...*/
#define O_WRONLY 00000001
#define O_CREAT 00000100
# define O_SYNC 04010000
#define O_FSYNC O_SYNC
#define O_APPEND 00002000
otherdescr=open(otherfile$, O_WRONLY | O_CREAT | O_APPEND);
END USEC
IF otherdescr<=0 THEN exitsub(3)
ENDIF
IF thisapp!=0 THEN
closedflag=0
thisfile$=CONCAT$("/tmp/pup_event_ipc/",thisapp)
IF thiscreateflag=1 THEN
USEC
#define O_RDONLY 00000000
#define O_CREAT 00000100
#define O_RDWR 00000002
thisdescr=open(thisfile$,O_RDWR | O_CREAT);
END USEC
ELSE
IF FILEEXISTS(thisfile$)==1 THEN
USEC
#define O_RDONLY 00000000
#define O_RDWR 00000002
thisdescr=open(thisfile$,O_RDWR);
END USEC
ELSE
closedflag=1
ENDIF
ENDIF
IF closedflag==0 THEN
IF thisdescr<=0 THEN exitsub(4)
fd1=inotify_init()
IF fd1<=0 THEN exitsub(5)
CONST IN_MODIFY=2
watchdir=inotify_add_watch(fd1,thisfile$,IN_MODIFY)
IF watchdir<=0 THEN exitsub(6)
ENDIF
ENDIF

'now decide what to do...
SELECT field1$
CASE "mailbox"
IF otherapp!=0 THEN
'need to post message to otherapp... format: "mailbox:otherapp:message"
outstr$=CONCAT$(message$,"\n")
wr=write(otherdescr,outstr$,strlen(outstr$))
PRINT "Mailbox acknowledge"
ELSE
'need to check if waiting mail for thisapp...
offt=lseek(thisdescr,0,SEEK_END)
IF offt==0 THEN
PRINT "Mailbox empty"
ELSE
offt=lseek(thisdescr,0,SEEK_SET)
numr=read(thisdescr,inbuf,size_inbuf)
ft=ftruncate(thisdescr,0)
IF numr>1 THEN
'get rid of carriage-return char...
inbuf[numr-1]=0
inbuf$=inbuf
PRINT inbuf$
ENDIF
ENDIF
ENDIF
CASE "waitmail"
'need to wait for mail for thisapp... format: "waitmail:thisapp"
waitsub()
CASE "block"
'need to wait for block-drive hotplug event, for thisapp... format: "block:thisapp"
IF FILEEXISTS(thisfile$)==0 THEN
PRINT "Not implemented"
ELSE
waitsub()
ENDIF
DEFAULT
'post msg to otherapp, wait for reply to thisapp... format: "otherapp:thisapp:message"
outstr$=CONCAT$(message$,"\n")
wr=write(otherdescr,outstr$,strlen(outstr$))
waitsub()
END SELECT

exitsub(0)

This is a remarkably small program to implement a complete IPC system. Essentially, it does this:
  1. Parses the message that has been passed on the commandline.
  2. Creates one or two files in /tmp/pup_event_ipc, and opens them.
  3. Based on the passed message, data will be written to one file, or data read from the other.
  4. Reading may or may not block, depending on the message.
When a file is opened for writing, it is done with the 'O_APPEND' mode, meaning that writes to it always append to the end of the file, totally ignoring the file-pointer.
Messages are line-based, that is, one message per line. This allows messages to accumulate, while waiting to be read -- hence, most important, no messages get lost.

Take the message "block:myapp" as an example. The name "myapp" is an arbitrary choice, anything unique is suitable. If this is posted by the CLI utility:
# pup_event_ipc "block:myapp"
If you were to trace through the above code, you would see that if file "/tmp/pup_event_ipc/block_myapp" is opened (or created if it does not exist), then there is a block on waiting for something to be written to the file.
When something is written to the file, it is read, and the file zeroised. The read contents are printed on stdout.

The above scenario means that some other application must write to /tmp/pup_event_ipc/block_myapp.
In the very latest builds of Puppy, we do have a an application that detects drive hotplug activity, a daemon named pup_event_frontend_d, that performs this information server function.

The function that such a server would have to perform can be show by a shell script:
for ONEFILE in `ls /tmp/pup_event_ipc/block_*`
do
 echo $BLOCKINFO >> $ONEFILE
done
That's it, done. All clients that are waiting on block-drive hotplug notification, will "wake up" and read the data. As mentioned before, it does not matter if a client is in a loop, doing other stuff:
while true; do
 EVENTS="`pup_event_ipc "block:myapp"`"
 ...other stuff...
done
While executing "other stuff", any new block-drive events will append to /tmp/pup_event_ipc/block_myapp (that is, queue )until it is next read.

Note: For the above example, the code in pup_event_ipc does not automatically create /tmp/pup_event_ipc/block_myapp. This is for usage in older puppies and other distros. In latest experimental Puppy builds however, the line "thiscreateflag=0" is commented out, we have the pup_event_frontend_d daemon that feeds kernel block-drive events (as described above).
pup_event_frontend_d is described further in the next section.

There is a lot that can be said about the flexibility of this inotify-based IPC mechanism. Just one example: although the pup_event_ipc utility does have a timeout option, you can cause a timeout whenever you want -- say, for example that a shell script has executed "pup_event_ipc 'block:myapp'" and is blocked, waiting on a modification to file /tmp/pup_event_ipc/block_myapp. Well, if you just execute "touch /tmp/pup_event_ipc/block_myapp", the read() function that is waiting will return, with zero data of course -- but voila, a simple way to cause a timeout.


Example code blocks

I will add to this section. For now, I have included code to show how the "server" can be implemented to serve block-drive event notifications.

Block-drive event notification

It has been explained above, that a client application may request block-drive event notification by posting a message of format "block:myapp" to the IPC mailbox.

Puppy Linux has a daemon, named pup_event_frontend_d, at /usr/local/pup_event (as well as the BaCon source file, pup_event_frontend_d.bac, and there is a wrapper script at /sbin). This daemon is started from /root/.xinitrc when X starts up. It does a lot of background stuff, such as manage the desktop drive icons. For that latter reason, it does have information about drive hotplug events.

It was an easy matter to add a bit of extra code to pup_even_frontend_d, to post this information to clients that want it, via the IPC mechanism. Example shell script code is shown in the above section, how this can be done.

The actual code inserted into pup_event_frontend_d is shown below. Basically, it looks for any files named /tmp/pup_event_ipc/block_*, then writes the drive hotplug information to them (string devevents$):
PROTO opendir, readdir
DECLARE dir1 TYPE DIR*
DECLARE ent1 TYPE struct dirent*

dir1=opendir("/tmp/pup_event_ipc")
IF dir1!=NULL THEN
WHILE TRUE DO
ent1=readdir(dir1)
IF ent1==NULL THEN BREAK
off1=INSTR((*ent1).d_name,"block_")
IF off1!=0 THEN
clientfile$=CONCAT$("/tmp/pup_event_ipc/",(*ent1).d_name)
outmsg$=CONCAT$(devevents$,"\n")
USEC
#define O_WRONLY 00000001
#define O_APPEND 00002000
int clientdescr;
clientdescr=open(clientfile$, O_WRONLY | O_APPEND);
if (clientdescr>0) {
int wr=write(clientdescr,outmsg$,strlen(outmsg$));
close(clientdescr);
}
END USEC
ENDIF
WEND

ENDIF

Note, BaCon is just a translator to C, which is then compiled using the Gnu C compiler. It is really just a thin translation layer, so many C-isms can be put into the BaCon code, and pure C can be embedded. I use BaCon as I prefer the syntax, plus it has many higher-level features, such as better string handling, and associative arrays. It is easy to read the above and translate to all-C if required.

...more coming...



Feedback welcome

I will be posting about progress on pup_event to my blog, http://bkhome.org/blog2/ 

Feedback will be welcome. If you can think of anything that might compromise the integrity of data transfer with this technique, or code modifications that will add improvements (while still keeping everything simple, including simple and small code).

Regards,
Barry Kauler


References

Reference on libc functions:
http://www.gnu.org/software/libc/manual/html_node/Function-Index.html
Wikipedia:
http://en.wikipedia.org/wiki/Inotify 
Intro to inotify:
http://www.linuxjournal.com/article/8478 
Monitor file system activity with inotify:
http://www.ibm.com/developerworks/library/l-ubuntu-inotify/ 
Monitor Linux File System Events with Inotify:
http://www.developertutorials.com/tutorials/linux/monitor-linux-inotify-050531-1133/ 
The inotify API:
http://linux.die.net/man/7/inotify 


(c) Copyright Barry Kauler, 2013. All reproduction rights of this page are reserved.