From AlphaWiki

Pipes: Pipes and shell like interaction

<JK 20/09/2003>

This page is meant as a primer on using pipes to connect Alpha with unix programmes. It is part of a more general discussion whose main page is AlphaAndUnix? where there are links to related pages.

It should be noted that the below functionality can also be implemented using [Sockets]. FrŽdŽric Boulanger's general purpose interactive shell package 'InSh' in the standard AlphaTcl distribution is based on sockets. There are pros and cons of both pipes and sockets: the socket implementation seems to be more general and powerful; the pipes implementation tends to be simpler. Some programmes run in one implementation and not in the other --- and nobody on Alpha-D seems to be able to explain why... On the page [Sockets] I have gathered some interesting snippets from the developer mailing list.


Pipes

Pipes are quite easy to uderstand. In principle all you need to do to communicate with programme foo is

 
     set fooPipe [open "|foo" RDWR]

Now you can write to the process foo and read its output simply with

 
     puts $fooPipe "do this"
     read $fooPipe

But in practice, you aren't going to do all this in Alpha's shell --- you'll like a special window (say "*Foo console*") where everything takes place, and reading and writing should happen automatically. The way to arrange this is by setting up an event handler. It is a procedure whose purpose is to treat the output as soon as it becomes available. A minimal pipe example could look like this:

 
    # This shell-like pipe depends on three global variables: one public
    # variable which the user will set:
    #       foo:           the name of the external executable
    # And two private globals, used internally between the procs:
    #       _fooPipe:       the name of the pipe
    #       _fooPromptPos   the position delimiting 'old' and 'new'

    # Event handler for _fooPipe
    proc _receiveAndDisplay { } {
	global _fooPipe _fooPromptPos
	if { [eof $_fooPipe] } {
	    # There is nothing more to read --- just stop:
	    stopFoo
	    return
	}
	set res [read $_fooPipe]
	# Insert the result in the window:
	if { [string length $res] } {
	    insertText -w "*Foo console*" "\r$res"
	    set _fooPromptPos [getPos]
	}
    }

    # Starter: open the pipe, create a console to write in, and set 
    # up the event handling mechanism:
    proc startFoo { } {
	global foo _fooPipe _fooPromptPos
	# Open the pipe:
	set _fooPipe [open "|$foo" RDWR]
	# Make sure the output comes line by line in the correct way:
	fconfigure $_fooPipe -buffering line -blocking 0
	# Create a console to write in:
	new -g 300 300 503 243 -n "*Foo console*" -fontsize 9 -shell 1
	# and initialise the prompt position:
	set _fooPromptPos [minPos]
	# Set up a handler for the output:
	fileevent $_fooPipe readable _receiveAndDisplay
    }

    # Stopper:
    proc stopFoo { } {
	global _fooPipe
	# Close the pipe:
	close $_fooPipe
	# And close the console window:
	killWindow -w "*Foo console*"
    }

    # Finally, we would like to have some sort of mechanism such that
    # pressing enter will send the last command to evaluation by foo. 
    # Here we take a very simplistic approach: the last command is taken
    # to be the text situated between the previous result (the prompt
    # position) and the cursor position:
    proc sendToFoo { } {
	global _fooPipe _fooPromptPos
	set cmd [getText $_fooPromptPos [getPos]]
	puts $_fooPipe $cmd
    }

    ascii 0x03 sendToFoo ;# this is Enter
    # (Of course in reality we would rather use <return>, but setting that
    # up conveniently would involve some more trouble like defining a new
    # mode and make a key binding specific to this mode...)

To try it out, issue these commands in Alpha's shell:

 
     set foo sh
     startFoo

The example above works well for programmes like bash, tcsh, gp, maxima. For some reason it doesn't work for tclsh... For bash, tcsh, gp the usual prompt is lost --- I don't know how it happens. In fact, all error messages are lost too. For maxima, prompts and error messages are preserved.


Remarks

Lost prompts

Concerning the problem with missing prompts (e.g. with tcsh), FrŽdŽric says (AlphaD, 10/06/2003): ''The problem is that many interactive programs use 'istty' to know if they are used from a terminal (and a human being) or by another program (through a pipe or a socket). Even with the -i option which forces the interactive mode, a shell can detect that it is not connected to a terminal (through things like 'isatty' or 'fcntl' and therefore inhibit some behaviors that doesn't make sense outside of a terminal, or than can be implemented only with special terminal functionalities (like cursor movement, text highlighting and so on).

To make every shell work the way we want, we should build a terminal emulator in Alpha...''

'Lost errors'

<Vince:14/06/2003>: ''Just one FYI on this topic. There is a trick (used in appPaths.tcl) by which you can capture both stdout and stderr using 'open "|..."', through the use of 'cat', like this:

 
    proc app::setupInput {cmd win} {
	global tcl_platform
	if {$tcl_platform(platform) == "unix"} {
	    updateExecPath
	}
	global catSig
	app::getSig "Please find your 'cat' application" catSig
	insertText -w $win $cmd "\n"
	set pipe [open "| \"$catSig\"" r+]
	fconfigure $pipe -buffering none
	fileevent $pipe readable [list app::handleErrorInput $win $pipe 1]
	set output [open "|$cmd 2>@ $pipe" r]
	fileevent $output readable [list app::handleStdoutInput $win $output $pipe]
    }

I should also point out there continue to be ongoing discussions on comp.lang.tcl and tcl-core about enhancing 'exec' to provide the capabilities of BLT's 'bgexec' which allows very easy access to both streams.''


The example above also works with tex-like programmes: if somefile.tex is a latex file in the current directory, you can do

 
     set foo "latex somefile"
     startFoo

If there are errors in the source file, you can interact with latex with the usual commands e, x, q, \r --- all followed by <enter>, of course. (In order for the command e to work, you must have set the environment variable TEXEDIT, and have installed the package AlphaServer by FrŽdŽric Boulanger.)

Once you have pipes you also need bells and whistles

Of course the example is not complete --- there are dozens of things to fine tune. To mention a few:

sendToFoo is really stupid about what to send. It ought to do some more parsing and detect input regions also other places in the shell window, in the style of the familiar Alpha shell, where you can place the cursor in any previous input region and evaluate. It would also be good to have some sort of prompt, and a command history completion mechanism. All of these mechanisms already exists for Alpha's shell, so perhaps it would be an idea to generalise that to an abstract mechanism which would only need to have defined a couple of constant strings like prompt patterns and such...

There are also many possibilities for preprocessing the command before sending it, like error correction: for examplpe if you try to send "2x^x-x" to Maxima, our preprocessor might be so clever as to suggest "Do you mean \"2*x^2-x\" ? Y/N", and afterwards be able to interpret your answer Y/N... (The more you elaborate on these questions, the more you are approaching the concept of Pipes.Worksheets --- some sort of ''intelligent documents with live connections to external processes...'')

The management of the special purpose console window should of course be improved, like checking for pre-existing consoles, having a special mode used only in this console, to allow for special interpretation of keystrokes (e.g. set up a mechanism that prevents the user from backspacing through the prompt), a closeHook to close the pipe when the console is closed, etc.

Of course, in practice you will also want to upgrade to a more flexible model where you can specify the command foo as a parameter instead of in a global variable. Perhaps you want to be able to have several connections and consoles open at the same time. For this to work you need to use dynamic pipe names (instead of above where the pipe is always called _fooPipe). Then of course the various procs need to take the pipe name as an argument; this holds also for the sendToFoo proc, so now you get a problem with the key binding! --- it can only be defined after you know the name of the pipe! The solution might be to let the console name (as well as the dynamic pipe name?) be derived from the process id and then have a sendToFoo proc that can figure out the pipe from the window name...

Many of these details have been worked out by FrŽdŽric Boulanger in his package 'InSh' (although it uses sockets instead of pipes) which is an implementation of a multi-purpose interactive shell. One advantage of his implementation using sockets is that the prompts survive in many cases where it is lost in pipe implementations. But currently some programmes refuse to run via InSh (gp and maxima).

Retrieved from http://alphatcl.sourceforge.net/wiki/pmwiki.php/Pipes/PipesAndShellLikeInteraction
Page last modified on April 26, 2006, at 12:42 AM