Recent Changes - Search:

WikiDoc

Categories

Batch-like interaction via pipes

Pipes.BatchLikeInteractionViaPipes History

Hide minor edits - Show changes to markup

April 19, 2006, at 11:51 PM by Nick -
Changed line 42 from:
    $ maxima --batch-string="diff(sin(x),x);"
to:
    $ maxima --batch-string="diff(sin(x),x);"
Changed lines 90-91 from:
    % set res [maxima diff( $f+$g, x )]
    % maxima expand( $res )
to:
    % set res [maxima diff( $f+$g, x )]
    % maxima expand( $res )
Changed line 126 from:
  # % maxima factor( $f )
to:
  # % maxima factor( $f )
Changed line 158 from:
      fconfigure $data(pipe) -buffering line -blocking 0
to:
      fconfigure $data(pipe) -buffering line -blocking 0
Changed line 160 from:
      fileevent $data(pipe) readable ::maxima::receive
to:
      fileevent $data(pipe) readable ::maxima::receive
Changed line 164 from:
      maximaRunningBatch {DISPLAY2D:FALSE$}
to:
      maximaRunningBatch {DISPLAY2D:FALSE$}
Changed line 171 from:
      catch { close $data(pipe) }
to:
      catch { close $data(pipe) }
Changed line 179 from:
	  return [expr { [lsearch -exact [file channels] $data(pipe)] != -1 }]
to:
	  return [expr { [lsearch -exact [file channels] $data(pipe)] != -1 }]
Changed line 191 from:
      if { [eof $data(pipe)] } {
to:
      if { [eof $data(pipe)] } {
Changed line 197 from:
      append data(transcript) [read $data(pipe)]
to:
      append data(transcript) [read $data(pipe)]
Changed line 199 from:
      if { [regexp -start $data(size) -- {\(C\d+\)\s*$} $data(transcript)] } {
to:
      if { [regexp -start $data(size) -- {\(C\d+\)\s*$} $data(transcript)] } {
Changed lines 201-202 from:
	  if { [regexp -start $data(size) -- {.*\((D\d+)\)(.*)\(C\d+\)\s*$} $data(transcript) "" data(label) tmpRes] } {
	      set data(res) [string trim $tmpRes]
to:
	  if { [regexp -start $data(size) -- {.*\((D\d+)\)(.*)\(C\d+\)\s*$} $data(transcript) "" data(label) tmpRes] } {
	      set data(res) [string trim $tmpRes]
Changed line 208 from:
	  set data(size) [string length $data(transcript)]
to:
	  set data(size) [string length $data(transcript)]
Changed line 216 from:
      append data(transcript) $cmd
to:
      append data(transcript) $cmd
Changed lines 218-219 from:
      fileevent $data(pipe) readable ::maxima::receive
to:
      fileevent $data(pipe) readable ::maxima::receive
Changed line 221 from:
      puts $data(pipe) $cmd
to:
      puts $data(pipe) $cmd
Changed lines 227-228 from:
      after cancel $timeout
      if { [string equal $data(res) "TIMEOUT"] } {
to:
      after cancel $timeout
      if { [string equal $data(res) "TIMEOUT"] } {
Changed line 232 from:
      return $data(res)
to:
      return $data(res)
Changed lines 240-243 from:
      set preCmd {DISPLAY2D:FALSE$ }
      set transcript [exec maxima --batch-string=${preCmd}${cmd}]
      if { [regexp -- {.*\n\(D\d+\) (.*)\nBye.$} $transcript "" res] } {
	  return $res
to:
      set preCmd {DISPLAY2D:FALSE$ }
      set transcript [exec maxima --batch-string=${preCmd}${cmd}]
      if { [regexp -- {.*\n\(D\d+\) (.*)\nBye.$} $transcript "" res] } {
	  return $res
Changed lines 245-246 from:
	  regexp -- {.*\n\(C\d+\)[^\n]*\n(.*)\nBye.$} $transcript "" err
	  error $err
to:
	  regexp -- {.*\n\(C\d+\)[^\n]*\n(.*)\nBye.$} $transcript "" err
	  error $err
Changed lines 253-255 from:
      set cmd [join $args " "]
      set cmd [string trim $cmd]
      if { ![string equal [string index $cmd end] {;}] } {
to:
      set cmd [join $args " "]
      set cmd [string trim $cmd]
      if { ![string equal [string index $cmd end] {;}] } {
Changed line 260 from:
	  return [maximaRunningBatch $cmd]
to:
	  return [maximaRunningBatch $cmd]
Changed line 262 from:
	  return [maximaBatch $cmd]
to:
	  return [maximaBatch $cmd]
Changed lines 277-280 from:
      set preCmd {DISPLAY2D:FALSE$ }
      set transcript [exec maxima --batch-string=${preCmd}${cmd}tex(%)\;]
      if { [regexp -- {.*\n\(D\d+\).*\$(\$.*\$)\$\n\(D\d+\).*\nBye.$} $transcript "" res] } {
	  return $res
to:
      set preCmd {DISPLAY2D:FALSE$ }
      set transcript [exec maxima --batch-string=${preCmd}${cmd}tex(%)\;]
      if { [regexp -- {.*\n\(D\d+\).*\$(\$.*\$)\$\n\(D\d+\).*\nBye.$} $transcript "" res] } {
	  return $res
Changed lines 282-283 from:
	  if { [regexp -- {.*DISPLAY2D : FALSE\n(.*)\n\(C\d+\)} $transcript "" err] } {
	      error $err
to:
	  if { [regexp -- {.*DISPLAY2D : FALSE\n(.*)\n\(C\d+\)} $transcript "" err] } {
	      error $err
Changed line 285 from:
	  error $transcript
to:
	  error $transcript
Changed line 292 from:
      if { [eof $data(pipe)] } {
to:
      if { [eof $data(pipe)] } {
Changed line 298 from:
      append data(transcript) [read $data(pipe)]
to:
      append data(transcript) [read $data(pipe)]
Changed line 300 from:
      if { [regexp -start $data(size) -- {\(C\d+\)\s*$} $data(transcript)] } {
to:
      if { [regexp -start $data(size) -- {\(C\d+\)\s*$} $data(transcript)] } {
Changed lines 302-303 from:
	  if { [regexp -start $data(size) -- {.*\$(\$.*\$)\$\n\(D\d+\) FALSE\n\(C\d+\)\s*$} $data(transcript) "" tmpRes] } {
	      set data(res) [string trim $tmpRes]
to:
	  if { [regexp -start $data(size) -- {.*\$(\$.*\$)\$\n\(D\d+\) FALSE\n\(C\d+\)\s*$} $data(transcript) "" tmpRes] } {
	      set data(res) [string trim $tmpRes]
Changed line 309 from:
	  set data(size) [string length $data(transcript)]
to:
	  set data(size) [string length $data(transcript)]
Changed line 319 from:
      maximaRunningBatch $cmd
to:
      maximaRunningBatch $cmd
Changed line 324 from:
      fileevent $data(pipe) readable ::maxima::receiveTex
to:
      fileevent $data(pipe) readable ::maxima::receiveTex
Changed lines 326-327 from:
      puts $data(pipe) tex(%)\;
to:
      puts $data(pipe) tex(%)\;
Changed lines 333-334 from:
      after cancel $timeout
      if { [string equal $data(res) "TIMEOUT"] } {
to:
      after cancel $timeout
      if { [string equal $data(res) "TIMEOUT"] } {
Changed line 338 from:
      return $data(res)
to:
      return $data(res)
Changed lines 344-346 from:
      set cmd [join $args " "]
      set cmd [string trim $cmd]
      if { ![string equal [string index $cmd end] {;}] } {
to:
      set cmd [join $args " "]
      set cmd [string trim $cmd]
      if { ![string equal [string index $cmd end] {;}] } {
Changed line 351 from:
	  return [maximaTexRunningBatch $cmd]
to:
	  return [maximaTexRunningBatch $cmd]
Changed line 353 from:
	  return [maximaTexBatch $cmd]
to:
	  return [maximaTexBatch $cmd]
Added lines 417-671:

.

January 23, 2006, at 01:11 PM by 212.180.15.164 -
Added lines 1-416:

(:title Batch-like interaction via pipes:)

<Joachim 21/10/2003>

NOTE: Everything described on this page is pure Tcl and works equally well from tclsh.


Batch-like interaction with a unix process is meant to describe the situation where some external process is running (a sort of server, if you like) to which you can send commands for evaluation and get the result back as a return value.

Of course every interaction is more or less like this --- the concept is meant to be in contrast to shell-like interactions in that there is no window to receive the results as they come asynchronously --- instead we wait until the result is there and get it in a pure stripped-off form as the value of a variable. Hence it can be used in scripts as a sort of extension.


Maxima

As an example, the computer algebra system MAXIMA is considered (uppercased to distinguish from the Tcl command maxima introduced below). The MAXIMA project (one of the oldest in symbolic computation, and a primary source of inspiration for Maple) is hosted at SourceForge. The best way to install it under OSX is probably with the help of Fink http://fink.sourceforge.net. Note that you currently need to grant access to the unstable tree...

The simplest form for batch-like interactions is exec: you spawn some process with a one-liner, gets back the result, and then the process dies again. Many math programmes allow some sort of one-line invocation like this. For example you can call Maxima (from a unix shell) with a command like

 
    $ maxima --batch-string="diff(sin(x),x);"

This will return a dozen of lines of welcome header, and then the result you are interested in "COS(x)". From within Tcl it is not that difficult to wrap this in an exec and filter the output to get a little procedure that can evaluate MAXIMA expressions. (This is the proc maxima::maximaBatch below.)

However the main emphasis of this page is on the slightly different situation where instead of having such short-lived processes we want to have a process running on a more permanent basis, and send instructions to it and get back the results. There are at least two reasons to want this: one is that if you have many calculations to do it is more economical to keep a single MAXIMA session running than to spawn a new session for each command. The second reason is that in this way the state of the external process is preserved between the calls, and you can build up a complicated computation step by step.

Design

In the example below, a procedure maxima is defined in Tcl: it takes a list of arguments and send it to a running maxima process for evaluation in the currest state of the system. Hence you can do for example

 
    % maxima f : (x+y)^4
    % maxima g : (x-2*y)^2
    % maxima diff( f+g, x )
    % maxima expand( % )

and the final return value will be the string

 
    4*y^3+12*x*y^2+12*x^2*y-4*y+4*x^3+2*x

Sometimes, if you are sending such commands from a Tcl script, you can also get away with keeping the record of the state of the variables in Tcl, and hence make every command a one-liner not depending on the internam state of MAXIMA. In the simple example above it would be just as easy to do:

 
    % set f "(x+y)^4"
    % set g "(x-2*y)^2"
    % set res [maxima diff( $f+$g, x )]
    % maxima expand( $res )

Here comes a simple maxima batch interaction implementation. The code can also be downloaded from http://www.cirget.uqam.ca/~kock/alpha-tcl/maxima.tcl . At some point this should all evolve into a maxima mode for AlphaTcl. (The file also contains support for a variant that returns in LaTeX format. This is used with the notion of Pipes.Worksheets to have active MAXIMA cells in a LaTeX document.)

 
  # This is a batch-like interface to MAXIMA. The command [maxima] (which 
  # is just synonym for [maxima::maxima]) will send all its arguments to
  # evaluation by MAXIMA and return the result.  When called isolated, 
  # MAXIMA will be spawned and then quit immediately after returning the 
  # result.  Alternatively, a MAXIMA session can be started with the 
  # command [maxima::start].  In this case subsequent calls to [maxima] 
  # will be evaluated in this session (until the session is killed with 
  # [maxima::stop]).  The advantages of this approach is that it is more
  # economical to call an existing process than to spawn a new one for 
  # each call, and second, that the state of MAXIMA is preserved, so that 
  # succesive calls can refer to MAXIMA variables (either user-defined 
  # variables or labels like D5).  (There is also a variant proc [maximaTex]
  # which returns the result in TeX, which is useful in live math TeX 
  # documents (worksheets).)
  #
  # Examples:
  # 
  # % source maxima.tcl
  # % maxima factor( 23423412342131234 )
  # 2*131*173491*515313977
  # % set f [maxima expand( (x+y+z)^4 )]
  # z^4+4*y*z^3+4*x*z^3+6*y^2*z^2+12*x*y*z^2+6*x^2*z^2+4*y^3*z+12*x*y^2*z
  # 	+12*x^2*y*z+4*x^3*z+y^4+4*x*y^3+6*x^2*y^2+4*x^3*y+x^4
  # % maxima factor( $f )
  # (z+y+x)^4
  # 
  # Installation: first of all you need to have maxima installed on your
  # computer.  The easiest way if probably using fink.  Now write this 
  # line to your prefs.tcl file:
  # 
  #    source /path/to/thefile/maxima.tcl
  #    
  #    





  namespace eval maxima {}

  # The array data contains:
  #     transcript -- a log of everything done (like a shell)
  #     res -- last result
  #     label -- label of the last result (e.g. (D5))
  #     size -- the offset (in the transcript) of the previous feedback,
  #             including the prompt (e.g. just after (C5))

  proc maxima::start { } {
      # Close any old pipe:
      stop
      # Initialise the array:
      variable data
      set data(size) 0
      # Open the pipe:
      set data(pipe) [open "|maxima" RDWR]
      fconfigure $data(pipe) -buffering line -blocking 0
      # Set up a handler for the output:
      fileevent $data(pipe) readable ::maxima::receive
      # Wait until the receiver has finished with the MAXIMA welcome header:
      vwait ::maxima::data(res)
      # Then send a configuration instruction:
      maximaRunningBatch {DISPLAY2D:FALSE$}
      return
  }

  proc maxima::stop { } {
      variable data
      # Close the pipe:
      catch { close $data(pipe) }
      # Reset the data array:
      unset -nocomplain data
  }

  proc maxima::isRunning { } {
      variable data
      if { [info exists data(pipe)] } {
	  return [expr { [lsearch -exact [file channels] $data(pipe)] != -1 }]
      } else { 
	  return 0 
      }
  }



  # Event handler for data(pipe).  Writes the result to the variable 
  # data(res), from where the proc [maxima] reads it immediately.
  proc maxima::receive { } {
      variable data
      if { [eof $data(pipe)] } {
	  # There is nothing more to read --- just stop:
	  stop
	  return
      }
      # Just read as much as possible, and append it:
      append data(transcript) [read $data(pipe)]
      # When we read a promt, stop reading:
      if { [regexp -start $data(size) -- {\(C\d+\)\s*$} $data(transcript)] } {
	  # We have found a new prompt.  Hence the result should be just before that:
	  if { [regexp -start $data(size) -- {.*\((D\d+)\)(.*)\(C\d+\)\s*$} $data(transcript) "" data(label) tmpRes] } {
	      set data(res) [string trim $tmpRes]
	  } else {
	      # We should only come in here at startup, then we need to set data(res)
	      # since the caller is waiting for this variable to be set:
	      set data(res) ""
	  }
	  set data(size) [string length $data(transcript)]
      }
  }

  # Requires trailing semicolon
  proc maxima::maximaRunningBatch { cmd } {
      variable data
      # First update the transcript:
      append data(transcript) $cmd
      # Set up a handler for the output:
      fileevent $data(pipe) readable ::maxima::receive

      # Then send the command:
      puts $data(pipe) $cmd
      # Timeout mechanisms: We are going to wait for the variable res.  
      # Make sure it is written at least ofter some time:
      set timeout [after 10000 {set ::maxima::data(res) "TIMEOUT"}]
      vwait maxima::data(res)
      # If we have come so far there is no more need for the time bomb:
      after cancel $timeout
      if { [string equal $data(res) "TIMEOUT"] } {
	  stop
	  error "TIMEOUT"
      }
      return $data(res)
  }


  # This proc doesn't use pipes or file events or anything.
  # 
  # Requires trailing semicolon
  proc maxima::maximaBatch { cmd } {
      set preCmd {DISPLAY2D:FALSE$ }
      set transcript [exec maxima --batch-string=${preCmd}${cmd}]
      if { [regexp -- {.*\n\(D\d+\) (.*)\nBye.$} $transcript "" res] } {
	  return $res
      } else {
	  regexp -- {.*\n\(C\d+\)[^\n]*\n(.*)\nBye.$} $transcript "" err
	  error $err
      }
  }

  proc maxima::maxima { args } {
      variable data
      # Build the command:
      set cmd [join $args " "]
      set cmd [string trim $cmd]
      if { ![string equal [string index $cmd end] {;}] } {
	  append cmd {;}
      }
      # See if we have maxima running already:
      if { [isRunning] } {
	  return [maximaRunningBatch $cmd]
      } else {
	  return [maximaBatch $cmd]
      }
  }




  # ====================================================================
  # Here comes the part with tex formated output


  # This proc doesn't use pipes or file events or anything.
  # 
  # Requires trailing semicolon
  proc maxima::maximaTexBatch { cmd } {
      set preCmd {DISPLAY2D:FALSE$ }
      set transcript [exec maxima --batch-string=${preCmd}${cmd}tex(%)\;]
      if { [regexp -- {.*\n\(D\d+\).*\$(\$.*\$)\$\n\(D\d+\).*\nBye.$} $transcript "" res] } {
	  return $res
      } else {
	  if { [regexp -- {.*DISPLAY2D : FALSE\n(.*)\n\(C\d+\)} $transcript "" err] } {
	      error $err
	  }
	  error $transcript
      }
  }


  proc maxima::receiveTex { } {
      variable data
      if { [eof $data(pipe)] } {
	  # There is nothing more to read --- just stop:
	  stop
	  return
      }
      # Just read as much as possible, and append it:
      append data(transcript) [read $data(pipe)]
      # When we read a promt, stop reading:
      if { [regexp -start $data(size) -- {\(C\d+\)\s*$} $data(transcript)] } {
	  # We have found a new prompt.  Hence the result should be just before that:
	  if { [regexp -start $data(size) -- {.*\$(\$.*\$)\$\n\(D\d+\) FALSE\n\(C\d+\)\s*$} $data(transcript) "" tmpRes] } {
	      set data(res) [string trim $tmpRes]
	  } else {
	      # We should only come in here at startup, then we need to set data(res)
	      # since the caller is waiting for this variable to be set:
	      set data(res) ""
	  }
	  set data(size) [string length $data(transcript)]
      }
  }



  # Requires trailing semicolon
  proc maxima::maximaTexRunningBatch { cmd } {
      variable data
      # First we just send the command to usual evaluation:
      maximaRunningBatch $cmd

      # Now send the tex formating command

      # Set up a handler for the output:
      fileevent $data(pipe) readable ::maxima::receiveTex
      # Send the tex formating command
      puts $data(pipe) tex(%)\;

      # Timeout mechanisms: We are going to wait for the variable res.  
      # Make sure it is written at least ofter some time:
      set timeout [after 10000 {set ::maxima::data(res) "TIMEOUT"}]
      vwait maxima::data(res)
      # If we have come so far there is no more need for the time bomb:
      after cancel $timeout
      if { [string equal $data(res) "TIMEOUT"] } {
	  stop
	  error "TIMEOUT"
      }
      return $data(res)
  }

  proc maxima::maximaTex { args } {
      variable data
      # Build the command:
      set cmd [join $args " "]
      set cmd [string trim $cmd]
      if { ![string equal [string index $cmd end] {;}] } {
	  append cmd {;}
      }
      # See if we have maxima running already:
      if { [isRunning] } {
	  return [maximaTexRunningBatch $cmd]
      } else {
	  return [maximaTexBatch $cmd]
      }
  }



  namespace eval maxima {
      namespace export maxima maximaTex
  }

  namespace import -force maxima::maxima maxima::maximaTex

Some implementation issues

A general problem to cope with is to determine when the output is complete. The output might consists of several lines and it might take a very long time before the last line of output arrives if it is a complicated calculation. The only way seems to be to look out for the prompt...

Second, since the proc that sends the command to the external process cannot also be the event handler, it is tricky for the calling proc to get the result back from the event handler and return it to the user. One solution which seems to work is to let the event handler write the result to a global variable when it is sure that the output is complete; then the calling proc waits (using vwait) for this variable to be written and when this happens its value is returned to the user. This is an awkward way of making the inherently asynchronous filehandler situation look like something synchronous which can be used in a usual linear script. Another possibility which is perhaps more direct is to let the calling proc have its own little ad hoc waiting scheme, instead of relying on a file event... It could simply go into a little loop that for every fraction of a second reads what ready in the pipe and continue like that until the prompt appears...

Emulating prompts

Further complications arise in the cases where the prompt is lost. For example it seems that maple only returns complete lines, so the prompt will only be sent back to the tcl interface as part of the next command. To handle such a situation one can envisage to wrap every command in a dummy command to mark end of output. For example in maple one could make the internal convention always to send command foo as foo; ENDOFOUTPUT;. Maple will then first process the command foo (the extension of whose output we have no control over per se), and afterwards process the command ENDOFOUTPUT which is not a known maple command, and will therefore just be repeated verbatim in the output. The callback proc will then patiently wait for the string ENDOFOUTPUT to appear in the stream, and when it does it will be filtered away, and the remaining text (the output from foo) will be written to the global variable res where the original procs avwaits it.

Implementation without fileevent

In a sense it is a little backwards to use an event loop to handle the output since we have to wait until the output is complete before we really handle it. Here is an attempt to implement maxima without fileevent, setting up instead its own little personal event loop. I don't know if there are any advantages to this... This has not been given further attention, and the script (downloadable from http://www.cirget.uaqm.ca/~kock/alpha-tcl/alt-maxima.tcl ) is not up-to-date compared to the mainstream maxima script above.

Page last modified on April 19, 2006, at 11:51 PM
Hosted on SourceForge.net Logo