Worksheets

<JK:2005-03-06> I think the following is only cheap and naive approach to the fancy functionality available in Lars Hellström's EEMenu, which recently has become part of the standard AlphaTcl distribution. (It doesn't work currently, due to recent changes in all that SIG business.)

Nevertheless, the following works well too. The worksheet package described below can be downloaded from http://mat.uab.es/~kock/alpha-tcl/worksheet.tcl.gz . This is for live Tcl in any document. If you also want live Maxima, you need to install also the maxima batch interaction script (described on the page Batch-like interaction (via pipes)), downloadable from http://mat.uab.es/~kock/alpha-tcl/maxima.tcl.gz .

Introduction to these scripts are at the bottom of this page.


A worksheet is a file in which certain areas --- called cells --- are active: their content can be sent to an external process for evaluation and the result is then inserted in the cell. The concept is very convenient particularly whenever you are documenting some code (i.e., whenever you write code!) or when writing scientific text whose content depends on computations. The idea is that the writer has access to the whole file --- it is just a file in his favourite text editor --- while the external process has only access to the cells, and only when the access is granted specifically by some trigger mechanism. A third level of interpretation makes the circle complete from the viewpoint of documentation: to arrange the structure and syntax of the document in such a way that some typesetting system can interpret it --- typically TeX, but it might also be HTML. For this to work, the tags delimiting the cells should be understandable by the interpreter (e.g. \begin{cell} \end{cell}, interpreted by a special purpose TeX package to render the code in verbatim or whatever...).

Worksheets are probably best known from the graphical interfaces to expensive math software like Maple or Mathematica (in Mathematica they are called notebooks). In fact the idea goes a long time back, at least to Donald Knuth's idea of literate programming and WEB documents (a long time before the word web was associated with the internet). The concept has been developed a lot in Emacs: famous Emacs worksheet interfaces to external math programmes include David Jacobson's Mathematica mode, later developed into a fancy TeX/Mathematica interaction Emacs mode by Dan Dill, which in turn inspired the Emacs TeX/Maxima interface Emaxima. The doc-projects idea in LaTeX is also worth mentionning.

Alpha already has support for worksheets (although the 'external' process is only the internal Tcl interpreter): to see it, just open a shell window (Cmd-Y) and delete everything including the prompt. Now you can type as in any text editing window, insert newlines with <return>, and so on. You open a cell by typing «» --- the rest of the line is now in a cell, and the trigger that sends its contents to evaluation is simply <return>. The result of the evaluation is automatically inserted below the input part of the cell. (Since in fact this mechanism is designed to be a shell, a new cell is opened immediately after the output region, but you can just delete it and continue writing outside the cell.) Since you can position your cursor anywhere in Alpha's shell, in fact you might argue that it is a worksheet rather than a shell, but unlike many worksheet implementations it has the peculiarity that evaluated cells are always appended to the bottom of the document --- perhaps this design was chosen to make the document more shell-like...

It is also clear from this example that a worksheet has all the functionality a shell has --- you could define a shell to be a worksheet where the cells take up all the space, and where only the last cell can be evaluated. Hence it will many times make sense to use worksheet *instead* of shells, and in other situations it would be convenient to regard a TeX document as a worksheet for some secondary math programme involved.

The idea of replacing shell centric interfaces by editor centric is also exposed in http://www.emacswiki.org/johnw/eshell.html

So Alpha needs some support for worksheets. Of course the most important prerequisite is the ability to communicate with external processes. A lot of experiments have been conducted with pipes and sockets, and for the most part it is not difficult to send a command to a process and receive its answer. However, so far the experiments have only aimed at shell-like implementations, and haven't yet started to tackle the worksheet problems.

In fact, it might be reasonable to envisage an implementation on top of a shell-like interface: for the sake of allowing cells to refer to each other, one approach would be to maintain a complete log of all information exchanged between the worksheet and the external process, and such a log is nothing but the text produced by the shell. This shell may or may not be visible on the screen. (In Emaxima it is actually visible).

So assuming a shell-like interaction is already in place see Pipes and shell-like interaction, only three tasks remain:

The markup conventions should be easy to arrange.

For the input parsing: given the cursor position in the worksheet, find out if it belongs to a cell, and if yes, get the input region of it together with the nature of the cell. (It is probably practical to have many sorts of cells, depending on the sort of evaluation is wanted --- it might even be possible that a single worksheet communicate with various different processes!) (Before sending the string to the external process it might be opportune to preprocess it a little bit, like checking for well-formedness, or wrapping it in meta code or whatever might seem necessary.)

The output parsing faces the problem of knowing when the output is complete. Often this might be done by waiting for the next prompt. In any case, the prompt must be stripped off. (These problems have already been dealt with in the implementation of Batch-like interaction (via pipes).)

For shell-like interactions you can often get away with blindly copying the reply from the external process --- with prompts and everything --- and hence it is possible to set up quite generic solutions that work for many different programmes. For worksheet-like interactions it will often seem necessary to parse the reply in some way, at least to strip off prompts, and therefore programme-specific handling seems to be required. You'll also want special treatment of error messages...

Example: TeX documents with live math

This example is a simple worksheet concept for use in LaTeX documents where some mathematical calculations are performed in MAXIMA.

Design: A cell consists of an input region possibly followed by an output region. An input region is a contiguous block of lines starting with %% (hence the MAXIMA input will never be read by the tex compiler); an output region is a contiguous block of lines starting with {}. This is just for the sake of marking it clearly while letting the marking be neutral to tex.

There is a key binding for inserting a cell template:

    %% •
    {}
    •

with the cursor at the first bullet. Here we write something like factor(x^2-1); and press the trigger to send the input region to MAXIMA. The result is the latex formatted expression $\left(x-1\right)\,\left(x+1\right)$ produced by MAXIMA and inserted after the {} tag, and the cursor ends up at the second bullet where we can continue typing. In a fancier version one could have %%%% to mean double dollar and %% for single dollar, but this distinction could also be implemented in terms of the trigger: one key binding for inserting $ and another for $$.

    The polynomial
    %% x^2-1;
    {} $x^2-1$   %(D2)
    is not irreducible.  It factors as
    %%%% factor(D2);
    {} $$\left(x-1\right)\,\left(x+1\right)$$ %(D4)

The whole concept is particularly useful for generating exercises, since these can be constructed backwards, using the answer to generate the question. E.g.

    Exercise 1.  Find the roots of the polynomial
    %% expand( (x-1)*(x+3)*(x-4) )
    {} $x^3-2\,x^2-11\,x+12$
    (without the help of a symbolic calculator).

    Exercise 2. Diagonalise the matrix
    %% D: matrix( [3,0], [0,5] )
    %% S: matrix( [1,2], [0,1] )
    %% S.D.S^^-1
    {} $\pmatrix{3&4\cr 0&5\cr }$    % D6

Implementation: the input proc parses and finds the string x^2-1. Then it sends the command x^2-1; tex(%); to MAXIMA

MAXIMA returns a lot of junk --- we only want $x^2-1$ as well as the label (D2) We put it into the output region appropriately.


Concrete instructions for the worksheet package of http://mat.uab.es/~kock/alpha-tcl/worksheet.tcl.gz and even more blurb:

 # Installation: write this line in your prefs.tcl file
 #
 #    source /path/to/thefile/worksheet.tcl
 #
 # (If you also want to use maxima, as in some of the examples below,
 # write also a line
 #
 #    source /path/to/thefile/maxima.tcl
 #
 # Of course in order for this to make sense you must to have maxima
 # installed on your computer.  Best way is probably via fink.)
 #
 # --------------------------------------------------------------------
 #
 # Description:
 #
 # This is a general worksheet package whose active cells are evaluated by
 # Alpha's Tcl interpreter, which in turn can link to other interpreters
 # like a shell, a computer algebra system, etc.
 #
 # The notion of worksheet is extremely flexible and there are myriads of
 # situations where it is useful.  At the most naive level, a worksheet is
 # just a sort of shell, whose structure is more like a document or a
 # scratchpad than like a command line.  That is, you have a window from
 # where you issue commands to the Tcl interpreter, and receive the results.
 # The difference is that the layout is much more flexible: you can insert
 # and delete active cells (prompts) anywhere, you can reuse them, delete
 # them, edit the file like any text file.  At the other extreme, it can be
 # merely a latex file with some power features: at any point in the tex
 # file you can open a prompt and evaluate a command.  At the most primitive
 # level, this means for example that your latex document has a built-in
 # calculator.  You can write some text like this:
 #
 # |This gives a total area of
 # |%% expr {230*31}
 # |7130
 # |square millimeters.
 #
 # In this example, just after 'area of' we pressed <return>, and then
 # invoked the prompt by Ctrl-Y ctrl-Y, then wrote the Tcl calculation, then
 # pressed Ctrl-Y ctrl-Y again to evaluate; the result was inserted on the
 # line below, and the cursor on the line below again, and we can continue
 # writing.  Observe that the prompt is at the same time a TeX comment, so
 # the tex compiler will not take notice of the input region of the active
 # cell.
 #
 # Now this simple example takes new dimensions when you realise that it is
 # no more complicated to use a fancy 'calculator' like MAXIMA, if installed
 # on your system.  With the Alpha interface to maxima, we might also write
 # something like this in our tex document:
 #
 # |Exercise 1.  Factor the polynomial
 # |%% maximaTex expand ( (x+2*y)*(3*x-z) )
 # |$-2\,y\,z-x\,z+6\,x\,y+3\,x^2$
 # |(using only pencil and paper).
 #
 # Again, all it takes is a couple of Ctrl-Y ctrl-Y.  In this example,
 # [maximaTex] is a proc that asks MAXIMA to evaluate and return the
 # result in tex.  Writing exams is a piece of cake:
 #
 # |Exercise 2.  Diagonalise the matrix
 # |%% set D [maxima { matrix( [3,0], [0,5] ) }]
 # |%% set S [maxima { matrix( [1,2], [0,1] ) }]
 # |%% maximaTex $S . $D . $S^^-1
 # |$\pmatrix{3&4\cr 0&5\cr }$
 #
 # And nothing prevents you from editing the latex code by hand afterwards,
 # for example changing the single dollar to double dollar if that's better.
 # Note that the whole block of %% prefixed lines evaluates in one go ---
 # you don't have to press the evaluate trigger three times.  It is also
 # easy to go back in the input region and change one of the numbers if
 # necessary, and hit the trigger again.  This will replace the output
 # region with the new value.
 #
 # But how can the mechanism know what the output region is?  There are two
 # approaches, depending on a preference flag: one approach is heuristic:
 # the mechanism always appends a tab-space-tab sequence to the output, and
 # recognises output regions as lines with a traling "\t \t".  Such a
 # trailing sequence is not very common anywhere, so this method very seldom
 # does any dammage (in any case these insertions are undo-able, if
 # necessary).  The second approach goes to the other extreme: it inserts a
 # date stamp and a checksum.  In this way you can always see exactly when
 # the output was generated, and before replacing it, the mechanism checks
 # the region against the stamped checksum to avoid overwriting something
 # that had been manually edited in the interim.  This is the preferred
 # method in serious tex documents or when dealing with complicated data.
 # Example:
 #
 # |So it remains to integrate
 # |%% set F "1/(x^2-c^2)^5"
 # |%% maximaTex $F
 # |%----------------------------------output written 2004/06/21 12:25:18 EDT----
 # |${{1}\over{\left(x^2-c^2\right)^5}}$
 # |%---------------end of output 2004/06/21 12:25:18 EDT checksum:0x862FD4B8----
 # |which is tedious but straight-forward --- the result is
 # |%% maximaTex integrate( $F,x )
 # |%----------------------------------output written 2004/06/21 12:25:48 EDT----
 # |$-{{35\,\log \left(x+c\right)}\over{256\,c^9}}+{{35\,\log \left(x-c
 # | \right)}\over{256\,c^9}}+{{105\,x^7-385\,c^2\,x^5+511\,c^4\,x^3-279
 # | \,c^6\,x}\over{384\,c^8\,x^8-1536\,c^{10}\,x^6+2304\,c^{12}\,x^4-
 # | 1536\,c^{14}\,x^2+384\,c^{16}}}$
 # |%---------------end of output 2004/06/21 12:25:48 EDT checksum:0x8C8929F4----
 #
 # The trigger key (Ctrl-Y Ctrl-Y) is a multi-function key: If there is no
 # input region, then it creates one.  If there is an input region with a
 # command or script to evaluate, then it evaluates it and writes the output
 # in the following output region.  If the trigger is pressed in an empty
 # input region, then this region (that's just the prompt then) is removed.
 # This is very practical in conjonction with the auto-prompt setting: the
 # auto-prompt setting inserts a new prompt each time an expression is
 # evaluated (like in a shell).  Then if you didn't want that prompt, just
 # press the trigger and it will vanish.
 #
 # There are also some navigation procs: Ctrl-y ctrl-up and ctrl-y ctrl-down
 # for navigating between input regions, and ctrl-y ctrl-o for selecting the
 # next output region.
 #
 # If you are using a worksheet as a scratch pad for many computations (like
 # using it as a substitute for a shell), then you will probably think that
 # the trigger key ctrl-y ctrl-y is too complicated to be practical.  In
 # that case you should issue the command
 # %% worksheet::worksheet
 # This will pep up this particular worksheet with some more shell-like
 # functionality.  For one thing <enter> is now a trigger in addition to
 # ctrl-y ctrl-y, and ctrl-up and ctrl-down jump between input regions.
 # There is also a history mechanism activated (bound to ctrl-left).
 # (These features are probably not convenient for true documents like a
 # latex document where only a couple of occasional active cells are
 # used.)