source: trunk/abcl/doc/manual/abcl.tex @ 13625

% -*- mode: latex; -*-
% http://en.wikibooks.org/wiki/LaTeX/
\documentclass[10pt]{book}
\usepackage{abcl}

\begin{document}
\title{A Manual for Armed Bear Common Lisp}
\date{October 2, 2011}
\author{Mark~Evenson, Erik~Huelsmann, Alessio~Stalla, Ville~Voutilainen}

\maketitle

\chapter{Introduction}

Armed Bear is a mostly conforming implementation of the ANSI Common
Lisp standard.  This manual documents the Armed Bear Common Lisp
implementation for users of the system.

\subsection{Version}
This manual corresponds to abcl-0.28.0, as yet unreleased.

\chapter{Running}

ABCL is packaged as a single jar file usually named either
``abcl.jar'' or something like``abcl-0.28.0.jar'' if you are using a
versioned package from your system vendor.  This byte archive can be
executed under the control of a suitable JVM by using the ``-jar''
option to parse the manifest, and select the named class
(org.armedbear.lisp.Main) for excution:

\begin{listing-shell}
  cmd$ java -jar abcl.jar
\end{listing-shell}

N.b. for the proceeding command to work, the ``java'' executable needs
to be in your path.

To make it easier to facilitate the use of ABCL in tool chains (such as
SLIME) the invocation is wrapped in a Bourne shell script under UNIX
or a DOS command script under Windows so that ABCL may be executed
simply as:

\begin{listing-shell}
  cmd$ abcl
\end{listing-shell}

\section{Options}

ABCL supports the following options:

\begin{verbatim}
--help
    Displays this message.
--noinform
    Suppresses the printing of startup information and banner.
--noinit
    Suppresses the loading of the '~/.abclrc' startup file.
--nosystem
    Suppresses loading the 'system.lisp' customization file. 
--eval <FORM>
    Evaluates the <FORM> before initializing REPL.
--load <FILE>
    Loads the file <FILE> before initializing REPL.
--load-system-file <FILE>
    Loads the system file <FILE> before initializing REPL.
--batch
    The process evaluates forms specified by arguments and possibly by those
    by those in the intialization file '~/.abcl', and then exits.

The occurance of '--' copies the remaining arguments, unprocessed, into
the variable EXTENSIONS:*COMMAND-LINE-ARGUMENT-LIST*.
\end{verbatim}

All of the command line arguments which follow the occurrence of ``--''
are passed into a list bound to the EXT:*COMMAND-LINE-ARGUMENT-LIST*
variable.

\section{Initialization}

If the ABCL process is started without the ``--noinit'' flag, it
attempts to load a file named ``.abclrc'' located in the user's home
directory and then interpret its contents.  

The user's home directory is determined by the value of the JVM system
property ``user.home''.

\chapter{Conformance}

\section{ANSI Common Lisp}
ABCL is currently a non-conforming ANSI Common Lisp implementation due
to the following issues:

\begin{itemize}
  \item Missing statement of conformance in accompanying documentation
  \item The generic function signatures of the DOCUMENTATION symbol do
    not match the CLHS.
\end{itemize}

ABCL aims to be be a fully conforming ANSI Common Lisp
implementation.  Any other behavior should be reported as a bug.

\section{Contemporary Common Lisp}
In addition to ANSI conformance, ABCL strives to implement features
expected of a contemporary Common Lisp.
\begin{itemize}
  \item Incomplete (A)MOP 
    % N.B. 
    % TODO go through AMOP with symbols, starting by looking for
    % matching function signature.
    % XXX is this really blocking ANSI conformance?  Answer: we have
    % to start with such a ``census'' to determine what we have.
  \item Incomplete Streams:  need suitable abstraction between ANSI
    and Gray streams.
    
\end{itemize}

\chapter{Interaction with host JVM}

% describe calling Java from Lisp, and calling Lisp from Java,
% probably in two separate sections.  Presumably, we can partition our
% audience into those who are more comfortable with Java, and those
% that are more comforable with Lisp

\section{Lisp to Java}

ABCL offers a number of mechanisms to interact with Java from
its lisp environment. It allows calling methods (and static methods) of
Java objects, manipulation of fields and static fields and construction
of new Java objects.

When calling Java routines, some values will automatically be converted
by the FFI from Lisp values to Java values. These conversions typically
apply to strings, integers and floats. Other values need to be converted
to their Java equivalents by the programmer before calling the Java
object method. Java values returned to Lisp are also generally converted
back to their Lisp counterparts. Some operators make an exception to this
rule and do not perform any conversion; those are the ``raw'' counterparts
of certain FFI functions and are recognizable by their name ending with
\code{-RAW}.

\subsection{Lowlevel Java API}

There's a higher level Java API defined in the
\ref{topic:Higher level Java API: JSS}(JSS package) which is available
in the contrib/ directory. This package is described later in this
document.  This section covers the lower level API directly available
after evaluating \code{(require 'JAVA)}.

\subsubsection{Calling Java object methods}

There are two ways to call a Java object method in the basic API:

\begin{itemize}
\item Call a specific method reference (pre-acquired)
\item Dynamic dispatch using the method name and
  the call-specific arguments provided by finding the
  \ref{section:Parameter matching for FFI dynamic dispatch}{best match}.
\end{itemize}

The dynamic dispatch variant is discussed in the next section.

\code{JAVA:JMETHOD} is used to acquire a specific method reference.
The function takes at two or more arguments. The first is Java class designator
(a \code{JAVA:JAVA-CLASS} object returned by \code{JAVA:JCLASS} or a string naming
a Java class). The second is a string naming the method.

Any arguments beyond the first two should be strings naming Java classes with
one exception as listed in the next paragraph. These
classes specify the types of the arguments for the method to be returned.

There's additional calling convention to the \code{JAVA:JMETHOD} function:
When the method is called with three parameters and the last parameter is an
integer, the first method by that name and matching number of parameters is
returned.

Once you have a reference to the method, you can call it using \code{JAVA:JCALL},
which takes the method as the first argument. The second argument is the
object instance to call the method on, or \code{NIL} in case of a static method.
Any remaining parameters are used as the remaining arguments for the call.

\subsubsection{Calling Java object methods: dynamic dispatch}

The second way of calling Java object methods is by using dynamic dispatch.
In this case \code{JAVA:JCALL} is used directly without acquiring a method
reference first. In this case, the first argument provided to \code{JAVA:JCALL}
is a string naming the method to be called. The second argument is the instance
on which the method should be called and any further arguments are used to
select the best matching method and dispatch the call.

\subsubsection{Dynamic dispatch: caveats}

Dynamic dispatch is performed by using the Java reflection API. Generally
it works fine, but there are corner cases where the API does not correctly
reflect all the details involved in calling a Java method. An example is
the following Java code:

\begin{listing-java}
ZipFile jar = new ZipFile("/path/to/some.jar");
Object els = jar.entries();
Method method = els.getClass().getMethod("hasMoreElements");
method.invoke(els);
\end{listing-java}

even though the method \code{hasMoreElements} is public in \code{Enumeration},
the above code fails with

\begin{listing-java}
java.lang.IllegalAccessException: Class ... can
not access a member of class java.util.zip.ZipFile$2 with modifiers
"public"
       at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:65)
       at java.lang.reflect.Method.invoke(Method.java:583)
       at ...
\end{listing-java}

because the method has been overridden by a non-public class and the
reflection API, unlike javac, is not able to handle such a case.

While code like that is uncommon in Java, it is typical of ABCL's FFI
calls. The code above corresponds to the following Lisp code:

\begin{listing-lisp}
(let ((jar (jnew "java.util.zip.ZipFile" "/path/to/some.jar")))
  (let ((els (jcall "entries" jar)))
    (jcall "hasMoreElements" els)))
\end{listing-lisp}

except that the dynamic dispatch part is not shown.

To avoid such pitfalls, all Java objects in ABCL carry an extra
field representing the ``intended class'' of the object. That is the class
that is used first by \code{JAVA:JCALL} and similar to resolve methods;
the actual class of the object is only tried if the method is not found
in the intended class. Of course, the intended class is always a superclass
of the actual class - in the worst case, they coincide. The intended class
is deduced by the return type of the method that originally returned
the Java object; in the case above, the intended class of \code{ELS}
is \code{java.util.Enumeration} because that's the return type of
the \code{entries} method.

While this strategy is generally effective, there are cases where the
intended class becomes too broad to be useful. The typical example
is the extraction of an element from a collection, since methods in
the collection API erase all types to \code{Object}. The user can
always force a more specific intended class by using the \code{JAVA:JCOERCE}
operator.

% \begin{itemize}
% \item Java values are accessible as objects of type JAVA:JAVA-OBJECT.
% \item The Java FFI presents a Lisp package (JAVA) with many useful
%   symbols for manipulating the artifacts of expectation on the JVM,
%   including creation of new objects \ref{JAVA:JNEW}, \ref{JAVA:JMETHOD}), the
%   introspection of values \ref{JAVA:JFIELD}, the execution of methods
%   (\ref{JAVA:JCALL}, \ref{JAVA:JCALL-RAW}, \ref{JAVA:JSTATIC})
% \item The JSS package (\ref{JSS}) in contrib introduces a convenient macro
%   syntax \ref{JSS:SHARPSIGN_DOUBLEQUOTE_MACRO} for accessing Java
%   methods, and additional convenience functions.
% \item Java classes and libraries may be dynamically added to the
%   classpath at runtime (JAVA:ADD-TO-CLASSPATH).
% \end{itemize}

\subsubsection{Calling Java class static methods}

Like with non-static methods, references to static methods can be acquired
by using the \code{JAVA:JMETHOD} primitive. In order to call this method,
it's not possible to use the \code{JAVA:JCALL} primitive however: there's a 
separate API to retrieve a reference to static methods. This
primitive is called \code{JAVA:JSTATIC}. 

Like \code{JAVA:JCALL}, \code{JAVA:JSTATIC} supports dynamic dispatch by
passing the name of the method as a string instead of passing a method reference.
The parameter values should be values to pass in the function call instead of
a specification of classes for each parameter.

\subsubsection{Parameter matching for FFI dynamic dispatch}

The algorithm used to resolve the best matching method given the name
and the arguments' types is the same as described in the Java Language
Specification. Any deviation should be reported as a bug.

% ###TODO reference to correct JLS section

\subsubsection{Instantiating Java objects}

Java objects can be instantiated (created) from Lisp by calling
a constructor from the class of the object to be created. The same way
\code{JAVA:JMETHOD} is used to acquire a method reference, the
\code{JAVA:JCONSTRUCTOR} primitive can be used to acquire a constructor
reference. It's arguments specify the types of arguments of the constructor
method the same way as with \code{JAVA:JMETHOD}.

The constructor can't be passed to \code{JAVA:JCALL}, but instead should
be passed as an argument to \code{JAVA:JNEW}.

\section{Lisp from Java}

In order to access the Lisp world from Java, one needs to be aware
of a few things. The most important ones are listed below.

\begin{itemize}
\item All Lisp values are descendants of LispObject.java
\item In order to 
\item Lisp symbols are accessible via either directly referencing the
  Symbol.java instance or by dynamically introspecting the
  corresponding Package.java instance.
\item The Lisp dynamic environment may be saved via
  \code{LispThread.bindSpecial(Binding)} and restored via
  \code{LispThread.resetSpecialBindings(Mark)}.
\item Functions may be executed by invocation of the
  Function.execute(args [...]) 
\end{itemize}

\subsection{Lisp FFI}

FFI stands for "Foreign Function Interface" which is the phase which
the contemporary Lisp world refers to methods of "calling out" from
Lisp into "foreign" languages and environments.  This document
describes the various ways that one interacts with Lisp world of ABCL
from Java, considering the hosted Lisp as the "Foreign Function" that
needs to be "Interfaced".

\subsection{Calling Lisp from Java}

Note: As the entire ABCL Lisp system resides in the org.armedbear.lisp
package the following code snippets do not show the relevant import
statements in the interest of brevity.  An example of the import
statement would be

\begin{listing-java}
  import org.armedbear.lisp.*;
\end{listing-java}

to potentially import all the JVM symbol from the `org.armedbear.lisp'
namespace.

Per JVM, there can only ever be a single Lisp interpreter.  This is
started by calling the static method `Interpreter.createInstance()`.

\begin{listing-java}
  Interpreter interpreter = Interpreter.createInstance();
\end{listing-java}

If this method has already been invoked in the lifetime of the current
Java process it will return null, so if you are writing Java whose
life-cycle is a bit out of your control (like in a Java servlet), a
safer invocation pattern might be:

\begin{listing-java}
  Interpreter interpreter = Interpreter.getInstance();
  if (interpreter == null) {
    interpreter = Interpreter.createInstance();
  }
\end{listing-java}


The Lisp \code{eval} primitive may be simply passed strings for evaluation,
as follows

\begin{listing-java}
  String line = "(load \"file.lisp\")";
  LispObject result = interpreter.eval(line);
\end{listing-java}

Notice that all possible return values from an arbitrary Lisp
computation are collapsed into a single return value.  Doing useful
further computation on the ``LispObject'' depends on knowing what the
result of the computation might be, usually involves some amount
of \code{instanceof} introspection, and forms a whole topic to itself
(c.f. [Introspecting a LispObject])

Using \code{eval} involves the Lisp interpreter.  Lisp functions may
be directly invoked by Java method calls as follows.  One simply
locates the package containing the symbol, then obtains a reference to
the symbol, and then invokes the \code{execute()} method with the
desired parameters.

\begin{listing-java}
    interpreter.eval("(defun foo (msg) (format nil \"You told me '~A'~%\" msg))");
    Package pkg = Packages.findPackage("CL-USER");
    Symbol foo = pkg.findAccessibleSymbol("FOO"); 
    Function fooFunction = (Function)foo.getSymbolFunction();
    JavaObject parameter = new JavaObject("Lisp is fun!");
    LispObject result = fooFunction.execute(parameter);
    // How to get the "naked string value"?
    System.out.println("The result was " + result.writeToString()); 
\end{listing-java}

If one is calling an primitive function in the CL package the syntax
becomes considerably simpler.  If we can locate the instance of
definition in the ABCL Java source, we can invoke the symbol directly.
For instnace, to tell if a `LispObject` contains a reference to a symbol.

\begin{listing-java}
    boolean nullp(LispObject object) {
      LispObject result = Primitives.NULL.execute(object);
      if (result == NIL) { // the symbol 'NIL' is explicity named in the Java
                           // namespace at ``Symbol.NIL''
                           // but is always present in the
                           // localnamespace in its unadorned form for
                           // the convenience of the User.
        return false;
      }
      return true;
   }
\end{listing-java}

\subsubsection{Introspecting a LispObject}
\label{topic:Introspecting a LispObject}

We present various patterns for introspecting an an arbitrary
`LispObject` which can represent the result of every Lisp evaluation
into semantics that Java can meaningfully deal with.

\subsubsection{LispObject as \code{boolean}}

If the LispObject a generalized boolean values, one can use
\code{getBooleanValue()} to convert to Java:

\begin{listing-java}
     LispObject object = Symbol.NIL;
     boolean javaValue = object.getBooleanValue();
\end{listing-java}

Although since in Lisp, any value other than NIL means "true"
(so-called generalized Boolean), the use of Java equality it quite a
bit easier to type and more optimal in terms of information it conveys
to the compiler would be:

\begin{listing-java}
    boolean javaValue = (object != Symbol.NIL);
\end{listing-java}

\paragraph{LispObject is a list}

If LispObject is a list, it will have the type `Cons`.  One can then use
the \code{copyToArray} to make things a bit more suitable for Java
iteration.

\begin{listing-java}
    LispObject result = interpreter.eval("'(1 2 4 5)");
    if (result instanceof Cons) {
      LispObject array[] = ((Cons)result.copyToArray());
      ...
    }
\end{listing-java}
    
A more Lispy way to iterated down a list is to use the `cdr()` access
function just as like one would traverse a list in Lisp:;

\begin{listing-java}
    LispObject result = interpreter.eval("'(1 2 4 5)");
    while (result != Symbol.NIL) {
      doSomething(result.car());
      result = result.cdr();
    }
\end{listing-java}

\chapter{Implementation Dependent Extensions}

As outlined by the CLHS ANSI conformance guidelines, we document the
extensions to the Armed Bear Lisp implementation made accessible to
the user by virtue of being an exported symbol in the JAVA, THREADS,
or EXTENSIONS packages.

\section{JAVA}

\subsection{Modifying the JVM CLASSPATH}

The JAVA:ADD-TO-CLASSPATH generic functions allows one to add the
specified pathname or list of pathnames to the current JVM classpath
allowing the dynamic loading of JVM objects:

\begin{listing-lisp}
CL-USER> (add-to-classpath "/path/to/some.jar")
\end{listing-lisp}

\subsection{API}

% include autogen docs for the JAVA package.
\include{java}

\section{THREADS}

Multithreading

\subsection{API}

% include autogen docs for the THREADS package.
\include{threads}

\section{EXTENSIONS}

The symbols in the EXTENSIONS package (nicknamed ``EXT'') constitutes
extensions to the ANSI standard that are potentially useful to the
user.  They include functions for manipulating network sockets,
running external programs, registering object finalizers, constructing
reference weakly held by the garbage collector and others.

See \ref{Extensible Sequences} for a generic function interface to
the native JVM contract for \code{java.util.List}.

\subsection{API}

% include autogen docs for the EXTENSIONS package.
\include{extensions}

\chapter{Beyond ANSI}

Naturally, in striving to be a useful contemporary Common Lisp
implementation, ABCL endeavors to include extensions beyond the ANSI
specification which are either widely adopted or are especially useful
in working with the hosting JVM.

\section{Implementation Dependent}
\begin{enumerate}
  \item Compiler to JVM 5 bytecode
  \item Pathname extensions
\end{enumerate}

\section{Pathname}

We implment an extension to the Pathname that allows for the
description and retrieval of resources named in a URI scheme that the
JVM ``understands''.  Support is built-in to the ``http'' and
``https'' implementations but additional protocol handlers may be
installed at runtime by having JVM symbols present in the
sun.net.protocol.dynmamic pacakge. See [JAVA2006] for more details.

ABCL has created specializations of the ANSI Pathname object to
enable to use of URIs to address dynamically loaded resources for the
JVM.  A URL-PATHNAME has a corresponding URL whose cannoical
representation is defined to be the NAMESTRING of the Pathname.

PATHNAME : URL-PATHNAME : JAR-PATHNAME

Both URL-PATHNAME and JAR-PATHNAME may be used anu where will a
PATHNAME is accepted witht the following caveats

A stream obtained via OPEN on a URL-PATHNAME cannot be the target of
write operations.

No canonicalization is performed on the underlying URI (i.e. the
implementation does not attempt to compute the current name of the
representing resource unless it is requested to be resolved.)  Upon
resolution, any cannoicalization procedures followed in resolving the
resource (e.g. following redirects) are discarded.  

The implementation of URL-PATHNAME allows the ABCL user to laod dynamically
code from the network.  For example, for Quicklisp.

\begin{listing-lisp}
  CL-USER> (load "http://beta.quicklisp.org/quicklisp.lisp")
\end{listing-lisp}

will load and execute the Quicklisp setup code.

\ref{XACH2011}
         
\section{Extensible Sequences}

\ref{RHODES2007}

The SEQUENCE package fully implements Christopher Rhodes' proposal for
extensible sequences.  These user extensible sequences are used
directly in \code{java-collections.lisp} provide these CLOS
abstractions on the standard Java collection classes as defined by the
\code{java.util.List} contract.

This extension is not automatically loaded by the implementation.   It
may be loaded via:

\begin{listing-lisp}
CL-USER> (require 'extensible-sequences)
\end{listing-lisp}

\section{Extensions to CLOS}

There is an additional syntax for specializing the parameter of a
generic function on a java class, viz. \code{(java:jclass CLASS-STRING)}
where \code{CLASS-STRING} is a string naming a Java class in dotted package
form.

For instance the following specialization would perhaps allow one to
print more information about the contents of a java.util.Collection
object

\begin{listing-lisp}
(defmethod print-object ((coll (java:jclass "java.util.Collection"))
                         stream)
  ;;; ...
)
\end{listing-lisp}

If the class had been loaded via a classloader other than the original
the class you wish to specialize on, one needs to specify the
classloader as an optional third argument.

\begin{listing-lisp}

(defparameter *other-classloader*
  (jcall "getBaseLoader" cl-user::*classpath-manager*))
  
(defmethod print-object ((device-id (java:jclass "dto.nbi.service.hdm.alcatel.com.NBIDeviceID" *other-classloader*))
                         stream)
  ;;; ...
)
\end{listing-lisp}

\section{Extensions to the Reader}

We implement a special hexadecimal escape sequence for specifying
characters to the Lisp reader, namely we allow a sequences of the form
\# \textbackslash Uxxxx to be processed by the reader as character whose code is
specified by the hexadecimal digits ``xxxx''.  The hexadecimal sequence
must be exactly four digits long, padded by leading zeros for values
less than 0x1000.

Note that this sequence is never output by the implementation.  Instead,
the corresponding Unicode character is output for characters whose
code is greater than 0x00ff.

\section{ASDF}

asdf-2.017 is packaged as core component of ABCL.  By default, ASDF is
not loaded, as it relies on the CLOS subsystem which can take a bit of
time to initialize.

\begin{listing-lisp}
CL-USER> (require 'asdf)
\end{listing-lisp}

\chapter{Contrib}

\section{abcl-asdf} 

Allow ASDF system definition which dynamically loads JVM artifacts
such as jar archives via a Maven encapsulation.

ASDF components added:  JAR-FILE, JAR-DIRECTORY, CLASS-FILE-DIRECTORY
and MVN.

\section{asdf-install}

An implementation of ASDF-INSTALL.  Superceded by Quicklisp (qv.)

\section{asdf-jar}

ASDF-JAR provides a system for packaging ASDF systems into jar
archives for ABCL.  Given a running ABCL image with loadable ASDF
systems the code in this package will recursively package all the
required source and fasls in a jar archive.

\section{jss}

Java Syntax sucks, so we introduce the \#" macro.


\chapter{History}

ABCL was originally the extension language for the J editor, which was
started in 1998 by Peter Graves.  Sometime in 2003, it seems that a
lot of code that had previously not been released publically was
suddenly committed that enabled ABCL to be plausibly termed an ANSI
Common Lisp implementation.  

In 2006, the implementation was transferred to the current
maintainers, who have strived to improve its usability as a
contemporary Common Lisp implementation.

In 201x, with the publication of this Manual explicitly stating the
conformance of Armed Bear Common Lisp to ANSI, we release abcl-1.0.




\section{References}

[Java2000]:  A New Era for Java Protocol Handlers.
\url{http://java.sun.com/developer/onlineTraining/protocolhandlers/}

[Xach2011]:  Quicklisp:  A system for quickly constructing Common Lisp
libraries.  \url{http://www.quicklisp.org/}


\end{document}

% TODO
%   1.  Create mechanism for swigging DocString and Lisp docs into
%       sections.