source: branches/streams/abcl/doc/design/pathnames/url-pathnames.markdown

URL Pathnames ABCL
==================

    Mark Evenson
    Created:  25 MAR 2010
    Modified: 21 JUN 2011

Notes towards an implementation of URL references to be contained in
Common Lisp `PATHNAME` objects within ABCL.


References
----------

RFC3986   Uniform Resource Identifier (URI): Generic Syntax


URL vs URI
----------

We use the term URL as shorthand in describing the URL Pathnames, even
though the corresponding encoding is more akin to a URI as described
in RFC3986.  


Goals
-----

1.  Use Common Lisp pathnames to refer to representations referenced
by a URL.

2.  The URL schemes supported shall include at least "http", and those
enabled by the URLStreamHandler extension mechanism.

3.  Use URL schemes that are understood by the java.net.URL object.

    Example of a Pathname specified by URL:
    
        #p"http://example.org/org/armedbear/systems/pgp.asd"
    
4.  MERGE-PATHNAMES 

        (merge-pathnames "url.asd"
            "http://example/org/armedbear/systems/pgp.asd")
        ==> "http://example/org/armedbear/systems/url.asd"

5.  PROBE-FILE returning the state of URL accesibility.

6.  TRUENAME "aliased" to PROBE-FILE signalling an error if the URL is
not accessible (see "Non-goal 1").

7.  DIRECTORY works for non-wildcards.

8.  URL pathname work as a valid argument for OPEN with :DIRECTION :INPUT.

9.  Enable the loading of ASDF2 systems referenced by a URL pathname.

10.  Pathnames constructed with the "file" scheme
(i.e. #p"file:/this/file") need to be properly URI encoded according
to RFC3986 or otherwise will signal FILE-ERROR.  

11.  The "file" scheme will continue to be represented by an
"ordinary" Pathname.  Thus, after construction of a URL Pathname with
the "file" scheme, the namestring of the resulting PATHNAME will no
longer contain the "file:" prefix.

12.  The "jar" scheme will continue to be represented by a jar
Pathname.


Non-goals 
---------

1.  We will not implement canonicalization of URL schemas (such as
following "http" redirects).

2.  DIRECTORY will not work for URL pathnames containing wildcards.


Implementation
--------------

A PATHNAME refering to a resource referenced by a URL is known as a
URL PATHNAME.

A URL PATHNAME always has a HOST component which is a proper list.
This list will be an property list (plist).  The property list
values must be character strings.

    :SCHEME
        Scheme of URI ("http", "ftp", "bundle", etc.)
    :AUTHORITY   
        Valid authority according to the URI scheme.  For "http" this
        could be "example.org:8080".
    :QUERY
        The query of the URI
    :FRAGMENT
        The fragment portion of the URI
        
The DIRECTORY, NAME and TYPE fields of the PATHNAME are used to form
the URI `path` according to the conventions of the UNIX filesystem
(i.e. '/' is the directory separator).  In a sense the HOST contains
the base URL, to which the `path` is a relative URL (although this
abstraction is violated somwhat by the storing of the QUERY and
FRAGMENT portions of the URI in the HOST component).

For the purposes of PATHNAME-MATCH-P, two URL pathnames may be said to
match if their HOST compoments are EQUAL, and all other components are
considered to match according to the existing rules for Pathnames.

A URL pathname must have a DEVICE whose value is NIL.

Upon creation, the presence of ".." and "." components in the
DIRECTORY are removed.  The DIRECTORY component, if present, is always
absolute.

The namestring of a URL pathname shall be formed by the usual
conventions of a URL.

A URL Pathname has type URL-PATHNAME, derived from PATHNAME.


URI Encoding 
------------

For dealing with URI Encoding (also known as [Percent Encoding]() we
adopt the following rules

[Percent Encoding]: http://en.wikipedia.org/wiki/Percent-encoding

1.  All pathname components are represented "as is" without escaping.

2.  Namestrings are suitably escaped if the Pathname is a URL-PATHNAME
    or a JAR-PATHNAME.

3.  Namestrings should all "round-trip":

    (when (typep p 'pathname)
       (equal (namestring p)
              (namestring (pathname p))))


Status
------

This design has been implemented.


History
-------

26 NOV 2010 Changed implemenation to use URI encodings for the "file"
  schemes including those nested with the "jar" scheme by like
  aka. "jar:file:/location/of/some.jar!/".

21 JUN 2011 Fixed implementation to properly handle URI encodings
  refering nested jar archive.