ABCL-Jar not working on Mac OS X

[Robert Goldman]

When I tried to package a system of mine, I got cryptic errors and an empty jar file.

Turning on debugging, I see this:

PRISM> (asdf-jar:package :abcl-prism)
Packaging ASDF definition of #<ASDF/SYSTEM:SYSTEM "abcl-prism">
Packaging contents in /var/tmp/abcl-prism-all.jar
  with recursive dependencies ITERATE,  JSS.
/Users/rpg/prismatic/code/quiksilver/abcl/package.lisp
 => abcl-prism/package.lisp
/Users/rpg/.cache/common-lisp/abcl-1.4.0-dev-fasl42-macosx-x64/Users/rpg/prismatic/code/quiksilver/abcl/package.abcl
 => Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
; Evaluation aborted on #<SIMPLE-ERROR {58717F69}>.
PRISM> 

I can't say for sure, but it looks suspiciously like the results of ASDF output translations might be causing ASDF-JAR to fail. Is there any chance that these pathnames are simply too long? Or contain characters that ABCL treats as illegal?

What's the appropriate solution for packaging up fasls so that I can load them from jars (I have no idea how else to package up Lisp so that it can be called from Java...)?

BTW, there is a long-standing, unresolved bug in the ABCL implementation of the ASDF bundling operations on Mac OSX, so that's not a solution. Thanks!

[Robert Goldman]

Tracing MAKE-PATHNAME shows an error, but I'm not sure what it is:

/Users/rpg/prismatic/code/quiksilver/abcl/package.lisp
 => abcl-prism/package.lisp  0: (MAKE-PATHNAME :DIRECTORY (:ABSOLUTE) :NAME NIL :TYPE NIL :VERSION NIL :DEFAULTS #P"/Users/rpg/prismatic/code/quiksilver/abcl/package.lisp")
  0: MAKE-PATHNAME returned #P"/"
  0: (MAKE-PATHNAME :HOST NIL :DEVICE :UNSPECIFIC :DIRECTORY (:ABSOLUTE :WILD-INFERIORS) :NAME :WILD :TYPE :WILD :VERSION NIL)
  0: MAKE-PATHNAME returned #P"/**/*.*"
  0: (MAKE-PATHNAME :HOST NIL :DEVICE NIL :DIRECTORY (:ABSOLUTE "Users" "rpg" ".cache" "common-lisp" "abcl-1.4.0-dev-fasl42-macosx-x64" "Users" "rpg" "prismatic" "code" "quiksilver" "abcl") :NAME "package" :TYPE "lisp" :VERSION NIL)
  0: MAKE-PATHNAME returned #P"/Users/rpg/.cache/common-lisp/abcl-1.4.0-dev-fasl42-macosx-x64/Users/rpg/prismatic/code/quiksilver/abcl/package.lisp"
  0: (MAKE-PATHNAME :DEFAULTS #P"/Users/rpg/.cache/common-lisp/abcl-1.4.0-dev-fasl42-macosx-x64/Users/rpg/prismatic/code/quiksilver/abcl/package.lisp" :TYPE "abcl")
  0: MAKE-PATHNAME returned #P"/Users/rpg/.cache/common-lisp/abcl-1.4.0-dev-fasl42-macosx-x64/Users/rpg/prismatic/code/quiksilver/abcl/package.abcl"
  0: (MAKE-PATHNAME :DEFAULTS #P"abcl-prism/package.lisp" :TYPE "abcl" :DIRECTORY ("abcl-prism"))
  0: MAKE-PATHNAME returned Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go

[Robert Goldman]

The following causes make-pathname to error out:

(make-pathname :defaults (make-pathname :directory "abcl-prism" :name "package" :type "lisp") :type "abcl" :directory '("abcl-prism"))

[Robert Goldman]

I have further narrowed the problem, I believe. I find the following in the code:

(output-entry 
 (make-pathname :defaults source-entry 
                :type "abcl"
                :directory (append root
                                   (rest (pathname-directory source-entry))))))

In my usage, root is NIL, and the call to REST rips the :RELATIVE qualifier off the pathname-directory of SOURCE-ENTRY. This leaves me with a ("abcl-prism") as the pathname directory component, which ABCL is rejecting as ill-formed.

I guess I don't see why this would ever work, if ROOT is left with the default NIL value.

[Mark Evenson]

I'm seeing problems as well, so this seems to a more general problem

What's your LISP-IMPLEMENTATION-VERSION? Mine is:

CL-USER> (lisp-implementation-version)

"1.4.0-dev"
"Java_HotSpot(TM)_64-Bit_Server_VM-Oracle_Corporation-1.8.0_11-b12"
"x86_64-Mac_OS_X-10.9.4"

[Mark Evenson]

Eduardo Bellino seems to have addressed this in r14717. Please try a new 1.4.0-dev to see if it solves your problem.

​http://abcl.org/trac/changeset/14717

[Robert Goldman]

I have just updated from SVN and rebuilt, and the asdf-jar packaging still fails for me, although it fails with a different error now.
Here's what I see in the slime repl:

luation aborted on #<FILE-ERROR {2B53822}>.
CL-USER> (asdf-jar:package "abcl-prism" :recursive nil)

Packaging ASDF definition of #<ASDF/SYSTEM:SYSTEM "abcl-prism">
Packaging contents in /var/tmp/abcl-prism.jar

And here's the debugger window contents:

Unsupported directory component "abcl-prism".
   [Condition of type FILE-ERROR]

Restarts:
 0: [RETRY] Retry SLIME REPL evaluation request.
 1: [*ABORT] Return to SLIME's top level.
 2: [ABORT] Abort thread.

Backtrace:
  0: (#<FUNCTION {2D5FD903}> #<FILE-ERROR {331A2257}> #<FUNCTION {2D5FD903}>)
  1: (APPLY #<FUNCTION {2D5FD903}> (#<FILE-ERROR {331A2257}> #<FUNCTION {2D5FD903}>))
  2: (SYSTEM::RUN-HOOK SYSTEM::*INVOKE-DEBUGGER-HOOK* #<FILE-ERROR {331A2257}> #<FUNCTION {2D5FD903}>)
  3: (INVOKE-DEBUGGER #<FILE-ERROR {331A2257}>)
  4: org.armedbear.lisp.Lisp.error(Lisp.java:382)
  5: org.armedbear.lisp.Pathname.getDirectoryNamestring(Pathname.java:808)
  6: org.armedbear.lisp.Pathname.getNamestring(Pathname.java:700)
  7: org.armedbear.lisp.zip.execute(zip.java:220)
  8: org.armedbear.lisp.zip.execute(zip.java:68)
  9: org.armedbear.lisp.Symbol.execute(Symbol.java:814)
 10: org.armedbear.lisp.LispThread.execute(LispThread.java:832)
 11: org.armedbear.lisp.asdf_jar_TMP_1.execute(asdf-jar.lisp:15)
 12: org.armedbear.lisp.CompiledClosure.execute(CompiledClosure.java:121)
 13: org.armedbear.lisp.LispThread.execute(LispThread.java:851)
 14: org.armedbear.lisp.Lisp.evalCall(Lisp.java:589)
 15: org.armedbear.lisp.Lisp.eval(Lisp.java:540)
 16: org.armedbear.lisp.Primitives$pf__eval.execute(Primitives.java:345)
 17: (SYSTEM:ZIP #P"/var/tmp/abcl-prism.jar" #<EQUAL HASH-TABLE 7 entries, 11 buckets {78C497F6}>)
 18: (PACKAGE "abcl-prism" :RECURSIVE NIL)
 19: (SYSTEM::%EVAL (PACKAGE "abcl-prism" :RECURSIVE NIL))
 --more--

The hashtable contents are as follows:

#<HASH-TABLE {78C497F6}>
--------------------
Count: 7
Size: 11
Test: EQUAL
Rehash size: 1.5
Rehash threshold: 0.75
[clear hashtable]
Contents: 
#P"/Users/rpg/prismatic/code/quiksilver/abcl/abcl-prism.asd" = #P"abcl-prism/abcl-prism.asd" [remove entry]
#P"/Users/rpg/.cache/common-lisp/abcl-1.4.0-dev-fasl42-macosx-x64/Users/rpg/prismatic/code/quiksilver/abcl/package.abcl" = #<PATHNAME <<error printing object>> {33A72123}> [remove entry]
#P"/Users/rpg/.cache/common-lisp/abcl-1.4.0-dev-fasl42-macosx-x64/Users/rpg/prismatic/code/quiksilver/abcl/utils.abcl" = #<PATHNAME <<error printing object>> {21709B53}> [remove entry]
#P"/Users/rpg/prismatic/code/quiksilver/abcl/package.lisp" = #P"abcl-prism/package.lisp" [remove entry]
#P"/Users/rpg/prismatic/code/quiksilver/abcl/utils.lisp" = #P"abcl-prism/utils.lisp" [remove entry]
#P"/Users/rpg/prismatic/code/quiksilver/abcl/add-script.lisp" = #P"abcl-prism/add-script.lisp" [remove entry]
#P"/Users/rpg/.cache/common-lisp/abcl-1.4.0-dev-fasl42-macosx-x64/Users/rpg/prismatic/code/quiksilver/abcl/add-script.abcl" = #<PATHNAME <<error printing object>> {67E5774B}> [remove entry]

I note that the mapped pathnames for all of the compiled files gives an error on printing, although the source files are all ok.

Unfortunately, when I attempt to do anything with the bad pathnames, even describe, it causes a crash in some print method. However, I have determined that the pathname-directory returned from one of these is "abcl-prism", and the pathname-directory of a working pathname is (:relative "abcl-prism"), so I believe there's still a mapping bug.

[Robert Goldman]

When I look at this further, I see that the logic for mapping the compiled files has not materially changed. The root entry is still appended, even if it is null, to the second entry in the pathname-directory of the source-entry. I am not sure why the second element alone is correct, but it's clear that it won't be either :relative or :absolute, so the mapped directory will still be incorrect if the root is NIL.

Question: if the root is NIL, should we simply take the first element of the source-mapping in its place?

It would really help to add a comment or two to the logic of the output-entry binding.

[Robert Goldman]

Is there some reason that the source-entry is translated into the output-entry by taking all of it as defaults, rather than just the name? I'm wondering if the directory components are combining in some anomalous way when computing the output-entry.

Again, if there was an explanation of what it was intended to compute, this would be easier to debug.

Thanks!

[Robert Goldman]

Put more clearly, are we trying to put the .abcl files next to the source files, or are we trying to put them in some structure that mirrors the structure of the ASDF cache according to Fare's output translations logic?

If it's the former, I'm stumped by the CADR invocation.

[Mark Evenson]

For a fallback strategy we do associate each file of type "lisp" with an "abcl" FASL, but this is just an optimization, probably no longer necessary in most cases.

Otherwise, we are mapping from the location of source pathnames on the current filesystem to destination pathnames in the jar artifact. After which ASDF compiles the files, and everything works. The one assumption is that ASDF packages in the location of source artifacts as relative pathnames that are descendents of where the "asd" file is located, as cases otherwise can cause some funky behavior. The current state of ASDF system definition actually declaring all their source artifacts is depressingly low in a sampling of systems distributed via Quicklisp. For recent work, I just rsynced everything below the "asd" file to make sure that things would work.

[Robert Goldman]

Sorry for the noise. This was entirely my bad. I ended up with a stale abcl-contrib.jar lying around in my source path, which blocked seeing the good code.

[Robert Goldman]

One last question if you don't mind: there seems like a Catch-22 about using these jar files. You have to add them to the ASDF path using the asdf-jar contrib, but how do you know where they are? Seems like the asdf-contrib.jar has a cleaner solution where you can simply (require :asdf-contrib), and all the systems in the jar file can become loadable. Is there something like this one can do with a jar file make with ASDF-JAR?

[Mark Evenson]

Replying to robert goldman:

One last question if you don't mind: there seems like a Catch-22 about using these jar files. You have to add them to the ASDF path using the asdf-jar contrib, but how do you know where they are? Seems like the asdf-contrib.jar has a cleaner solution where you can simply (require :asdf-contrib), and all the systems in the jar file can become loadable. Is there something like this one can do with a jar file make with ASDF-JAR?

The jar is 'abcl-contrib.jar' located via (require :abcl-contrib). This is considered a "system jar" to be distributed with 'abcl.jar', which would usually be in a location not directly writable by the user.

One possibility would be if you are looking to distribute a standalone application, simply adding such packages to abcl-contrib.jar.

But for locating ASDF systems in general, I suppose that the right way forward would be to use the ASDF configuration DSL to provide the location of your systems, which just happen to be packaged in jar files. ABCL uses a EXT:JAR-PATHNAME to represent such locations which is an extension of CL:PATHNAME, so if we were to ensure that the ASDF configuration DSL can deal with such extensions, things might be able to work. Now if only I could find an ASDF maintainer to talk to about this…

[Robert Goldman]

Let me start a new ticket for this, since I have dragged us way off topic.