Opened 7 years ago

Closed 7 years ago

Last modified 6 years ago

#365 closed defect (fixed)

ABCL-Jar not working on Mac OS X

Reported by: Robert Goldman Owned by: Mark Evenson
Priority: major Milestone: 1.3.2
Component: abcl-contrib Version: 1.4.0-dev
Keywords: Cc:
Parent Tickets:

Description

When I tried to package a system of mine, I got cryptic errors and an empty jar file.

Turning on debugging, I see this:

PRISM> (asdf-jar:package :abcl-prism)
Packaging ASDF definition of #<ASDF/SYSTEM:SYSTEM "abcl-prism">
Packaging contents in /var/tmp/abcl-prism-all.jar
  with recursive dependencies ITERATE,  JSS.
/Users/rpg/prismatic/code/quiksilver/abcl/package.lisp
 => abcl-prism/package.lisp
/Users/rpg/.cache/common-lisp/abcl-1.4.0-dev-fasl42-macosx-x64/Users/rpg/prismatic/code/quiksilver/abcl/package.abcl
 => Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
; Evaluation aborted on #<SIMPLE-ERROR {58717F69}>.
PRISM> 

I can't say for sure, but it looks suspiciously like the results of ASDF output translations might be causing ASDF-JAR to fail. Is there any chance that these pathnames are simply too long? Or contain characters that ABCL treats as illegal?

What's the appropriate solution for packaging up fasls so that I can load them from jars (I have no idea how else to package up Lisp so that it can be called from Java...)?

BTW, there is a long-standing, unresolved bug in the ABCL implementation of the ASDF bundling operations on Mac OSX, so that's not a solution. Thanks!

Change History (17)

comment:1 Changed 7 years ago by Robert Goldman

Tracing MAKE-PATHNAME shows an error, but I'm not sure what it is:

/Users/rpg/prismatic/code/quiksilver/abcl/package.lisp
 => abcl-prism/package.lisp  0: (MAKE-PATHNAME :DIRECTORY (:ABSOLUTE) :NAME NIL :TYPE NIL :VERSION NIL :DEFAULTS #P"/Users/rpg/prismatic/code/quiksilver/abcl/package.lisp")
  0: MAKE-PATHNAME returned #P"/"
  0: (MAKE-PATHNAME :HOST NIL :DEVICE :UNSPECIFIC :DIRECTORY (:ABSOLUTE :WILD-INFERIORS) :NAME :WILD :TYPE :WILD :VERSION NIL)
  0: MAKE-PATHNAME returned #P"/**/*.*"
  0: (MAKE-PATHNAME :HOST NIL :DEVICE NIL :DIRECTORY (:ABSOLUTE "Users" "rpg" ".cache" "common-lisp" "abcl-1.4.0-dev-fasl42-macosx-x64" "Users" "rpg" "prismatic" "code" "quiksilver" "abcl") :NAME "package" :TYPE "lisp" :VERSION NIL)
  0: MAKE-PATHNAME returned #P"/Users/rpg/.cache/common-lisp/abcl-1.4.0-dev-fasl42-macosx-x64/Users/rpg/prismatic/code/quiksilver/abcl/package.lisp"
  0: (MAKE-PATHNAME :DEFAULTS #P"/Users/rpg/.cache/common-lisp/abcl-1.4.0-dev-fasl42-macosx-x64/Users/rpg/prismatic/code/quiksilver/abcl/package.lisp" :TYPE "abcl")
  0: MAKE-PATHNAME returned #P"/Users/rpg/.cache/common-lisp/abcl-1.4.0-dev-fasl42-macosx-x64/Users/rpg/prismatic/code/quiksilver/abcl/package.abcl"
  0: (MAKE-PATHNAME :DEFAULTS #P"abcl-prism/package.lisp" :TYPE "abcl" :DIRECTORY ("abcl-prism"))
  0: MAKE-PATHNAME returned Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go
Serious printing error: 
org.armedbear.lisp.Go

comment:2 Changed 7 years ago by Robert Goldman

The following causes make-pathname to error out:

(make-pathname :defaults (make-pathname :directory "abcl-prism" :name "package" :type "lisp") :type "abcl" :directory '("abcl-prism"))

comment:3 Changed 7 years ago by Robert Goldman

I have further narrowed the problem, I believe. I find the following in the code:

(output-entry 
 (make-pathname :defaults source-entry 
                :type "abcl"
                :directory (append root
                                   (rest (pathname-directory source-entry))))))

In my usage, root is NIL, and the call to REST rips the :RELATIVE qualifier off the pathname-directory of SOURCE-ENTRY. This leaves me with a ("abcl-prism") as the pathname directory component, which ABCL is rejecting as ill-formed.

I guess I don't see why this would ever work, if ROOT is left with the default NIL value.

comment:4 Changed 7 years ago by Mark Evenson

I'm seeing problems as well, so this seems to a more general problem

What's your LISP-IMPLEMENTATION-VERSION? Mine is:

CL-USER> (lisp-implementation-version)

"1.4.0-dev"
"Java_HotSpot(TM)_64-Bit_Server_VM-Oracle_Corporation-1.8.0_11-b12"
"x86_64-Mac_OS_X-10.9.4"

comment:5 Changed 7 years ago by Mark Evenson

Owner: set to Mark Evenson
Status: newaccepted

comment:6 Changed 7 years ago by Mark Evenson

Resolution: fixed
Status: acceptedclosed

Eduardo Bellino seems to have addressed this in r14717. Please try a new 1.4.0-dev to see if it solves your problem.

http://abcl.org/trac/changeset/14717

comment:7 Changed 7 years ago by Mark Evenson

Milestone: 1.4.0
Version: 1.4.0-dev

comment:8 Changed 7 years ago by Robert Goldman

Resolution: fixed
Status: closedreopened

I have just updated from SVN and rebuilt, and the asdf-jar packaging still fails for me, although it fails with a different error now.
Here's what I see in the slime repl:

luation aborted on #<FILE-ERROR {2B53822}>.
CL-USER> (asdf-jar:package "abcl-prism" :recursive nil)

Packaging ASDF definition of #<ASDF/SYSTEM:SYSTEM "abcl-prism">
Packaging contents in /var/tmp/abcl-prism.jar

And here's the debugger window contents:

Unsupported directory component "abcl-prism".
   [Condition of type FILE-ERROR]

Restarts:
 0: [RETRY] Retry SLIME REPL evaluation request.
 1: [*ABORT] Return to SLIME's top level.
 2: [ABORT] Abort thread.

Backtrace:
  0: (#<FUNCTION {2D5FD903}> #<FILE-ERROR {331A2257}> #<FUNCTION {2D5FD903}>)
  1: (APPLY #<FUNCTION {2D5FD903}> (#<FILE-ERROR {331A2257}> #<FUNCTION {2D5FD903}>))
  2: (SYSTEM::RUN-HOOK SYSTEM::*INVOKE-DEBUGGER-HOOK* #<FILE-ERROR {331A2257}> #<FUNCTION {2D5FD903}>)
  3: (INVOKE-DEBUGGER #<FILE-ERROR {331A2257}>)
  4: org.armedbear.lisp.Lisp.error(Lisp.java:382)
  5: org.armedbear.lisp.Pathname.getDirectoryNamestring(Pathname.java:808)
  6: org.armedbear.lisp.Pathname.getNamestring(Pathname.java:700)
  7: org.armedbear.lisp.zip.execute(zip.java:220)
  8: org.armedbear.lisp.zip.execute(zip.java:68)
  9: org.armedbear.lisp.Symbol.execute(Symbol.java:814)
 10: org.armedbear.lisp.LispThread.execute(LispThread.java:832)
 11: org.armedbear.lisp.asdf_jar_TMP_1.execute(asdf-jar.lisp:15)
 12: org.armedbear.lisp.CompiledClosure.execute(CompiledClosure.java:121)
 13: org.armedbear.lisp.LispThread.execute(LispThread.java:851)
 14: org.armedbear.lisp.Lisp.evalCall(Lisp.java:589)
 15: org.armedbear.lisp.Lisp.eval(Lisp.java:540)
 16: org.armedbear.lisp.Primitives$pf__eval.execute(Primitives.java:345)
 17: (SYSTEM:ZIP #P"/var/tmp/abcl-prism.jar" #<EQUAL HASH-TABLE 7 entries, 11 buckets {78C497F6}>)
 18: (PACKAGE "abcl-prism" :RECURSIVE NIL)
 19: (SYSTEM::%EVAL (PACKAGE "abcl-prism" :RECURSIVE NIL))
 --more--

The hashtable contents are as follows:

#<HASH-TABLE {78C497F6}>
--------------------
Count: 7
Size: 11
Test: EQUAL
Rehash size: 1.5
Rehash threshold: 0.75
[clear hashtable]
Contents: 
#P"/Users/rpg/prismatic/code/quiksilver/abcl/abcl-prism.asd" = #P"abcl-prism/abcl-prism.asd" [remove entry]
#P"/Users/rpg/.cache/common-lisp/abcl-1.4.0-dev-fasl42-macosx-x64/Users/rpg/prismatic/code/quiksilver/abcl/package.abcl" = #<PATHNAME <<error printing object>> {33A72123}> [remove entry]
#P"/Users/rpg/.cache/common-lisp/abcl-1.4.0-dev-fasl42-macosx-x64/Users/rpg/prismatic/code/quiksilver/abcl/utils.abcl" = #<PATHNAME <<error printing object>> {21709B53}> [remove entry]
#P"/Users/rpg/prismatic/code/quiksilver/abcl/package.lisp" = #P"abcl-prism/package.lisp" [remove entry]
#P"/Users/rpg/prismatic/code/quiksilver/abcl/utils.lisp" = #P"abcl-prism/utils.lisp" [remove entry]
#P"/Users/rpg/prismatic/code/quiksilver/abcl/add-script.lisp" = #P"abcl-prism/add-script.lisp" [remove entry]
#P"/Users/rpg/.cache/common-lisp/abcl-1.4.0-dev-fasl42-macosx-x64/Users/rpg/prismatic/code/quiksilver/abcl/add-script.abcl" = #<PATHNAME <<error printing object>> {67E5774B}> [remove entry]

I note that the mapped pathnames for all of the compiled files gives an error on printing, although the source files are all ok.

Unfortunately, when I attempt to do anything with the bad pathnames, even describe, it causes a crash in some print method. However, I have determined that the pathname-directory returned from one of these is "abcl-prism", and the pathname-directory of a working pathname is (:relative "abcl-prism"), so I believe there's still a mapping bug.

comment:9 Changed 7 years ago by Robert Goldman

When I look at this further, I see that the logic for mapping the compiled files has not materially changed. The root entry is still appended, even if it is null, to the second entry in the pathname-directory of the source-entry. I am not sure why the second element alone is correct, but it's clear that it won't be either :relative or :absolute, so the mapped directory will still be incorrect if the root is NIL.

Question: if the root is NIL, should we simply take the first element of the source-mapping in its place?

It would really help to add a comment or two to the logic of the output-entry binding.

comment:10 Changed 7 years ago by Robert Goldman

Is there some reason that the source-entry is translated into the output-entry by taking all of it as defaults, rather than just the name? I'm wondering if the directory components are combining in some anomalous way when computing the output-entry.

Again, if there was an explanation of what it was intended to compute, this would be easier to debug.

Thanks!

comment:11 Changed 7 years ago by Robert Goldman

Put more clearly, are we trying to put the .abcl files next to the source files, or are we trying to put them in some structure that mirrors the structure of the ASDF cache according to Fare's output translations logic?

If it's the former, I'm stumped by the CADR invocation.

comment:12 Changed 7 years ago by Mark Evenson

For a fallback strategy we do associate each file of type "lisp" with an "abcl" FASL, but this is just an optimization, probably no longer necessary in most cases.

Otherwise, we are mapping from the location of source pathnames on the current filesystem to destination pathnames in the jar artifact. After which ASDF compiles the files, and everything works. The one assumption is that ASDF packages in the location of source artifacts as relative pathnames that are descendents of where the "asd" file is located, as cases otherwise can cause some funky behavior. The current state of ASDF system definition actually declaring all their source artifacts is depressingly low in a sampling of systems distributed via Quicklisp. For recent work, I just rsynced everything below the "asd" file to make sure that things would work.

Last edited 7 years ago by Mark Evenson (previous) (diff)

comment:13 Changed 7 years ago by Robert Goldman

Resolution: fixed
Status: reopenedclosed

Sorry for the noise. This was entirely my bad. I ended up with a stale abcl-contrib.jar lying around in my source path, which blocked seeing the good code.

comment:14 Changed 7 years ago by Robert Goldman

One last question if you don't mind: there seems like a Catch-22 about using these jar files. You have to add them to the ASDF path using the asdf-jar contrib, but how do you know where they are? Seems like the asdf-contrib.jar has a cleaner solution where you can simply (require :asdf-contrib), and all the systems in the jar file can become loadable. Is there something like this one can do with a jar file make with ASDF-JAR?

Last edited 7 years ago by Robert Goldman (previous) (diff)

comment:15 in reply to:  14 Changed 7 years ago by Mark Evenson

Replying to robert goldman:

One last question if you don't mind: there seems like a Catch-22 about using these jar files. You have to add them to the ASDF path using the asdf-jar contrib, but how do you know where they are? Seems like the asdf-contrib.jar has a cleaner solution where you can simply (require :asdf-contrib), and all the systems in the jar file can become loadable. Is there something like this one can do with a jar file make with ASDF-JAR?

The jar is 'abcl-contrib.jar' located via (require :abcl-contrib). This is considered a "system jar" to be distributed with 'abcl.jar', which would usually be in a location not directly writable by the user.

One possibility would be if you are looking to distribute a standalone application, simply adding such packages to abcl-contrib.jar.

But for locating ASDF systems in general, I suppose that the right way forward would be to use the ASDF configuration DSL to provide the location of your systems, which just happen to be packaged in jar files. ABCL uses a EXT:JAR-PATHNAME to represent such locations which is an extension of CL:PATHNAME, so if we were to ensure that the ASDF configuration DSL can deal with such extensions, things might be able to work. Now if only I could find an ASDF maintainer to talk to about this…

comment:16 Changed 7 years ago by Robert Goldman

Let me start a new ticket for this, since I have dragged us way off topic.

comment:17 Changed 6 years ago by Mark Evenson

Milestone: 1.4.01.3.2
Note: See TracTickets for help on using tickets.