CFFI foreign-alloc breaks abcl

[charmon]

After r14858 the cffi foreign allocation (and maybe other things) break.

To reproduce, run the CFFI-TESTS from Quicklisp

(ql:quickload :cffi)
(asdf:test-system :cffi)

[Mark Evenson]

Failures on abcl-1.4.0-dev "RC" <​http://paste.lisp.org/display/326806>

[Mark Evenson]

@pipping triaged this as happening from <​http://abcl.org/trac/changeset/14857>, a patch applied by @easye from @olof patchset on <mailto:armedbear-devel@…> c.f. <​https://mailman.common-lisp.net/pipermail/armedbear-devel/2016-August/003696.html>

[Mark Evenson]

Mark Evenson bisected the changeset to <​http://abcl.org/trac/changeset/14853>. He ran a complete deletion of all ABCL build artifacts, as well as clearing the application fasls cache.

The first bad revision is:
changeset:   2613:671ae612fd4b
user:        mevenson@1c010e3e-69d0-11dd-93a8-456734b0d56f
date:        Fri Sep 02 21:31:09 2016 +0000
summary:     Consolidated fixes for EXTENSIONS:RUN-PROGRAM (ferada)

[Mark Evenson]

Previous triage was incorrect: the change that breaks CFFI is r14858

[Mark Evenson]

CFFI-TESTS failures have been reproduced on

"Java_HotSpot(TM)_64-Bit_Server_VM-Oracle_Corporation-1.8.0_102-b14" 
"x86_64-Mac_OS_X- 10.12"

"Java_HotSpot(TM)_64-Bit_Server_VM-Oracle_Corporation-1.8.0_60-b27"
"amd64-Linux-3.16.0-0.bpo.4-amd64"  

[Mark Evenson]

An example of errors I am seeing

<​http://paste.lisp.org/display/327834>

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000800fb2f60, pid=75411, tid=0x0000000000018b13
#
# JRE version: OpenJDK Runtime Environment (8.0_102-b14) (build 1.8.0_102-b14)
# Java VM: OpenJDK 64-Bit Server VM (25.102-b14 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# C  [libc.so.7+0x146f60]  short+0x60
#
# Core dump written. Default location: /usr/home/evenson/work/abcl/java.core
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  T H R E A D  ---------------

Current thread (0x00000008137fa800):  JavaThread "interpreter" [_thread_in_native, id=101139, stack(0x00007fffde8eb000,0x00007fffdeceb000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000000

Registers:
RAX=0x0000000000000000, RBX=0x00007fffdecdd440, RCX=0x00000008025bb8b0, RDX=0x00000008025bdc60
RSP=0x00007fffdecdd4b8, RBP=0x00007fffdecdd4e0, RSI=0x00000008a18063c7, RDI=0x0000000000000000
R8 =0x0000000000000001, R9 =0x00000008137fa800, R10=0x00000008a24d6920, R11=0x00000008a24d68d0
R12=0x0000000000000000, R13=0x00000008a24d6c04, R14=0x0000000000000001, R15=0x000000080120bcc0
RIP=0x0000000800fb2f60, EFLAGS=0x0000000000000005, ERR=0x0000000000000004
  TRAPNO=0x000000000000000c

Top of Stack: (sp=0x00007fffdecdd4b8)
0x00007fffdecdd4b8:   00000008a1802014 00007fffdecdd6f0
0x00007fffdecdd4c8:   00007fffdecdd4e8 0000000000000000
0x00007fffdecdd4d8:   0000000802f572e8 00007fffdecdd4f0
0x00007fffdecdd4e8:   00000008a24d674c 000000000000000a
0x00007fffdecdd4f8:   00007fffdecddf48 00007fffdecdd5c0
0x00007fffdecdd508:   00000008a24d5dad 0000000000000001
0x00007fffdecdd518:   00007fffdecddee8 00007fffdecdded0
0x00007fffdecdd528:   00007fffdecddf48 00000008a1801fe0
0x00007fffdecdd538:   00007fffdecdd640 00007fffdecdd440
0x00007fffdecdd548:   00007fffdecdd470 0000000000000001
0x00007fffdecdd558:   0000000000000001 0000000000000001
0x00007fffdecdd568:   00007fffdecdd5d0 00007fffdecdd4f0
0x00007fffdecdd578:   00007fffdecdd5f0 00007fff00000001
0x00007fffdecdd588:   0000000801dbf312 1347061bcb9b2f02
0x00007fffdecdd598:   0000000000000000 00000008135d9940
0x00007fffdecdd5a8:   0000000000000001 00007fffdecdd640
0x00007fffdecdd5b8:   00007fffdecdd5f0 00007fffdecddf30
0x00007fffdecdd5c8:   000000089d61a802 00007fffdecdd640
0x00007fffdecdd5d8:   000000080073b000 00007fffdecdd648
0x00007fffdecdd5e8:   0000000800622000 000000089d628e60
0x00007fffdecdd5f8:   000000080060bc32 00000008a11ada78
0x00007fffdecdd608:   000000080081f5a0 00007fffdecddb50
0x00007fffdecdd618:   0000000800606517 000000080259f178
0x00007fffdecdd628:   00000008a0bab540 0000000813fff000
0x00007fffdecdd638:   00000008a0bab540 0000000802f572e8
0x00007fffdecdd648:   00000008137fa800 00007fffdecddf30
0x00007fffdecdd658:   000000089d61a63f 00007fffdecdd618
0x00007fffdecdd668:   00007fffdecdded0 000000009d61d10f
0x00007fffdecdd678:   00007fffdecdd5d0 00007fffdecdd610
0x00007fffdecdd688:   0000000100622000 00007fffdecddf48
0x00007fffdecdd698:   000000089d628ec0 00007fffdecddfe8
0x00007fffdecdd6a8:   0000000000000000 00000008a1801fe0 

Instructions: (pc=0x0000000800fb2f60)
0x0000000800fb2f40:   48 83 c7 08 48 8b 56 08 48 83 c6 08 48 39 c2 75
0x0000000800fb2f50:   0f 4c 29 c2 48 f7 d0 48 21 c2 4c 85 ca 74 dd 90
0x0000000800fb2f60:   8a 07 48 ff c7 8a 16 48 ff c6 84 c0 74 04 38 c2
0x0000000800fb2f70:   74 ee 48 0f b6 c0 48 0f b6 d2 48 29 d0 c3 90 90 

Register to memory mapping:

RAX=0x0000000000000000 is an unknown value
RBX=0x00007fffdecdd440 is pointing into the stack for thread: 0x00000008137fa800
RCX=0x00000008025bb8b0: gHotSpotVMLongConstants+0xfc38 in /usr/local/openjdk8/jre/lib/amd64/server/libjvm.so at 0x0000000801800000
RDX=0x00000008025bdc60: gHotSpotVMLongConstants+0x11fe8 in /usr/local/openjdk8/jre/lib/amd64/server/libjvm.so at 0x0000000801800000
RSP=0x00007fffdecdd4b8 is pointing into the stack for thread: 0x00000008137fa800
RBP=0x00007fffdecdd4e0 is pointing into the stack for thread: 0x00000008137fa800
RSI=0x00000008a18063c7: _fini+0x15f in /usr/home/evenson/quicklisp/dists/quicklisp/software/cffi_0.17.1/tests/libtest.so at 0x00000008a1800000
RDI=0x0000000000000000 is an unknown value
R8 =0x0000000000000001 is an unknown value
R9 =0x00000008137fa800 is a thread
R10=0x00000008a24d6920: ffi_closure_unix64+0xb2 in /usr/local/lib/libffi.so.6 at 0x00000008a24d1000
R11=0x00000008a24d68d0: ffi_closure_unix64+0x62 in /usr/local/lib/libffi.so.6 at 0x00000008a24d1000
R12=0x0000000000000000 is an unknown value
R13=0x00000008a24d6c04: ffi_type_longdouble+0xe4 in /usr/local/lib/libffi.so.6 at 0x00000008a24d1000
R14=0x0000000000000001 is an unknown value
R15=0x000000080120bcc0: __stack_chk_guard+0 in /lib/libc.so.7 at 0x0000000800e6c000


Stack: [0x00007fffde8eb000,0x00007fffdeceb000],  sp=0x00007fffdecdd4b8,  free space=4041k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.7+0x146f60]  short+0x60
C  [libffi.so.6+0x574c]  float+0x4c
C  0x00007fffdecddf48

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  com.sun.jna.Native.invokeInt(JI[Ljava/lang/Object;)I+0
j  com.sun.jna.Function.invoke([Ljava/lang/Object;Ljava/lang/Class;Z)Ljava/lang/Object;+219
j  com.sun.jna.Function.invoke(Ljava/lang/reflect/Method;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Object;Ljava/util/Map;)Ljava/lang/Object;+249
j  com.sun.jna.Function.invoke(Ljava/lang/Class;[Ljava/lang/Object;Ljava/util/Map;)Ljava/lang/Object;+37
j  com.sun.jna.Function.invoke(Ljava/lang/Class;[Ljava/lang/Object;)Ljava/lang/Object;+7
j  sun.reflect.GeneratedMethodAccessor39.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+48
J 2517 C2 sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (10 bytes) @ 0x0000000804488f3c [0x0000000804488ee0+0x5c]
J 5168 C1 java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (62 bytes) @ 0x0000000804bbb0ac [0x0000000804bbac60+0x44c]
J 5165 C1 org.armedbear.lisp.Java.jcall(Lorg/armedbear/lisp/Primitive;[Lorg/armedbear/lisp/LispObject;Z)Lorg/armedbear/lisp/LispObject; (577 bytes) @ 0x0000000804bc275c [0x0000000804bbdc40+0x4b1c]
J 2997 C2 org.armedbear.lisp.Java$pf_jcall_raw.execute([Lorg/armedbear/lisp/LispObject;)Lorg/armedbear/lisp/LispObject; (7 bytes) @ 0x0000000803e36ba4 [0x0000000803e36b60+0x44]
J 2972 C2 org.armedbear.lisp.Primitive.execute(Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispObject;)Lorg/armedbear/lisp/LispObject; (34 bytes) @ 0x0000000803bf6834 [0x0000000803bf64e0+0x354]
J 1581 C2 org.armedbear.lisp.Symbol.execute(Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispObject;)Lorg/armedbear/lisp/LispObject; (47 bytes) @ 0x0000000803988678 [0x0000000803988620+0x58]
J 2466 C2 org.armedbear.lisp.LispThread.execute(Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispObject;)Lorg/armedbear/lisp/LispObject; (121 bytes) @ 0x00000008047c1fa0 [0x00000008047c18a0+0x700]
j  org.armedbear.lisp.cffi_abcl_tmpDGYBIDNA_60.execute(Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispObject;)Lorg/armedbear/lisp/LispObject;+301
 sa_mask[0]=11111111111111111111111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGQUIT: [libjvm.so+0x7c4440], sa_mask[0]=11111111111111111111111111111110, sa_flags=SA_RESTART|SA_SIGINFO

[…]

---------------  S Y S T E M  ---------------

OS:BSD
uname:FreeBSD 11.0-RC3 FreeBSD 11.0-RC3 #0 r305786: Wed Sep 14 02:19:25 UTC 2016     root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
rlimit: STACK 524288k, CORE infinity, NPROC 8499, NOFILE 116919, AS infinity
load average:3.92 2.95 2.15

CPU:total 3 (3 cores per cpu, 1 threads per core) family 6 model 60 stepping 3, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, clmul, lzcnt, tsc, tscinvbit

Memory: 4k page, physical 4157428k(335720k free), swap 4292870144k(1948108k free)

vm_info: OpenJDK 64-Bit Server VM (25.102-b14) for bsd-amd64 JRE (1.8.0_102-b14), built on Oct  6 2016 11:44:00 by "evenson" with gcc 4.2.1 Compatible FreeBSD Clang 3.8.0 (tags/RELEASE_380/final 262564)

time: Thu Oct  6 12:48:58 2016
elapsed time: 117 seconds (0d 0h 1m 57s)

[Mark Evenson]

Ticket retargeted after milestone closed

[Mark Evenson]

As Anton points out in <​https://mailman.common-lisp.net/pipermail/armedbear-devel/2016-October/003721.html>, cffi-0.18 fixes problems with ABCL's use of jna-4.2.2 with <​https://github.com/cffi/cffi/commit/4385468ff863d42bdb30108f0c3b9366c3fc2242>.

This explains why neither ferada nor pipping could reproduce my error, as they were presumably running a patched version of CFFI.

WORKAROUND
==========
Until the next Quicklisp updates to cffi-0.18, one may fix things for a given installation via

cd ~/quicklisp/local-projects && git clone -b v0.18.0 https://github.com/cffi/cffi

Ferada's patch to add support for JAVA:JNEW-RUNTIME-CLASS has been added as r14903