Ok, Thiemo Seufer has committed a change to gc-common.c that gets rid of an extra instruction cache flush on non GENCGC platforms (that is, CHENEYGC). Turns out this is a no-op on x86 and x86-64 due to their cache-coherency, but might be required for a generational collector on other platforms should one ever get ported to, say, PPC or MIPS.

As for the additional sync call, this may be necessary on multi-processor systems as the instructions seen in the icache by one processor may be seen as data in another and the data-cache syncing is necessary. So this should stay as is.

Here are some cl-bench results from recent SBCL builds on PPC with a couple minor caching changes. The first is the vanilla SBCL, the second is with an os_icache_flush removed from the cheneygc and the second is with this change and a sync removed from ppc_flush_cache_line. Here's the patch:

 
===================================================================
RCS file: /cvsroot/sbcl/sbcl/src/runtime/gc-common.c,v
retrieving revision 1.33
diff -u -r1.33 gc-common.c
--- src/runtime/gc-common.c     5 Dec 2005 18:01:29 -0000       1.33
+++ src/runtime/gc-common.c     10 Dec 2005 21:47:35 -0000
@@ -314,8 +314,10 @@
         fheaderl = fheaderp->next;
         prev_pointer = &nfheaderp->next;
     }
+#ifdef LISP_FEATURE_GENCGC
     os_flush_icache((os_vm_address_t) (((long *)new_code) + nheader_words),
                     ncode_words * sizeof(long));
+#endif
 #ifdef LISP_FEATURE_GENCGC
     gencgc_apply_code_fixups(code, new_code);
 #endif
Index: src/runtime/ppc-assem.S
===================================================================
RCS file: /cvsroot/sbcl/sbcl/src/runtime/ppc-assem.S,v
retrieving revision 1.7
diff -u -r1.7 ppc-assem.S
--- src/runtime/ppc-assem.S     23 Oct 2005 19:29:00 -0000      1.7
+++ src/runtime/ppc-assem.S     10 Dec 2005 21:47:35 -0000
@@ -574,7 +574,6 @@
        dcbf 0,REG(3)
        sync
        icbi 0,REG(3)
-       sync
        isync
        blr
        SET_SIZE(ppc_flush_cache_line)

 

(I've posted this to the cclan list, but I thought I'd put a copy of this here for posterity's sake.) So I'm trying to package up some C code to use as a test for gcc-xml-ffi. I have some C code that I'd like to use to generate a shared library, which will then be loaded by SBCL at runtime. To do this I can 1) hand-code the Makefile rules with the approrpriate platform-specific compiler/linker incantations to get the shared library, 2) use autotools, or 3) try to rig up some sort of asdf-based system to do this. I've done 1 for darwin and got 2 to work, but I wasn't super happy with distributing this giant mess of m4 files just to compile some shared libraries, so I decided to explore 3. Actually, as I was considering this, I decided that I also wanted to use asdf to invoke gccxml to generate the xml for gcc-xml-ffi.

So far so good. SBCL has examples of how to do this in sb-posix (and somewhere else, IIRC this is a violation of OAOO, but let's put that aside for the moment.

But I've run into a couple issues:

Clearly, asdf is an extensible system and through the right extensions I can make this work, but if anyone has any suggestions or comments on how to address these issues, I'd appreciate it.

Ok, now new functionality in here, other than the package mechanism, which is in fact in the chutil package. In any event, here are new versions of chutil and meta-ffi. Enjoy.

• chutil
• meta-ffi

Not that would want to, but if you want to pacakge up a distribution, typing make dist should do the right thing.

whiskey tango foxtrot indeed... thanks for the feedback Xach!

It seems the good people of #lisp aren't a big fan of i) autotools, ii) m4, and iii) needlessly large distributions. Addressing these three complaints in one fell swoop, I present new versions of chutil and meta-ffi.

• chutil
• meta-ffi

Ok, I'm finally getting my lazy act together and trying to package some of the software I've been working on for release. The first two packages are rather trivial, chutil and meta-ffi. They don't do much. chutil is some relatively trivial utilities and meta-ffi is a MOP metaclass for doing FFI to foreign structs. meta-ffi allows for making a :metaclass uffi-foreign-struct-class and then define slots with :foreign-field class. It's pretty trivial but a nice way for me to learn how to make a metaclass. There's no examples or documentation, but I hope to fix that in the near future.

• chutil
• meta-ffi

My knowledge of unix memory management is fairly limited, but i was unpleasantly surprised to discover that the following:

(defparameter a1 (make-array '(1024 1024 128) :element-type '(unsigned-byte 8)))
(gc)

will cause SBCL to rather unceremoniously exit. Apparpently, there are some compiler settings that determine the runtime memory layout and the gc sections only get 128MB each. I need to do some homework to figure out how to address this. I'd like to be able to use SBCL for image processing but, images from the d2x are 75MB each and I'd like to have more than one in memory at a time (or at least more than one representation of an image - especially a floating point representation and 16-bit per channel per pixel representation.

Ok, so it looks like the kewl kids on #lisp don't want Xach to syndicate my blog on planet.sbcl.org as that's reserved for output from CVS commit logs, chandler/chavatar's sbcl bulid-tester (very cool, BTW), and comments from ranking SBCL developers. I still don't want all of my drivel going to planet.lisp.org, which means it's relegated to the SBCL category of my blog, which means nobody will ever see it, but that's fine with me, at least for the moment.

Ok, on to my next SBCL problem. Profiling fails on PPC. I sent a bug to sbcl-devel, but so far the interest in learning more about the problem or fixing it seems rather low. I guess nobody else is trying to profile SBCL code on PPC at the moment. Changing *timer-overhead-iterations* from 500000 to 100000 fixes sb-profile:report, at least in the trivial case. This is helpful, but I'm not quite sure what the 500000 was for in the first place, or, more to the point, why bumping this down to 100000 fixes the problem.

I still don't understand how or why we get a large negative number instead of an unsigned-byte. Clearly more investigative work is required here.

Now that Xach has set up planet.sbcl.org, I'll move my gripes, helfpful (yeah, right...) comments, opinions, etc... about SBCL to the new SBCL category of my blog and maybe Xach will be kind enough ot pick up the feed.

As a first entry, let me continue to try to convince the SBCL gurus on #lisp that changing the size of compact-info-entries-index is a good thing.

As I have mentioned before, I've been trying to generate FFI definitions for the MacOS Carbon APIs and have run into a problem attempting to run purify, which gets called automatically by save-lisp-and-die. (I'm ignoring the fact that without threads and callbacks on the PPC, the carbon integration is going to be lacking but Brian Mastenbrook has been working on callbacks and hopefully I, or better yet, somebody who knows what they're doing, will have a chance to take a look at threads and the required gencgc stuff one of these days. Now back to the original thread).

The call to purify fails, giving an error saying that 65536 not being an (integer 0 65535) or something similar. Purify (and perhaps other things?) attempts to create a compact environment that contains the same information as an existing environment in a more compact representation. There is a variable compact-info-env-entries-bits that is used to determine the size (in bits) of the compact-info-env-entries-index type, which, in turn, is used as the type of the compact-info-inv/cache-index and as the type of the simple-array compact-info-env/index.

As an environment gets more than 65535 entries, it becomes impossible to compact it do tue the size of the index into the underlying data structures. So the obvious thing to try is to boost the size of the index. I made compact-info-env-entries-bits 32 and have been using this for a few weeks (on sbcl 0.8.20.xx on PPC) with no problems.

Enough background. Now for some specifics:

Why is this necessary? Without this patch one can only have 65535 functions in an environment.

Here's a testcase that illustrates the problem:

(defparameter l nil)                                                                                             
(dotimes (i 65537)                                                                                               
  (setf l (cons (let ((g (gensym)) (z i))                                                                        
                  (setf (symbol-function g) #'(lambda () z)) g) l)))                                             
(purify)                                                                                                         

And the output is:

The value 65536 is not of type (UNSIGNED-BYTE 16).                                                               
   [Condition of type TYPE-ERROR]                                                                                
                                                                                                                

What are the impacts of the change? The compact-info-entries-index type itself is only used by the index and cache-index fields of the compact-info-env struct. These data structures hold indices into a table of the particular things being stored in this compact-info-env. Note that this change doesn't change the size of the things in the table, just the size of the index and the cached index into the table, AFAICT.

There is one potential problem. At some point a table-size gets calculated based on the number of names in the environment. We call primify to get a prime number size for the table. primify declares its argument x as an (unsigned-byte x) and then iterates over the odd numbers >= x until it finds a prime number. But positive-prime declares its argument to be a fixnum, so it's possible that we could have a table size that is > most-positive-fixnum which cause positive-primep to fail. I haven't run into this in practice, but we might want to watch it for this and/or come up with a better approach for getting the size of the table. There's a comment about using almost-primify from hash-table.lisp, but this must be an out-of-date comment, as I can't find this function in hash-table.lisp. But failing when we get larger than most-positive-fixnum is much better than failing when we hit 65536 entries in a compact info env.

I'm getting tired of flogging this horse, but I think it's important that this get fixed. If anyone disagrees with what I've proposed, I'd love to hear it.

Index: globaldb.lisp
===================================================================
RCS file: /cvsroot/sbcl/sbcl/src/compiler/globaldb.lisp,v
retrieving revision 1.36
diff -u -r1.36 globaldb.lisp
--- globaldb.lisp       12 Jun 2004 13:55:49 -0000      1.36
+++ globaldb.lisp       26 Mar 2005 15:03:13 -0000
@@ -476,7 +476,7 @@
 ;;;; compact info environments

 ;;; The upper limit on the size of the ENTRIES vector in a COMPACT-INFO-ENV.
-(def!constant compact-info-env-entries-bits 16)
+(def!constant compact-info-env-entries-bits 32)
 (deftype compact-info-entries-index () `(unsigned-byte ,compact-info-env-entries-bits))

 ;;; the type of the values in COMPACT-INFO-ENTRIES-INFO