Thu, 31 Mar 2005 18:48:34 GMT revised at: Mon, 16 Apr 2007 20:54:47 GMT
Now that Xach has set up planet.sbcl.org , I'll move my gripes, helfpful (yeah, right...) comments, opinions, etc... about SBCL to the new SBCL category of my blog and maybe Xach will be kind enough ot pick up the feed.
As a first entry, let me continue to try to convince the SBCL gurus on #lisp that changing the size of compact-info-entries-index is a good thing.
As I have mentioned before, I've been trying to generate FFI definitions for the MacOS Carbon APIs and have run into a problem attempting to run purify, which gets called automatically by save-lisp-and-die. (I'm ignoring the fact that without threads and callbacks on the PPC, the carbon integration is going to be lacking but Brian Mastenbrook has been working on callbacks and hopefully I, or better yet, somebody who knows what they're doing, will have a chance to take a look at threads and the required gencgc stuff one of these days. Now back to the original thread).
The call to purify fails, giving an error saying that 65536 not being an (integer 0 65535) or something similar. Purify (and perhaps other things?) attempts to create a compact environment that contains the same information as an existing environment in a more compact representation. There is a variable compact-info-env-entries-bits that is used to determine the size (in bits) of the compact-info-env-entries-index type, which, in turn, is used as the type of the compact-info-inv/cache-index and as the type of the simple-array compact-info-env/index.
As an environment gets more than 65535 entries, it becomes impossible to compact it do tue the size of the index into the underlying data structures. So the obvious thing to try is to boost the size of the index. I made compact-info-env-entries-bits 32 and have been using this for a few weeks (on sbcl 0.8.20.xx on PPC) with no problems.
Enough background. Now for some specifics:
Why is this necessary? Without this patch one can only have 65535 functions in an environment.
Here's a testcase that illustrates the problem:
(defparameter l nil)
(dotimes (i 65537)
(setf l (cons (let ((g (gensym)) (z i))
(setf (symbol-function g) #'(lambda () z)) g) l)))
And the output is:
The value 65536 is not of type (UNSIGNED-BYTE 16).
[Condition of type TYPE-ERROR]
What are the impacts of the change? The compact-info-entries-index type itself is only used by the index and cache-index fields of the compact-info-env struct. These data structures hold indices into a table of the particular things being stored in this compact-info-env. Note that this change doesn't change the size of the things in the table, just the size of the index and the cached index into the table, AFAICT.
There is one potential problem. At some point a table-size gets calculated based on the number of names in the environment. We call primify to get a prime number size for the table. primify declares its argument x as an (unsigned-byte x) and then iterates over the odd numbers >= x until it finds a prime number. But positive-prime declares its argument to be a fixnum, so it's possible that we could have a table size that is > most-positive-fixnum which cause positive-primep to fail. I haven't run into this in practice, but we might want to watch it for this and/or come up with a better approach for getting the size of the table. There's a comment about using almost-primify from hash-table.lisp, but this must be an out-of-date comment, as I can't find this function in hash-table.lisp. But failing when we get larger than most-positive-fixnum is much better than failing when we hit 65536 entries in a compact info env.
I'm getting tired of flogging this horse, but I think it's important that this get fixed. If anyone disagrees with what I've proposed, I'd love to hear it.
Index: globaldb.lisp RCS file: /cvsroot/sbcl/sbcl/src/compiler/globaldb.lisp,v retrieving revision 1.36 diff -u -r1.36 globaldb.lisp --- globaldb.lisp 12 Jun 2004 13:55:49 -0000 1.36 +++ globaldb.lisp 26 Mar 2005 15:03:13 -0000 @@ -476,7 +476,7 @@ ;;;; compact info environments
;;; The upper limit on the size of the ENTRIES vector in a COMPACT-INFO-ENV. -(def!constant compact-info-env-entries-bits 16) +(def!constant compact-info-env-entries-bits 32) (deftype compact-info-entries-index () `(unsigned-byte ,compact-info-env-entries-bits))
;;; the type of the values in COMPACT-INFO-ENTRIES-INFO