diff --git a/linux-dynamic.e b/linux-dynamic.e
index ee34e0e..3809a0e 100644
--- a/linux-dynamic.e
+++ b/linux-dynamic.e
@@ -2,9 +2,14 @@
~ ~~ More system calls for Linux ~~
~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~
-~ Everything that takes a struct is here, because that lets us define the
-~ system call next to the allocation stuff, for ease of reference. There are
-~ also higher-level facilities, later in the file.
+~ Here's some more system calls for the Linux kernel, augmenting the ones in
+~ linux.e. We needed the basic I/O stuff early, to use in building the
+~ interpreter, but there's less-basic stuff that benefits from dynamic
+~ definition. That stuff is here.
+~
+~ In particular, everything that takes a struct is here, because that lets
+~ us define the system call next to the allocation stuff, for ease of
+~ reference. There are also higher-level facilities, later in the file.
~ Since the only way to know the struct layout is by reading the kernel
diff --git a/transform.e b/transform.e
index 3079414..9637aee 100644
--- a/transform.e
+++ b/transform.e
@@ -192,6 +192,13 @@
~ The key insight is that, some sense, the difference between source code
~ and binary is the ability to have comments.
~
+~ The hex transform keeps some large, complex state as it runs, unlike the
+~ label and log-load transforms, which only require very simple state. This
+~ state is called the "output metadata". It's quite an intense topic, and to
+~ use the hex transform you only need to know that it exists, not how it's
+~ structured. So, details on the output metadata are described below, under
+~ "hex transform implementation", rather than here.
+~
~ The hex transform DOES NOT WORK yet. It's still in development.
~ TODO update this note when it does work
@@ -280,6 +287,9 @@
{ drop zero-input-buffer-metadata } if-else ;
+~ We have a bunch of accessors for the transformation state structure, which
+~ are all functions from pointers to pointers. We also define a global
+~ variable which points to the transformation state.
: transformation-state-saved-here ;
: transformation-state-saved-latest 8 + ;
: transformation-state-output-buffer-start 2 8 * + ;
@@ -2603,17 +2613,95 @@ allocate-transformation-state s" transformation-state" variable
~ The following code is all part of implementing the hex transform. For
~ conceptual overview, see the top of this file.
~
-~ The hex transform DOES NOT WORK yet. It's still in development.
-~ TODO update this note when it does work
-
-~ The hex transform keeps some large, complex state as it runs, unlike the
-~ label and log-load transforms, which only require very simple state. This
-~ state is called the "output metadata"; a buffer for storing it is provided
-~ by the transform's caller, and referenced during the transform via
-~ transformation-state-output-metadata.
-~
-~ The output metadata is a header structure followed by an array of
-~ entry structures, as follows:
+~ The hex transform operates under some very complex memory-management
+~ assumptions, even compared to the other transforms, so it's worth going over
+~ all that and studying how it comes together, even though much of it is
+~ material that has been explained elsewhere. This explanation is here with
+~ the implementation details rather than at the top of the file, because it's
+~ only necessary to understanding how to modify the transform, not how to use
+~ it.
+~
+~ This is the bit of the hex transform where all the memory-management
+~ assumptions are in play at once, so, a quick refresher. The transform
+~ itself is running in the outer context; hex-trace and its siblings are
+~ part of Evocation's main log and dictionary. This is also true for the
+~ label and log-load transforms. Evocation has two global variables,
+~ "here" and "latest", which are the root of all references to the log and
+~ dictionary, respectively. Remember, the log is the structure used for
+~ memory allocation, and the dictionary, which forms the majority of the
+~ log, is the structure used for looking up words by name.
+~
+~ Every transform, in its various ways, is changing what it means to
+~ "execute" and "compile" code; therefore, when the transforms are invoked,
+~ they are passed an "output buffer" which is used as the destination for
+~ any compilation that occurs, and any scratch space that is needed. When
+~ more than one transform is running at once (such as when the hex transform
+~ operates on code that is itself running the label transform), each
+~ transform has its own separate output buffer.
+~
+~ Furthermore, the transformation facility helps each transform to swap out
+~ the values of "here" and "latest", and restore them later, so that the code
+~ being transformed will see the output buffer as if it is the "real" log, and
+~ will see the separate dictionary stored within the output buffer as if it is
+~ the "real" dictionary.
+~
+~ All this stuff about the output buffer applies to every transform. The
+~ hex transform takes it one step further, and as such there are two
+~ additional memory regions that are important to understanding it. The first
+~ is the label-loop buffer, which is allocated out of the inner log, and used
+~ to hold draft versions of the binary output that the inner compilation
+~ process would eventually emit to the standard output stream, if it were
+~ being allowed to emit output in the normal way. The second is the output
+~ metadata buffer, which is allocated out of the outer log, and used to hold
+~ metadata such as comments and formatting pertaining to code structure, which
+~ will need to be referenced when actually outputting the final hex dump.
+~
+~ So, the label-loop buffer can be thought of as scratch space for the
+~ compiled binary output from the code being transformed, and the output
+~ metadata buffer can be thought of as the scratch space for the formatted hex
+~ dump that will be made out of that output. It's important not to confuse
+~ either of them with the output buffer, which is the scratch space used as
+~ part of the compilation process.
+~
+~ Confusingly, the output buffer and the label-loop buffer both contain
+~ logs and dictionaries, at least potentially, if the code being compiled is
+~ Forth code. It is also possible to use the hex transform on assembly code,
+~ in which case the label-loop buffer won't have a log in it but the output
+~ buffer still will. In practice, the hex transform frequently winds up
+~ needing to be careful about the log and dictionary in the output buffer, but
+~ rarely does anything that directly touches the log and dictionary in the
+~ label-loop buffer, so you can usually ignore the latter when reading this
+~ code.
+~
+~ Allocating the output buffer and output metadata buffer is the
+~ responsibility of the caller of hex-transform. Allocating the label-loop
+~ buffer is the responsibility of label-loop, which is called by the code
+~ being transformed, but the hex transform intervenes in that allocation to
+~ save information about it.
+~
+~ The output buffer is referenced during the transform via the
+~ transformation-state-output-buffer-start field of the global
+~ "transformation-state" variable, and the output metadata buffer is
+~ referenced via the transformation-state-output-metadata field of that same
+~ variable. The label-loop buffer is referenced from the
+~ hex-output-metadata-label-loop-buffer-start and -length fields of the
+~ output metadata's top-level header, and the code being transformed passes
+~ around its own reference to that same buffer.
+~
+~ Finally, recall that, when Evocation itself is the thing being
+~ transformed, the code under transformation includes its own copy of the
+~ transformation facility, along with all the rest of Evocation. When the code
+~ under transformation invokes the label and log-load transforms, it will do
+~ so using the copies of those transforms which have been transformed by the
+~ hex transform. The same goes for most Evocation internals invoked by the
+~ compilation process, except for whatever specific exceptions the hex
+~ transform creates. Do not confuse the inner copies of anything with the
+~ outer copies. Also try not to worry too much about the inner-inner copies,
+~ let the inner transforms deal with them.
+~
+~ Got all that background fresh in your mind? Okay, time to get concrete!
+~ The output metadata is a header structure followed by an array of entry
+~ structures, as follows:
~
~ (header start)
~ 0x00 - 0x07 Label-loop buffer start
@@ -2624,6 +2712,11 @@ allocate-transformation-state s" transformation-state" variable
~ ... + 0x10 - ... + 0x17 String pointer
~ (entry fields repeat at successive offsets, until...)
~ ... + 0x00 - ... + 0x07 (end) Zero, used as a delimiter
+
+
+~ We have a bunch of accessors and traversers for the output metadata, which
+~ are all functions from pointers to pointers. Don't confuse these with the
+~ accessors for the transformation state, defined far, far above.
: hex-output-metadata-label-loop-buffer-start ;
: hex-output-metadata-label-loop-buffer-length 8 + ;
: hex-output-metadata-first-entry 2 8 * + ;
@@ -3151,10 +3244,36 @@ allocate-transformation-state s" transformation-state" variable
' ] entry-to-execution-token execute
;
+~ This word gets installed as a trap at the beginning of every Forth word,
+~ so that it can examine the control stack and produce kinds of metadata that
+~ require information that only exists at runtime. See
+~ hex-semicolon-alterante, below, for details of how it's installed.
+~
+~ It is worth noting that hex-trap can coexist with the more-targeted traps
+~ installed by hex-colon-alternate. It runs before them, because
+~ hex-colon-alternate prepends its stuff first and then
+~ hex-semicolon-alternate slides that all forward to prepend some more.
: hex-trace
~ value@ here @ swap 0 hex-output-metadata-entry-type-line-comment
~ add-hex-output-metadata-entry
- ." tracey tracey" newline ;
+ newline
+ ~ This stack-walking code is modeled off the code in list-callers, in
+ ~ dynamic.e.
+ ~
+
+ ~ This is the bit of the hex transform where all the memory-management
+ ~ assumptions are in play at once, so, in understanding it, it's probably
+ ~ a good idea to skim back over the background described under "hex
+ ~ transform implementation", above, and make sure it's fresh in your mind.
+ ~
+ ~ It's important to notice that, as we walk it, the values of "here" and
+ ~ "latest" reflect the log and dictionary of the code being transformed, not
+ ~ the log and dictionary of the outer program. Therefore, we can use
+ ~ containing-entry to find the entry headers. We can't use is-in-log,
+ ~ though, because the output buffer is
+ control@ { dup r0 @ > }
+ { dup @ symbolize-pointer newline 8 + } while
+ drop ;
~ We want to trace every call to a Forth word, for use in generating
~ metadata. To do this, we want to add a prefix trap to the calls, but we
@@ -3557,13 +3676,13 @@ allocate-transformation-state s" transformation-state" variable
~ an input string. It is directly analogous to "quit", in interpret.e, but is
~ more complex.
~
-~ (comment buffer start, output buffer start, output point,
+~ (output metadata buffer start, output buffer start, output point,
~ input string pointer, delimiter pointer
~ -- output buffer start, output point)
: hex-transform
main-input-buffer dup push-input-buffer
3roll attach-string-to-input-buffer
- ~ (comment buffer start, output buffer start, output point,
+ ~ (output metadata buffer start, output buffer start, output point,
~ delimiter pointer)
~ Save the old values of "here" and "latest", and set the initial values
|