diff options
Diffstat (limited to 'transform.e')
| -rw-r--r-- | transform.e | 147 |
1 files changed, 133 insertions, 14 deletions
diff --git a/transform.e b/transform.e index 3079414..9637aee 100644 --- a/transform.e +++ b/transform.e @@ -192,6 +192,13 @@ ~ The key insight is that, some sense, the difference between source code ~ and binary is the ability to have comments. ~ +~ The hex transform keeps some large, complex state as it runs, unlike the +~ label and log-load transforms, which only require very simple state. This +~ state is called the "output metadata". It's quite an intense topic, and to +~ use the hex transform you only need to know that it exists, not how it's +~ structured. So, details on the output metadata are described below, under +~ "hex transform implementation", rather than here. +~ ~ The hex transform DOES NOT WORK yet. It's still in development. ~ TODO update this note when it does work @@ -280,6 +287,9 @@ { drop zero-input-buffer-metadata } if-else ; +~ We have a bunch of accessors for the transformation state structure, which +~ are all functions from pointers to pointers. We also define a global +~ variable which points to the transformation state. : transformation-state-saved-here ; : transformation-state-saved-latest 8 + ; : transformation-state-output-buffer-start 2 8 * + ; @@ -2603,17 +2613,95 @@ allocate-transformation-state s" transformation-state" variable ~ The following code is all part of implementing the hex transform. For ~ conceptual overview, see the top of this file. ~ -~ The hex transform DOES NOT WORK yet. It's still in development. -~ TODO update this note when it does work - -~ The hex transform keeps some large, complex state as it runs, unlike the -~ label and log-load transforms, which only require very simple state. This -~ state is called the "output metadata"; a buffer for storing it is provided -~ by the transform's caller, and referenced during the transform via -~ transformation-state-output-metadata. -~ -~ The output metadata is a header structure followed by an array of -~ entry structures, as follows: +~ The hex transform operates under some very complex memory-management +~ assumptions, even compared to the other transforms, so it's worth going over +~ all that and studying how it comes together, even though much of it is +~ material that has been explained elsewhere. This explanation is here with +~ the implementation details rather than at the top of the file, because it's +~ only necessary to understanding how to modify the transform, not how to use +~ it. +~ +~ This is the bit of the hex transform where all the memory-management +~ assumptions are in play at once, so, a quick refresher. The transform +~ itself is running in the outer context; hex-trace and its siblings are +~ part of Evocation's main log and dictionary. This is also true for the +~ label and log-load transforms. Evocation has two global variables, +~ "here" and "latest", which are the root of all references to the log and +~ dictionary, respectively. Remember, the log is the structure used for +~ memory allocation, and the dictionary, which forms the majority of the +~ log, is the structure used for looking up words by name. +~ +~ Every transform, in its various ways, is changing what it means to +~ "execute" and "compile" code; therefore, when the transforms are invoked, +~ they are passed an "output buffer" which is used as the destination for +~ any compilation that occurs, and any scratch space that is needed. When +~ more than one transform is running at once (such as when the hex transform +~ operates on code that is itself running the label transform), each +~ transform has its own separate output buffer. +~ +~ Furthermore, the transformation facility helps each transform to swap out +~ the values of "here" and "latest", and restore them later, so that the code +~ being transformed will see the output buffer as if it is the "real" log, and +~ will see the separate dictionary stored within the output buffer as if it is +~ the "real" dictionary. +~ +~ All this stuff about the output buffer applies to every transform. The +~ hex transform takes it one step further, and as such there are two +~ additional memory regions that are important to understanding it. The first +~ is the label-loop buffer, which is allocated out of the inner log, and used +~ to hold draft versions of the binary output that the inner compilation +~ process would eventually emit to the standard output stream, if it were +~ being allowed to emit output in the normal way. The second is the output +~ metadata buffer, which is allocated out of the outer log, and used to hold +~ metadata such as comments and formatting pertaining to code structure, which +~ will need to be referenced when actually outputting the final hex dump. +~ +~ So, the label-loop buffer can be thought of as scratch space for the +~ compiled binary output from the code being transformed, and the output +~ metadata buffer can be thought of as the scratch space for the formatted hex +~ dump that will be made out of that output. It's important not to confuse +~ either of them with the output buffer, which is the scratch space used as +~ part of the compilation process. +~ +~ Confusingly, the output buffer and the label-loop buffer both contain +~ logs and dictionaries, at least potentially, if the code being compiled is +~ Forth code. It is also possible to use the hex transform on assembly code, +~ in which case the label-loop buffer won't have a log in it but the output +~ buffer still will. In practice, the hex transform frequently winds up +~ needing to be careful about the log and dictionary in the output buffer, but +~ rarely does anything that directly touches the log and dictionary in the +~ label-loop buffer, so you can usually ignore the latter when reading this +~ code. +~ +~ Allocating the output buffer and output metadata buffer is the +~ responsibility of the caller of hex-transform. Allocating the label-loop +~ buffer is the responsibility of label-loop, which is called by the code +~ being transformed, but the hex transform intervenes in that allocation to +~ save information about it. +~ +~ The output buffer is referenced during the transform via the +~ transformation-state-output-buffer-start field of the global +~ "transformation-state" variable, and the output metadata buffer is +~ referenced via the transformation-state-output-metadata field of that same +~ variable. The label-loop buffer is referenced from the +~ hex-output-metadata-label-loop-buffer-start and -length fields of the +~ output metadata's top-level header, and the code being transformed passes +~ around its own reference to that same buffer. +~ +~ Finally, recall that, when Evocation itself is the thing being +~ transformed, the code under transformation includes its own copy of the +~ transformation facility, along with all the rest of Evocation. When the code +~ under transformation invokes the label and log-load transforms, it will do +~ so using the copies of those transforms which have been transformed by the +~ hex transform. The same goes for most Evocation internals invoked by the +~ compilation process, except for whatever specific exceptions the hex +~ transform creates. Do not confuse the inner copies of anything with the +~ outer copies. Also try not to worry too much about the inner-inner copies, +~ let the inner transforms deal with them. +~ +~ Got all that background fresh in your mind? Okay, time to get concrete! +~ The output metadata is a header structure followed by an array of entry +~ structures, as follows: ~ ~ (header start) ~ 0x00 - 0x07 Label-loop buffer start @@ -2624,6 +2712,11 @@ allocate-transformation-state s" transformation-state" variable ~ ... + 0x10 - ... + 0x17 String pointer ~ (entry fields repeat at successive offsets, until...) ~ ... + 0x00 - ... + 0x07 (end) Zero, used as a delimiter + + +~ We have a bunch of accessors and traversers for the output metadata, which +~ are all functions from pointers to pointers. Don't confuse these with the +~ accessors for the transformation state, defined far, far above. : hex-output-metadata-label-loop-buffer-start ; : hex-output-metadata-label-loop-buffer-length 8 + ; : hex-output-metadata-first-entry 2 8 * + ; @@ -3151,10 +3244,36 @@ allocate-transformation-state s" transformation-state" variable ' ] entry-to-execution-token execute ; +~ This word gets installed as a trap at the beginning of every Forth word, +~ so that it can examine the control stack and produce kinds of metadata that +~ require information that only exists at runtime. See +~ hex-semicolon-alterante, below, for details of how it's installed. +~ +~ It is worth noting that hex-trap can coexist with the more-targeted traps +~ installed by hex-colon-alternate. It runs before them, because +~ hex-colon-alternate prepends its stuff first and then +~ hex-semicolon-alternate slides that all forward to prepend some more. : hex-trace ~ value@ here @ swap 0 hex-output-metadata-entry-type-line-comment ~ add-hex-output-metadata-entry - ." tracey tracey" newline ; + newline + ~ This stack-walking code is modeled off the code in list-callers, in + ~ dynamic.e. + ~ + + ~ This is the bit of the hex transform where all the memory-management + ~ assumptions are in play at once, so, in understanding it, it's probably + ~ a good idea to skim back over the background described under "hex + ~ transform implementation", above, and make sure it's fresh in your mind. + ~ + ~ It's important to notice that, as we walk it, the values of "here" and + ~ "latest" reflect the log and dictionary of the code being transformed, not + ~ the log and dictionary of the outer program. Therefore, we can use + ~ containing-entry to find the entry headers. We can't use is-in-log, + ~ though, because the output buffer is + control@ { dup r0 @ > } + { dup @ symbolize-pointer newline 8 + } while + drop ; ~ We want to trace every call to a Forth word, for use in generating ~ metadata. To do this, we want to add a prefix trap to the calls, but we @@ -3557,13 +3676,13 @@ allocate-transformation-state s" transformation-state" variable ~ an input string. It is directly analogous to "quit", in interpret.e, but is ~ more complex. ~ -~ (comment buffer start, output buffer start, output point, +~ (output metadata buffer start, output buffer start, output point, ~ input string pointer, delimiter pointer ~ -- output buffer start, output point) : hex-transform main-input-buffer dup push-input-buffer 3roll attach-string-to-input-buffer - ~ (comment buffer start, output buffer start, output point, + ~ (output metadata buffer start, output buffer start, output point, ~ delimiter pointer) ~ Save the old values of "here" and "latest", and set the initial values |