From 77e8ce989010fbd94080e780fb0b5cb6c10434e8 Mon Sep 17 00:00:00 2001 From: Irene Knapp Date: Mon, 28 Sep 2026 18:07:52 -0700 Subject: move all the source files into subdirectories and rename one, linux-dynamic.e to language/dynamic/linux.e. also make sure all references in comments and documentation refer to the new locations and names. this directory structure reflects code organization which had existed in Irenes' head, but was likely not obvious to other readers. now hopefully it is more discoverable. Force-Push: yes Change-Id: Icfef3eae812a50fe20202191999e68e1f7db445a --- compilation/elf.e | 179 ++ compilation/execution.e | 593 +++++ compilation/labels.e | 296 +++ compilation/transform.e | 6512 +++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 7580 insertions(+) create mode 100644 compilation/elf.e create mode 100644 compilation/execution.e create mode 100644 compilation/labels.e create mode 100644 compilation/transform.e (limited to 'compilation') diff --git a/compilation/elf.e b/compilation/elf.e new file mode 100644 index 0000000..f39a1a0 --- /dev/null +++ b/compilation/elf.e @@ -0,0 +1,179 @@ +~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~ ~~ Executable file format ~~ +~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~ +~ Before we do anything specific to the actual program we're building, we +~ do a lot of ELF-specific stuff to ensure that our output is in a format +~ Linux knows how to run. +~ +~ This relies on the label facility defined in compilation/labels.e. Make +~ sure to load that first. +~ +~ This file makes use of magic comments for the hex transform, described in +~ detail in compilation/transform.e, but its use is trivial and doesn't +~ require accounting conventions. + +~ ~~~~~~~~~~~~~~~~~~~~~ +~ ~~ ELF file header ~~ +~ ~~~~~~~~~~~~~~~~~~~~~ +~ +~ First, we output ELF's top-level file header. This header describes the +~ entire file. An ELF always has exactly one of this header, which is always +~ at the start of the file. +~ +~ The program we're building should call this word as the first output it +~ generates. +~ +~ The most interesting thing here is the entry pointer. +~ +~ It's also worth noticing that we set the ABI version to 3. An ABI version +~ of 0 (for the System V ABI) would work fine too, but 3 allows FreeBSD's +~ Linux compatibility features to kick in. + +: elf-file-header + ~ : blank-line + ~ : ELF file header + ~ : indent + ~ : * denotes mandatory fields according to breadbox + current-offset 3unroll + + ~ : 1 suppress + 0x7f pack8 s" ELF" pack-raw-string ~ : -4 *magic number + 2 pack8 ~ : -1 64-bit + 1 pack8 ~ : -1 little-endian + 1 pack8 ~ : -1 ELF header format v1 + 3 pack8 ~ : -1 Linux ABI + 0 pack64 ~ : -8 (padding) + ~ : blank-line + + 2 pack16 ~ : -2 *executable + 0x3e pack16 ~ : -2 *Intel x86-64 + 1 pack32 ~ : -4 ELF format version + ~ : blank-line + + L@' cold-start L@' origin + pack64 ~ : -8 *entry point + ~ This includes the origin, intentionally. + + L@' elf-program-header pack64 ~ : -8 *program header offset + ~ We place the program header immediately after the ELF header. This + ~ offset is from the start of the file. + 0 pack64 ~ : -8 section header offset + 0 pack32 ~ : -4 processor flags + ~ : blank-line + + L@' elf-header-size pack16 ~ : -2 ELF header size + L@' elf-program-header-size pack16 ~ : -2 *program header entry size + 1 pack16 ~ : -2 *number of program header entries + 0 pack16 ~ : -2 section header entry size + 0 pack16 ~ : -2 number of section header entries + 0 pack16 ~ : -2 section name string table index + + ~ Though hardcoding the size of this header would work fine, it's easier + ~ to use the label system to keep track of its size. The only place this is + ~ actually referenced is right here in the header. + current-offset 4 roll - L!' elf-header-size + L' elf-header-size set-label-non-offset + ~ : deindent + ; + + +~ ~~~~~~~~~~~~~~~~~~~~~~~~ +~ ~~ ELF program header ~~ +~ ~~~~~~~~~~~~~~~~~~~~~~~~ +~ +~ Second, we output ELF's program header, which lists the memory regions +~ ("segments") we want to have and where we want them to come from. There may +~ be any number of these entries, one per segment, , and they may be anywhere +~ in the file as long as they're consecutive. +~ +~ We list just a single region, which is the entire contents of the ELF file +~ from disk, and we put the program header immediately after the file header. +~ The program we're building should call this word as the second output it +~ generates. +~ +~ It would be more typical to use this header to ask the loader to give us +~ separate code and data segments, and perhaps a stack or heap, but this keeps +~ things simple, and we can create those things for ourselves later. +~ +~ We do have a little stack space available, though we don't explicitly +~ request any; the kernel allocates it for us as part of exec() so that it can +~ pass us argc and argv (which we ignore). That stack space will be at a +~ random address, different every time, because of ASLR; that's a neat +~ security feature, so we leave it as-is. Note that ASLR doesn't happen when +~ you run under gdb, so if you aren't seeing it, that's probably why. + +~ ~~ +: elf-program-header + ~ : blank-line + ~ : ELF program header + ~ : indent + ~ * denotes mandatory fields according to breadbox + current-offset L!' elf-program-header + current-offset 3unroll + + 1 pack32 ~ : -4 *"loadable" segment type + 0x05 pack32 ~ : -4 *read+execute permission + 0 pack64 ~ : -8 *offset in file + L@' origin pack64 ~ : -8 *virtual address + ~ required, but can be anything, subject to alignment + 0 pack64 ~ : -8 physical address (ignored) + ~ : blank-line + + L@' total-size pack64 ~ : -8 *size in file + L@' total-size pack64 ~ : -8 *size in memory + ~ : blank-line + + 0 pack64 ~ : -8 segment alignment + ~ for relocation, but this doesn't apply to us + + ~ As with the file header, we use the label system to keep track of the + ~ program header's size. + current-offset 4 roll - L!' elf-program-header-size + L' elf-program-header-size set-label-non-offset + ~ : deindent + ; + + +~ This alternate version of the program header isn't used by evoke.e, but +~ may be useful for other programs, such as hex.e. Its only difference from +~ the base version is that it adds the write permission. +: elf-program-header-writable + ~ : blank-line + ~ : ELF program header + ~ : indent + ~ * denotes mandatory fields according to breadbox + current-offset L!' elf-program-header + current-offset 3unroll + + 1 pack32 ~ : -4 *"loadable" segment type + 0x07 pack32 ~ : -4 *read+write+execute permission + 0 pack64 ~ : -8 *offset in file + L@' origin pack64 ~ : -8 *virtual address + ~ required, but can be anything, subject to alignment + 0 pack64 ~ : -8 physical address (ignored) + ~ : blank-line + + L@' total-size pack64 ~ : -8 *size in file + L@' total-size pack64 ~ : -8 *size in memory + ~ : blank-line + + 0 pack64 ~ : -8 segment alignment + ~ for relocation, but this doesn't apply to us + + ~ As with the file header, we use the label system to keep track of the + ~ program header's size. + current-offset 4 roll - L!' elf-program-header-size + L' elf-program-header-size set-label-non-offset + ~ : deindent + ; + +~ ~~~~~~~~~~~~~~~~ +~ ~~ That's it! ~~ +~ ~~~~~~~~~~~~~~~~ +~ +~ ELF is a simple format, really. Now you can output your own machine code +~ that you generate however you want; make sure to define the labels "origin" +~ and "cold-start". Origin will control the address the code loads at; +~ cold-start will be the first thing that runs. The origin is arbitrary, but +~ can't be zero. + diff --git a/compilation/execution.e b/compilation/execution.e new file mode 100644 index 0000000..ecf5ec4 --- /dev/null +++ b/compilation/execution.e @@ -0,0 +1,593 @@ +~ ~~~~~~~~~~~~~~~~~~~~~ +~ ~~ Execution model ~~ +~ ~~~~~~~~~~~~~~~~~~~~~ +~ +~ We use Forth-style dual stacks, one for values and one for control. We +~ use rsp for values, just like C does. We use rbp for the control stack, +~ which is a special Forth-y stack: These are pointers into the bodies of +~ Forth words, not return addresses. +~ +~ The choice of rsp and rbp for the stack pointers imitates Jonesforth; +~ I'm hopeful that it gives us convenient addressing modes, and will report +~ back about that when I feel that I understand the implications. +~ +~ In Forth, everything is a "word", including mutable variables. +~ Conceptually, a word is a unit of execution, which may be implemented +~ either in machine code or as an array of pointer to other words. +~ +~ This polymorphism is implemented by having each word's contents begin +~ with a "codeword", which is a pointer to machine code that "interprets" +~ the rest of the contents. In the case of words implemented in machine +~ code, the codeword points directly to that code, which is normally right +~ next to it. +~ +~ Variables, to Forth, are simply one more thing that can be executed; the +~ effect of executing a variable is to push its address onto the value +~ stack. +~ +~ We adopt this model of words, codewords, and variables-as-words. It's +~ really nice how it doesn't force anything else on us, not even a heap, +~ though we do end up using the log as a sort-of heap. +~ +~ We specifically implement a version of calling and returning that Forth +~ calls indirect threaded code: The control stack is a stack of pointers +~ into the middle of interpreted words. The interpreter snippet, called +~ docol, implements calling. Each word is responsible for making sure +~ returning works properly. Interpreted words accomplish this by ending with +~ the word "exit", while machine-code words accomplish it by ending with a +~ verbatim snippet called "next". +~ +~ Conceptually, "next" returns, but more specifically it accomplishes this +~ by doing the caller's next dispatch for it; thus control never actually +~ goes back to the caller's interpreter after initial setup. For performance +~ reasons, "next" is always inlined, so we define it as a macro. +~ +~ The docol routine is just ordinary code, not a macro. It's defined later +~ in this file, as a label. +~ +~ Notionally, we could consider not having a dictionary, and not giving +~ our words names. However, it feels silly to stop when we're so close to +~ being a full Forth, and using names for things solves a bootstrapping +~ problem related to log management - see the write-up of cold-start about +~ how the log is created, below. So, we add an additional header before the +~ codeword for this purpose. +~ +~ The Forth dictionary is usually a linked list of every word that has +~ ever been defined, with the newest at the head; the names of words are +~ stored in string fields, often right next to the link pointer. We adopt +~ this model, with the field sizes and order shown in the quick reference +~ below. We break with Forth tradition in one way: Rather than having a +~ length field, we use a null-terminated string. Thus, there's no length +~ limit on names. This necessitates breaking out the flags (to be explained +~ later) into their own byte, rather than taking bits from the length field +~ for them. +~ +~ There's an important performance consideration: Executable words +~ reference each other by pointers to their respective codewords. However, +~ dictionary entries reference each other by pointers to their respective +~ link fields. Traversing from the link field to the codeword is easy, +~ though it's a non-constant-time operation: Just walk the string. In order +~ to make Forth words easy to "decompile", it would be nice to also have a +~ way to traverse backwards. We solve this by making the name field be +~ null-terminated at both ends. Fun, yeah? +~ +~ +~ +~ +~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~ Quick Reference +~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~ +~ The layout of an interpreted word: +~ +~ (overall start) +~ 0x00 - 0x07 Link (to next-oldest word) +~ 0x08 - 0x08 HM00000I Flags +~ H - hidden +~ M - metadata +~ I - immediate +~ all other bits reserved +~ (name start) +~ 0x09 - 0x09 Null byte (terminates name) +~ 0x0a - name-end - 1 Name, as UTF-8 +~ name-end - name-end Null byte (terminates name) +~ (padding start) +~ name-end + 1 - codeword-start - 1 Zero-pad to 8-byte boundary +~ (it's possible this will be zero bytes long) +~ (codeword start) +~ ... + 0x00 - ... + 0x08 Codeword (ie. address of docol) +~ (8-byte chunks) Addresses of other words +~ - ... (end) Address of "exit" word +~ +~ The layout of a machine-code word is different only from the codeword on: +~ +~ ... + 0x00 - ... + 0x08 Addresss of next byte +~ ... + 0x08 - ???? Arbitrary machine code +~ - ... (end) Inlined implementation of next +~ +~ Also, words always start at 8-byte boundaries. +~ +~ +~ REGISTER usage conventions: +~ +~ * rsi is the "instruction pointer" for the "interpreter". +~ That is, it points to some word-pointer inside an array of +~ word-pointers inside the content of the word they're part of. It always +~ points to the next word that should be executed, whose execution hasn't +~ begun yet. +~ +~ * rbp points to the top of the control stack +~ These are former values of rsi, to eventually be returned to, from +~ successively older callers as you look further up the stack. The stack +~ grows downwards in memory. Since values are kept separately, the only +~ thing on the control stack is return addresses, one per layer of call. +~ +~ * rsp points to the top of the value stack +~ The value stack has no specific format, but it grows downwards in +~ memory. In particular there's no concept of stack frames, because items +~ on the stack don't belong to any particular word; the value stack in +~ Forth is in part a mechanism for passing values between words. +~ +~ Additionally, immediately after beginning execution of a word: +~ +~ * rax points to the address of the codeword being executed +~ The value of rax is purely for the callee's benefit, and does not need +~ to be preserved. +~ +~ Other registers are purely discretionary, and are not preserved across +~ calls. +~ +~ +~ FLAG usage: +~ +~ * DF should be 0 +~ We use lodsq extensively and that makes it increment rsi after using it. +~ +~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +~ The macros next, beforenext, pushcontrol, and popcontrol are implemented +~ in language/static/execution-support.e. It's a good idea to go read about +~ them now. +~ +~ TODO the following high-level summary should go... somewhere? +~ When a docol word is invoked, it immediately does pushcontrol (see +~ execution-support.e) to push rsi onto the control stack, whose top is +~ indicated by rbp. When an assembly word is invoked, it leaves rsi and rbp +~ alone; assembly words don't normally call other words, so there's no need +~ for them to appear on the control stack at all. +~ +~ When an assembly word returns, it looks for an execution token pointed +~ to by rsi, increments rsi, and invokes that word as if calling it. This +~ is the threaded calling approach to Forth execution. When a docol word +~ returns, it calls exit, which needs to remove itself from the control +~ stack, so exit pops the top of the control stack into rsi to get back into +~ its caller's context, then does the same steps an assembly word would do. + + +~ Constants +~ ~~~~~~~~~ +~ +~ These are used in cold-start, just below. + +: log-requested-address 0x0000001000000000 ; ~ (very arbitrary) +: log-size 0x0000000001000000 ; ~ 16 MiB +: control-stack-size 0x10000 ; ~ 64 KiB + + +~ Routine cold-start +~ ~~~~~~~~~~~~~~~~~~ +~ +~ This is the entry point of the whole program, the very first code we +~ actually execute. Linkers traditionally call this _start, but let's live a +~ little. Anyway, the ELF header points to it and exec() jumps to it; from the +~ kernel's perspective, there's no such thing as names. Also, though this code +~ could be anywhere in the code part of the output, in order to make the +~ hexdump pretty we put it at the start. +~ +~ The kernel gives us most registers zeroed, and rsp pointing to the +~ command-line stuff (argc, argv, envp), which is at an ASLR'd address with +~ some stack space allocated for us, despite the fact we didn't request any. +~ It also gives us all the flags clear except IF, but we don't rely on that. +~ Lastly, of course, it loads our code segment and sets the instruction +~ pointer where we asked; we don't need to check what those addresses are, +~ because they're not randomized. +~ +~ Notably, though cold-start points rbp to the control stack, it leaves +~ the control stack empty, so trying to return will pop a value that doesn't +~ exist, and crash. Eventually, the bottom value on the control stack will +~ be the word "quit", which loops forever and is the closest thing Forth has +~ to a "main" word, but it's up to warm-start to make that happen. +~ +~ Cold-start is really only responsible for one-time initialization. +~ +~ +~ +~ Registers in: +~ +~ * rsp points to the logical top of the value stack +~ The kernel sets this up for us, and we need to save it somewhere so +~ Forth can use it. +~ +~ Registers out: +~ +~ * rsi points within "warm-start" +~ * rsp points to the top of the value stack +~ * rbp points to the top of the control stack +~ +~ Stack in: +~ +~ * argc, argv, envp in the usual Unix way +~ We ignore them, though. +~ +~ Stack out: +~ +~ * The value of "log", as a pointer +~ The meaning of this will be explained below. +~ +~ Registers within: +~ +~ * rdi points to the base the log was allocated at, once it exists +~ This is the same value that "log" will hold, once we reach a point +~ where we have variables. Of course, variables are stored on the log, +~ hence this temporary measure. +~ +~ We also take this opportunity to define soeme memory layout parameters +~ that this routine will be responsible for doing something with: +~ +~ (output memory start, current output point +~ -- output memory start, current output point) +: output-cold-start + ~ : blank-line + ~ : blank-line + ~ : ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ + ~ : ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ + ~ : ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ + ~ : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + ~ : blank-line + ~ : This is the start of the cold-start routine, which is the first code + ~ : of any kind the executable runs. It has the responsibility of creating + ~ : the Forth execution environment, then invoking the warm-start routine, + ~ : which is defined below. + ~ : blank-line + ~ : indent + + current-offset L' cold-start set-label + cld ~ clear the DF flag + + ~ Prepare the log. + ~ + ~ We could ask for a data segment in the program header, but where's the + ~ fun in that? Instead, we call mmap(). + ~ + ~ If we wanted the kernel to do ASLR for us, passing address zero would + ~ cause it to pick somewhere at random, but instead we choose our own + ~ location. It's still not guaranteed to be where we ask for, so we still + ~ do the work to record where it wound up. We could pass the "fixed" flag + ~ and the kernel would trust us, but this gives us more options for + ~ interoperating with other runtimes. + ~ + 9 :rax mov-reg64-imm64 ~ : mmap() + log-requested-address :rdi mov-reg64-imm64 + ~ : 0 address (very arbitrary) + log-size :rsi mov-reg64-imm64 ~ : size (one meg) + 0x07 :rdx mov-reg64-imm64 + ~ : 0 protection (read+write+exec) + 0x22 :r10 mov-extrareg64-imm64 ~ : flags (private+anonymous) + 0 :r8 mov-extrareg64-imm64 ~ : file descriptor (ignored) + 0 :r9 mov-extrareg64-imm64 ~ : offset (ignored) + syscall + + ~ The return value of the system call is in rax, we'll use it in a sec. + ~ We need to save this somewhere in case we ever want to munmap() it; + ~ there's no widely-used name for it so we have to make one up. S0 and r0 + ~ are widely-used names for the physical tops (logical bottoms) of the + ~ value and control stacks, respectively, and we will eventually set those + ~ up as well, so we should keep those names in mind. The control stack + ~ lives within the log, while the value stack is its own segment. This + ~ value, though, is the physical bottom of the segment, meaning that it + ~ stays the same even as we allocate and deallocate things within it. This + ~ is unlike the two stack pointers, so we give it a name that doesn't + ~ suggest similarity: "log". + ~ + ~ Once Forth is fully set up, its internal variables will be accessed + ~ through variable-words like any other Forth data, including "log". To + ~ get to that point, though, we need to be able to hold onto variable data + ~ between now and then. In fact, if we don't have at least one of "log" + ~ and "here" (its counterpart which points to the logical top end), all + ~ our efforts to hold onto anything seem a bit doomed. + ~ + ~ So, we temporarily dedicate rdi to "log" - only within this routine - + ~ and store everything else in ways that let us find things by reference + ~ to it. We choose rdi because it works with the indexing modes we care + ~ about, and its name suggests its function. + ~ + ~ The strategy Jonesforth uses is not applicable to us; Jonesforth + ~ takes advantage of the linker to let its code segment refer to specific, + ~ pre-allocated objects in the data segment. We are our own linker, and we + ~ don't care to have a data segment. Hence, this approach. + ~ + ~ Keying things off "log" is the fundamental decision, but to make sure + ~ our variables are accessible both during early bootstrapping, and later, + ~ we also have to be thoughtful about data structures. More on that in a + ~ moment. + ~ + :rax :rdi mov-reg64-reg64 + + ~ We also initialize rbp. We could hold off and let "quit" do this, but + ~ doing it now is the easiest way to initialize the r0 variable, since + ~ there's no instruction that moves a 64-bit immediate to memory. + ~ + ~ This is the moment at which we decide where the control stack starts! + ~ Fun, right? "Allocation" is just a fancy word for picking where we want + ~ something, then being consistent about it - like placing furniture in + ~ your home. See below for a little more thought about why here in + ~ particular. + ~ + :rdi control-stack-size :rbp lea-reg64-disp32-reg64 + + ~ Now we save some stuff onto the log. These are the locations that + ~ will eventually be the backing stores of the Forth variables, but we + ~ don't create the word headers yet, since there's no requirement that + ~ they be next to the backing stores. We'll do that later, once we have + ~ word-writing infrastructure in place. For now, we just use their offsets + ~ relative to the physical bottom of the log, which are fixed. + ~ + ~ These will be the permanent homes of these values, though we have + ~ copies of them elsewhere while we're still in this routine. + ~ + :rdi control-stack-size 0x00 + :rdi mov-reg64-disp32-reg64 ~ log + :rsp control-stack-size 0x08 + :rdi mov-reg64-disp32-reg64 ~ s0 + :rbp control-stack-size 0x10 + :rdi mov-reg64-disp32-reg64 ~ r0 + L@' final-word-name :rax mov-reg64-imm64 + :rax control-stack-size 0x18 + :rdi mov-reg64-disp32-reg64 ~ latest + :rdi control-stack-size 0x28 + :rax lea-reg64-disp32-reg64 + :rax control-stack-size 0x20 + :rdi mov-reg64-disp32-reg64 ~ here + ~ + ~ * "log" is the physical bottom of the log + ~ The log grows upwards in memory, so this is also the logical + ~ bottom. This comes from the address mmap() just returned to us. + ~ * "s0" is the logical bottom of the value stack + ~ The value stack grows downwards in memory, so this is the physical + ~ top of it. This comes from the stack pointer the kernel initialized us + ~ with. + ~ * "r0" is the logical bottom of the control stack + ~ The control stack also grows downwards, so this is its pysical top + ~ as well. We allocate this dedicated space within the log right here, + ~ in this routine, through our choice of where to put things. + ~ * "here" is the physical start of the unallocated space in the log + ~ We allocate log space from bottom to top, by incrementing this + ~ value. So, it would also be accurate to say that it points immediately + ~ after the physical top of the allocated space. At any rate, the + ~ address it points to is the first one that hasn't been used yet. + ~ * "latest" is the address of the most-recently-defined word's header + ~ Defining new words changes this value. + ~ + ~ s0 and r0 are mostly used when we want to initialize or reinitialize + ~ their respective stacks - that is, discard all their contents at once. + ~ + ~ The value of r0 is the same address these variables start at, so + ~ you'll want to do a close read of the implementation of pushcontrol + ~ and convince yourself that it only ever writes things just below the rbp + ~ address it receives, never right on top of it. + ~ + ~ As an aside, by the way, please notice that strictly speaking, r0 + ~ could be a constant... but it isn't known until runtime, so we might as + ~ well make it a variable. That will play nicely with any astonishing + ~ memory shenanigans someone might wish to do in the future. + ~ + ~ Notice also that "here" points immediately after itself. This is just + ~ a convenience, making it the last one like that so that the concern is + ~ dealt with in a single place and is easy to keep up-to-date with code + ~ changes. + ~ + ~ A little more detail about why we offset everything by + ~ control_stack_size: We're carving out some space at the bottom of the + ~ log - which grows low-to-high - to be the control stack - which grows + ~ high-to-low. So the control stack is allocated out of the log as a + ~ fixed-size, one-time thing, and then the variables come immediately + ~ after that. We do need to use 32-bit displacement indexing to access + ~ them this way, but that's no big deal. + ~ + ~ This is perhaps questionable, they should maybe be separate segments + ~ created with separate calls to mmap(), but for now we're not worried + ~ about overflow so we use the same allocation for both. + ~ + ~ We'll come back to these variables a bit later and generate the word + ~ headers that point at them, but now we're almost ready to switch to + ~ proper threaded-execution, so we finish that setup first... + + ~ Push the value of "log" onto the value stack so that it can be the + ~ breadcrumb the threaded code needs to find... the backing store of + ~ "log". Yes, self-reference can be weird like that sometimes. There's + ~ nothing stopping "quit" from reading rdi, it just violates the + ~ abstraction... + :rdi push-reg64 + + ~ We are about to set up rsi, we did rbp already, and rsp came to us + ~ already set up. That's all that "next" needs, so take it away! + L@' warm-start L@' origin + :rsi mov-reg64-imm64 + pack-next + ~ : deindent + ; + + +~ Routine warm-start +~ ~~~~~~~~~~~~~~~~~~ +~ +~ This routine runs as Forth code. That is, indirect threaded execution has +~ been initialized, and the structure of this routine is an array of codeword +~ pointers. It was directly jumped to from cold-start. There's nowhere to +~ return to, so it needs to never return. +~ +~ (output buffer start, output point, input string pointer +~ -- output buffer start, output point) +: output-warm-start + ~ : blank-line + ~ : blank-line + ~ : ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ + ~ : ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ + ~ : ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ + ~ : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + ~ : blank-line + ~ : This is the start of the warm-start routine, which is the first Forth + ~ : code the executable runs, after the cold-start routine has created the + ~ : Forth execution environment. + ~ : + ~ : It does some initial setup here at the top; then the vast bulk of it + ~ : is actually the log-load routine, directly included; then finally there + ~ : is a small amount of finishing work done at the tail end of the + ~ : warm-start routine. + ~ : + ~ : The routine's body is interpreted Forth code, meaning that it is an + ~ : an array of pointers to the codewords of Forth words, occasionally + ~ : interspersed with literal values. + ~ : + ~ : The codeword pointers were located by using the label system, which + ~ : means the labels they reference must have been defined somewhere else, + ~ : normally by the label transform. The label-transformed code normally + ~ : occurs after the warm-start routine in the executable, which means these + ~ : are forward references to words defined far below. + ~ : blank-line + ~ : indent + + ~ While it's not actually a requirement that codeword pointers be + ~ word-aligned, it's highly likely that it helps performance. (Whether it + ~ does is up to Intel's microcode.) + 3unroll + 8 packalign + current-offset L!' warm-start + ~ (input string pointer, output buffer start, output point) + + ~ Before handing off to us, cold-start pushed a single value onto the + ~ stack, a pointer to the beginning of the log. Now, we load our entire + ~ Forth implementation onto that log, beginning with the minimal set of + ~ words needed to define more words. We do this because we need variables as + ~ infrastructure so we can eventually have dynamic definitions. + ~ + ~ There's something non-obvious here: words implemented statically as + ~ part of the executable image can't contain things that vary at runtime. + ~ That means that even if these words tried to implement some sort of + ~ dynamic lookup, they would have no way to find the root of whatever + ~ dynamic data structure they use. Dynamism needs to be bootstrapped. + ~ + ~ In a more traditional C-style program, static code could look up + ~ variables based on fixed addresses that are the same on every run. + ~ Failing that, we could dedicate a register to it, though that's a + ~ considerable expense. We chose not to do either of those things, because + ~ we want the versatility that comes with not being picky about our + ~ address space: It allows us to contemplate future improvements such as + ~ ASLR, or embedding into other processes that impose their own addressing + ~ constraints, or even coexisting with multiple versions of ourselves. + ~ That choice does mean we have the hard version of this bootstrapping + ~ problem, and copying ourselves to the log is how we solve it. + ~ + ~ We do have the log address right now, though that won't last. In case + ~ it's unclear why not: keeping it on the stack would require all future + ~ references to walk the stack, and somehow know when they've reached the + ~ bottom. The stack is a good place to keep things with clearly delimited + ~ lifetimes and visibility, but when we want something to live for our + ~ entire program and be easy to find from any code within it, we need to + ~ do something else. Anyway, since we have the address, we can use it for + ~ the next little bit of setup. We have a bunch of helper words, from + ~ language/static/log-load.e, which make this easier. + ~ + ~ The first few words we define are our variables, which hardcode the + ~ addresses they will return - but since we're doing this at runtime, + ~ "hardcoding" can reflect where our log is. This is the fundamental + ~ trick that makes the log usable. + ~ + ~ One more thing to notice: We already allocated the backing stores of + ~ these variables, and populated their initial values, in cold-start. The + ~ words we're defining return those same addresses for the same backing + ~ stores. So, we have continuity: Stuff defined in terms of the + ~ variable-words we're defining now will interoperate with the stuff that + ~ we define using the log-load helpers, which includes those very words. + ~ Both the log-load code and the later code are dealing with the same data + ~ structures, they're just using a different technique to find them. + ~ + ~ This is the only hardcoding we need to do; by building on top of it, + ~ we will soon reach a point where the rest of the system can be defined + ~ within itself. + + L@' log-load-log offset-to-target-address-space pack64 + ~ : -8 log-load-log #-- (codeword pointer) + L@' litstring offset-to-target-address-space pack64 + ~ : -8 litstring #-- (codeword pointer) + s" log" packstring 8 packalign + L@' log-load-variable offset-to-target-address-space pack64 + ~ : -8 log-load-variable #-- (codeword pointer) + + L@' log-load-s0 offset-to-target-address-space pack64 + ~ : -8 log-load-s0 #-- (codeword pointer) + L@' litstring offset-to-target-address-space pack64 + ~ : -8 litstring #-- (codeword pointer) + s" s0" packstring 8 packalign + L@' log-load-variable offset-to-target-address-space pack64 + ~ : -8 log-load-variable #-- (codeword pointer) + + L@' log-load-r0 offset-to-target-address-space pack64 + ~ : -8 log-load-r0 #-- (codeword pointer) + L@' litstring offset-to-target-address-space pack64 + ~ : -8 litstring #-- (codeword pointer) + s" r0" packstring 8 packalign + L@' log-load-variable offset-to-target-address-space pack64 + ~ : -8 log-load-variable #-- (codeword pointer) + + L@' log-load-latest offset-to-target-address-space pack64 + ~ : -8 log-load-latest #-- (codeword pointer) + L@' litstring offset-to-target-address-space pack64 + ~ : -8 litstring #-- (codeword pointer) + s" latest" packstring 8 packalign + L@' log-load-variable offset-to-target-address-space pack64 + ~ : -8 log-load-variable #-- (codeword pointer) + + L@' log-load-here offset-to-target-address-space pack64 + ~ : -8 log-load-here #-- (codeword pointer) + L@' litstring offset-to-target-address-space pack64 + ~ : -8 litstring #-- (codeword pointer) + s" here" packstring 8 packalign + L@' log-load-variable offset-to-target-address-space pack64 + ~ : -8 log-load-variable #-- (codeword pointer) + + ~ Having done that, nothing else needs to be defined in an unusual way, so + ~ we can go ahead and dispatch to the log-load transform, and do the rest of + ~ the code through that. + + ~ (input string pointer, output buffer start, output point) + 3roll log-load-transform + ~ (output buffer start, output point) + + ~ : blank-line + ~ : We're now done with the log-load routine; the next little bit of code + ~ : here is the final bit of the warm-start routine. + + ~ Now everything we need has been added to the log and we're almost ready + ~ to jump into it. It's inconvenient for code under the log-load transform + ~ to interact with the label system, so if we want to point + ~ main-input-buffer at a string, this is the place to do it. + + ~ The next layer is built now, so let's move on to it. + L@' litstring offset-to-target-address-space pack64 + ~ : -8 litstring #-- (codeword pointer) + s" quit" packstring 8 packalign + L@' log-load-find-execution-token offset-to-target-address-space pack64 + ~ : -8 log-load-find-execution-token #-- (codeword pointer) + L@' swap offset-to-target-address-space pack64 + ~ : -8 swap #-- (codeword pointer) + ~ : Get rid of that heap pointer on the stack, we're finally done with it! + L@' drop offset-to-target-address-space pack64 + ~ : -8 drop #-- (codeword pointer) + L@' execute offset-to-target-address-space pack64 + ~ : -8 execute #-- (codeword pointer) + ~ : deindent + ; + + +~ Where next? +~ ~~~~~~~~~~~ +~ +~ The definitions of "docol" and "exit" are very tightly bound up with the +~ execution model. They're defined and explained in language/static/core.e, +~ because they need to be part of the build process in two different ways, +~ like the other core functionality. + diff --git a/compilation/labels.e b/compilation/labels.e new file mode 100644 index 0000000..2a02a4c --- /dev/null +++ b/compilation/labels.e @@ -0,0 +1,296 @@ +~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~ ~~ Machine label facility ~~ +~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~ +~ Compilers and assemblers always have a need to resolve symbolic names for +~ parts of their code to numeric values. Usually, these symbolic names are +~ called labels. +~ +~ It's a surprisingly deep problem. The output itself can end up depending +~ on the actual values, for a variety of reasons. For example, the size of +~ a relative-jump instruction may vary depending on how far away its target +~ is, which changes the position of everything after. Or, there may be a size +~ limit on some particular segment and code may have to be entirely +~ reorganized to make things fit, splitting it into pieces or adding +~ trampolines. +~ +~ Often, this winds up being a hidden layer with immense complexity, +~ implemented as part of a linker, that even compiler maintainers find obscure +~ and confusing. +~ +~ It's Irenes' position that we should engage with complexity rather than +~ encysting it. So, in Evocation we take a different approach, heavily +~ inspired by the semantics of a tool called flatassembler. We provide +~ downstream code with the basic operations set-label and use-label which it +~ can combine in whatever order it wants. We also provide a harness, +~ label-loop, which takes an execution token for the downstream code +~ generator and runs it over and over, computing the mathematical fixed point +~ of the label assignments. +~ +~ On the first pass, label-loop guesses a value of zero for any label that's +~ used before it's set. On subsequent passes, each label starts with the value +~ it had on the previous pass. When we have a pass where labels end with the +~ same value they started with, we say they've converged, and we announce +~ success. If a hundred passes go by without convergence, we fail instead. +~ +~ Importantly, the label facility does not have an opinion on where or how +~ the values of labels should be output, nor does it have concepts of +~ dependency tracking or topological sorting. It only cares about the sequence +~ of set- and use-operations. Code relying on the facility is free to make +~ arbitrarily complex decisions about the layout, size, contents etc of its +~ output, and to use label values as part of those decisions. It is the +~ responsibility of this relying code to ensure the process converges, by +~ whatever means it wishes. +~ +~ Since the label facility is not directly involved in output, it is also +~ permissible for code relying on it to make retroactive changes to the +~ output, as long as all such changes are completed before returning control +~ to label-loop. +~ +~ Most of the time, the shorthand words L@' and L!' will be all you need to +~ use from your own code. + +~ The labels data structure is a linked list of dictionary entries, with +~ the same header format as the main Evocation dictionary, but instead of +~ being executable, each holds two words of data: status bits, and a value, +~ in that order. There's no "docol" pointer or anything of the sort. +~ +~ Status is a bit field: +~ bit zero is whether it's been used +~ bit one is whether it's been defined +~ bit two is whether it was used before being defined +~ bit three is whether the guessed value wound up equaling the actual value +~ +~ Just as the Evocation dictionary uses the global variable "latest" as a +~ handle (a pointer to a pointer) beginning a linked list of entries, so the +~ label dictionary also uses a handle variable, named "labels". We initialize +~ it here. +8 allocate s" labels" variable 0 labels ! + +~ This is analogous to word-heading, but prints label information. +~ (entry pointer --) +: label-heading + dup entry-to-name dup emitstring space + stringlen 1+ 54 swap - 0 max indent + entry-to-execution-token dup 8 + @ .hex64 space @ .hex8 + newline ; + +~ This prints the headings for all the labels that have been created. Note +~ that labels that have been created once stay in the dictionary forever, even +~ if subsequent passes neither use nor define them. That's because the control +~ loop has no way to know if they're still important; it's up to downstream +~ code to decide that. Everything has been carefully set up so that disused +~ labels don't hurt anything. +: list-labels + labels oldest-entry-in { dup } + { dup label-heading labels next-newer-entry-in } while drop ; + +~ This looks up a label by name if it exists, or returns 0 if it doesn't. +~ The return value if it's found is an entry pointer. +~ +~ Keep in mind that this is a pointer to the entry in the label dictionary, +~ not the label's value. It's common for a label's value to itself be a +~ pointer to a dictionary entry, in the program being compiled, so it's easy +~ to confuse that. +~ +~ (name string pointer -- label entry pointer) +: find-label labels swap find-in ; + +~ This creates a new label given a name for it, initializing its value and +~ status to zero and adding it to the label dictionary. This is responsible +~ for the initial guess of zero on the first pass. +~ +~ (name string pointer -- ) +: new-label labels create-in 0 , 0 , ; + +~ These helpers take a label entry pointer and return pointers to the status +~ and value fields. +: label-status entry-to-execution-token ; +: label-value entry-to-execution-token 8 + ; + +~ This looks up a label by name if it exists, or creates it if it doesn't. +~ Either way, it returns an entry pointer. It's named after the function +~ "intern" that many Lisp dialects have. +~ +~ (name string pointer -- label entry pointer) +: intern-label + dup find-label + dup { swap drop } + { drop dup new-label labels swap find-in } if-else ; + +~ This returns the value of a label, also doing all necessary status checks +~ and updates to keep track of the circumstances under which it was used. The +~ label loop relies on all read accesses going through this word. +~ +~ (label entry pointer -- label value) +: use-label + ~ If it hasn't been defined yet, mark it used-before-set. + dup label-status dup @ dup 0x02 & not + { 0x04 | swap ! } { drop drop } if-else + + ~ Mark it used. It's faster to just do this than to check if it's needed. + dup label-status dup @ 0x01 | swap ! + + ~ TODO putting a provide-label magic comment command here would be lovely + + ~ Return the guessed value. + label-value @ ; + +~ This overwrites the value of a label, also doing all necessary status +~ checks and updates to keep track of the cirumstances under which it was +~ set. The label loop relies on all write accesses going through this word. +~ +~ (new label value, label entry pointer --) +: set-label + dup label-status @ + ~ (new value, entry, initial status) + + ~ Exit, and print a diagnostic, if the label was already set on this pass. + ~ + ~ Without cooperation from user code, there's no great way to communicate + ~ up the call stack that execution should stop, so we don't try to. + dup 0x02 & { drop swap drop + ." Failed by attempting to redefine " + entry-to-name emitstring newline exit } if + + ~ We always set the defined bit to true. We leave the other status bits + ~ as-is, except that we check whether it's unused up to now, and whether + ~ the previous value equals the new value. If either of those holds, we set + ~ the guessed-equals-actual bit to true. + 0x02 | + ~ (new value, entry, updated status) + 3unroll dup 4 unroll label-value @ swap dup 3unroll = + ~ (entry, updated status, new value, equality) + 3roll dup 4 unroll 0x01 & not + ~ (entry, updated status, new value, equality, unused) + || { swap 0x08 | swap } if + ~ (entry, updated status, new value) + 3roll dup 3unroll + ~ (updated status, entry, new value, entry) + label-value ! label-status ! ; + +~ Since labels have dictionary entry headers but aren't in the main +~ dictionary, we get to use the entry flag bits for other things. The only one +~ we have right now is a feature that acts as a hint to anything trying to +~ display metadata about labels, to the effect that this one represents +~ something other than an offset in the file. +~ +~ (label entry pointer --) +: set-label-non-offset dup entry-flags@ 0x01 | swap entry-flags! ; +~ (label entry pointer -- boolean) +: is-label-non-offset entry-flags@ 0x01 & ; + +~ This is a convenience helper which downstream code can use to check how +~ many bytes it has output thus-far. +~ (output memory start, current output point +~ -- output memory start, current output point, offset) +: current-offset 2dup swap - ; + +~ This is a concise syntax for referencing the entry pointer of a label, +~ when you know its name statically. It reads a word of text input and calls +~ intern-label on it. +: L' + word value@ + interpreter-flags @ 0x01 & + { ' litstring entry-to-execution-token , + here @ swap packstring 8 packalign here ! + ' intern-label entry-to-execution-token , + dropstring } + { intern-label dropstring-with-result } if-else + ; make-immediate + +~ This is a concise syntax for calling use-label with a label whose name +~ you know statically. It performs L', then calls use-label. +: L@' + ' L' entry-to-execution-token execute + interpreter-flags @ 1 & + { ' use-label entry-to-execution-token , } + { use-label } if-else + ; make-immediate + +~ This is a concise syntax for calling set-label with a label whose name +~ you know statically. It performs L', then calls set-label. +: L!' + ' L' entry-to-execution-token execute + interpreter-flags @ 1 & + { ' set-label entry-to-execution-token , } + { set-label } if-else + ; make-immediate + +~ This is an internal helper that label-loop uses to check if a specific +~ label entry has "converged", based on its status bits. Downstream code won't +~ need to call this directly. +~ +~ For a label to have "converged", at least one of the following must be +~ true: +~ +~ 1. The label must never have been used (bit zero clear); +~ 2. The label was both used and defined, but not used before it was defined +~ (bits zero and one set; bit two clear); +~ 3. The label was both used and defined, and the guessed value equalled the +~ actual value (bits zero, one, and three set). +~ +~ (label entry pointer -- boolean) +: check-label-converged + label-status @ + dup 0x01 & not swap + dup 0x03 & 0x03 = swap dup 0x04 & not swap 3unroll && swap + 0x0b & 0x0b = + || || ; + +~ This is an internal helper that label-loop uses to check if the overall +~ label assignments have converged, based on their status bits. Downstream +~ code won't need to call this directly. +~ +~ This returns true if and only if check-label-converged returns true for +~ all the labels that have been created. +~ +~ (-- boolean) +: check-labels-converged + 1 + labels @ { dup } + { dup check-label-converged 3roll && swap + @ } while drop ; + +~ This is an internal helper that label-loop calls between passes, to update +~ the status bits and get ready for the next one. Downstream code won't need +~ to call this directly. +~ +: reset-labels + labels @ { dup } + { dup label-status 0 swap ! + @ } while drop ; + +~ This is the top-level word that invokes the entire label system. All the +~ code generation happens inside it. +~ +~ The execution token it's passed should have the interface: +~ +~ (output start, current output pointer +~ -- output start, current output pointer) +~ +~ In general, this is the same interface that code generation words should +~ use to communicate with each other. For example, all the words in +~ compilation/elf.e use this interface. +~ +~ (execution token -- output start, output length) +: label-loop + 0 swap + ~ TODO every time you double this to fix a crash, you must publicly + ~ apologize for deferring a real fix. those are the rules + 0x80000 allocate dup + ~ (iteration count, execution token, output start, output point) + { 3 pick 100 > } + { 2 pick execute 4 roll 1+ 4 unroll + + check-labels-converged + { 4 roll drop + 3 roll drop + current-offset swap drop + exit } if + drop dup + reset-labels } while + ~ TODO run some analysis on the labels to detect labels that are never + ~ defined, but are referenced. List them. + drop drop drop ." Failed after " . ." iterations." newline ; + diff --git a/compilation/transform.e b/compilation/transform.e new file mode 100644 index 0000000..80ef98b --- /dev/null +++ b/compilation/transform.e @@ -0,0 +1,6512 @@ +~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~ ~~ Code transformation facility ~~ +~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~ +~ The process of producing an executable binary out of Evocation involves +~ various bootstrapping phases during which code operates under different +~ constraints, and must be written with different styles. In some cases, +~ substantially the same code must be output multiple times in slightly +~ different ways, and it would be both arduous and verbose to write each of +~ these directly. +~ +~ To solve this problem, this file implements a concept of code +~ transformation. There are two transforms, the label transform and the +~ log-load transform, each of which takes a string containing Evocation source +~ code and produces compiled code that has been modified to operate in a +~ specific way. The transforms rely on the label facility provided by +~ compilation/labels.e, and expect to run from within label-loop. +~ +~ The label transform produces code that uses one label per word it defines, +~ to statically reference everything. Thus, when output to an executable +~ binary, this code will function without external dependencies. The tradeoff +~ is that it has no way to reference data that exists only at runtime. +~ +~ The log-load transform relies on labels, but doesn't add any of its own. +~ It produces a compiled routine which, when run, dynamically looks up all the +~ references in the log, and appends the original code to the log. This adds +~ work that must be done when the runtime starts up, but the benefit is that +~ it can reference data that doesn't exist at compile-time. Most crucially, +~ it can reference the "here" and "latest" pointers in the log, which are +~ required for all the usual word-definition stuff to work, and whose +~ addresses are not known until runtime. +~ +~ The log-load transform may also be useful for experimental tasks such as +~ creating additional, independent logs, or injecting Evocation into another +~ process's address space. +~ +~ Please notice that both these transforms, in different ways, navigate the +~ same underlying design tension: The Forth compilation model hardcodes +~ references at the time compilation happens, and Evocation makes the choice +~ to not decide the address of the log until runtime. Thus the label transform +~ can't be sufficient on its own. Other Forths avoid this problem by +~ hardcoding an address for the log, or by using OS-provided load-time +~ symbol relocation. Evocation, however, does it on hard mode, mostly for fun. +~ +~ Because it was clear from early on that the label transform couldn't stand +~ alone, and that another one would be necessary, we've refrained from adding +~ too many features to it. Since we have multiple transforms, they should each +~ be kept simple and well-defined, so that they can be composed in creative +~ new ways down the line. When adding additional behavior, always give thought +~ to whether it belongs in an existing transform or a new one. +~ +~ +~ About the label transform +~ ~~~~~~~~~~~~~~~~~~~~~~~~~ +~ +~ The label transform operates on code that compiles itself, and ensures +~ that the result of the compilation is suitable to be included in an +~ executable binary as words that are statically referenced by their +~ addresses. To achieve this, it causes each newly-defined word to have a +~ corresponding label whose value is the offset of its codeword, and it causes +~ all compiled invocations of other words to be resolved by using these labels. +~ The label transform is suitable for code that must be directly invoked by +~ the warm-start routine provided by compilation/execution.e. +~ +~ The most fundamental technique the label transform performs is to separate +~ words that run in compile mode from words that run immediately. There is no +~ distinction made between words running in immediate mode, and words declared +~ as immediate. Immediate words are looked up and executed based on their +~ "real", currently-executing definitions. Compiled words, including +~ literals, are looked up via the label facility. +~ +~ Since the label facility is able to resolve forward references, there is +~ no hard requirement that everything in the file be topologically sorted. +~ However, the transform will refuse to create forward references to compiled +~ words. If you want them, you can create them by hand by calling use-label +~ yourself. This restriction is in place because allowing forward references +~ would be a significant difference from un-transformed code that could easily +~ become confusing, and because it simplifies the implementation a bit. +~ +~ Compilation words do make extensive reference to the global variables +~ "here" and "latest". In particular, flow-control words such as if-else +~ expect the log to have recent compilation outputs on it, and to be able to +~ mutate them in-place. In order to make this work, we provide temporary +~ values of these two variables which point to the location of the output +~ buffer. This allows pointer resolution to work correctly without additional +~ effort, but notice that the buffer's address will differ from the address +~ the resulting program loads itself at. There's no simple way to avoid this +~ concern, since the variables must point to one of those addresses or the +~ other, not both. +~ +~ We resolve the issue by running our own, alternate versions of most of the +~ critical word-defining words, including for example "create", ":", and ";". +~ These alternates run instead of the normal versions of these words, and use +~ the label facility to compute the addresses that will be needed at runtime. +~ The code being compiled is responsible for not doing anything else that +~ would rely on "here" and "latest" matching their runtime addresses, though +~ it is otherwise allowed to modify and rely on them in all the usual ways. +~ The alternate versions are defined in this file as their own words, +~ "label-create-alternate" and so on. +~ +~ Note that these alternates are applied via a purely lexical +~ transformation: when a word would be looked up in the dictionary to +~ interpret, first check if it's one of these. That means the transformation +~ won't apply to indirect callers of these words, nor to tick-quotes of them. +~ The code being compiled is responsible for not doing either of those things. +~ +~ Notably, the transformation uses the same "interpreter-flags" variable as +~ the rest of Evocation. There's no need to keep it separate like there is +~ with the other variables. This makes it easy to change modes. +~ +~ The label transformation and its alternates rely on various labels, all of +~ which must be defined elsewhere, lest the label loop fail to converge: +~ "lit", "origin", "docol", "exit", ":", ";", and ";asm". +~ +~ All of these limitations result in the compiled code being, in effect, +~ written in a dialect which is like Evocation, but more restricted. This is +~ acceptable, because the label transform is intended for compiling code that +~ is an early part of Evocation itself, and the necessary code has all been +~ written to follow these restrictions. +~ +~ +~ About the log-load transform +~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~ +~ The log-load transform also operates on code that compiles itself; it +~ produces a compiled routine which, when run, appends the original code to +~ the log. As the routine is run, each reference to another word is resolved +~ by looking up the name of the target word in the log. Furthermore, these +~ lookups are done using log-load-find, defined in language/static/log-load.e, +~ which accepts a pointer to the log's base address as a parameter. See that +~ file for more explanation of what the log is and why it's important. Thus, +~ unlike normal accesses to the log, this routine doesn't rely on already +~ having the log's base address hardcoded into it at the time of its own +~ compilation. The log-load transform is suitable for implementing the core +~ responsibilities of the warm-start routine provided by +~ compilation/execution.e, relying on only a few specific words that it +~ statically references via labels. +~ +~ Much like the label transform, the log-load transform provides alternate +~ versions of certain immediate words used in word definition. Also like the +~ label transform, it provides its own copies of "here" and "latest". +~ +~ The log-load transform provides alternates for a significantly broader set +~ of words than the label transform, including all the flow-control words such +~ as if-else. It runs its own alternates immediately, but unlike the label +~ transform, immediate execution for the log-load transform is not actually +~ immediate; it is compiled into words which will have those immediate effects +~ at the time the generated routine is run. The generated routine can itself +~ be thought of as a compilation process, producing its output on the log, so +~ doing things later for us still means doing them immediately during the +~ routine. +~ +~ The log-load transform does impose a no-forward-references requirement, +~ though it is applied at the time the routine is run, rather than at the time +~ of the transformation. +~ +~ The log-load transformation and its alternates rely on the following +~ labels, all of which must be defined elsewhere: TODO +~ +~ +~ About the hex transform +~ ~~~~~~~~~~~~~~~~~~~~~~~ +~ +~ The hex transform's role is a bit different. Whereas the label and +~ log-load transforms are used as part of generating an executable binary, the +~ hex transform produces a commented hex dump that, when later processed by +~ the "hex" tool, produces that same binary. It achieves this by modifying +~ various words which are core parts of Evocation to keep track of appropriate +~ comments and other metadata, and modifying the output facility to output a +~ hex dump with any attached comments, rather than raw bytes. +~ +~ By its nature, the hex transform needs to be able to cope with other +~ transforms running inside itself; that is, the transformation facility needs +~ to be reentrant. The other transforms don't have to cope with that, since +~ the code they run is mostly just about creating word definitions, but the +~ code given to the hex transform will normally include a call to label-loop, +~ all the memory manipulation that happens as part of binary generation, and +~ various output words. +~ +~ While the label and log-load transforms allow Evocation to be a +~ self-hosting compiler, the hex transform attains the even higher bar of +~ making Evocation a "self-bootstrapping" compiler, able to create a running +~ copy of itself without having any sort of pre-existing compiler at all, only +~ a hex converter or similar small tool. The idea is that the hex converter is +~ small and simple enough that it's easy to perform a BINARY audit on, and +~ that there need be no other binary artifacts in the chain of trust. +~ +~ This is not a new concept, but as far as Irenes are aware this name for it +~ is a new coinage. The approach of using a hex-to-binary conversion tool as +~ the initial bootstrapping stage is due to MesCC, which is quite inspiring in +~ that regard. +~ +~ The key insight is that, some sense, the difference between source code +~ and binary is the ability to have comments. +~ +~ The hex transform keeps some large, complex state as it runs, unlike the +~ label and log-load transforms, which only require very simple state. This +~ state is called the "output metadata". Entries in the output metadata are +~ created and modified as the code under transform runs, by "magic comments" +~ embedded within it. There is a complex command language for these comments, +~ which must be understood to write code designed to make good use of the hex +~ transform. Fortunately, that's really just assembly-language +~ implementations. +~ +~ It's quite an intense topic, and many readers may wish only to know that +~ the hex transform exists, not how it's structured. So, details on the output +~ metadata and the magic comments are described below, under "hex transform +~ implementation", rather than here. +~ +~ The hex transform DOES NOT WORK yet. It's still in development. +~ TODO update this note when it does work + + +~ Address-management helpers +~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~ +~ The facilities in this section are used as helper code in the +~ implementations of all three transforms. + + +~ We have a bunch of accessors for the transformation state structure, which +~ are all functions from pointers to pointers. We also define a global +~ variable which points to the transformation state. +: transformation-state-saved-here ; +: transformation-state-saved-latest 8 + ; +: transformation-state-output-buffer-start 2 8 * + ; +: transformation-state-user-stack-depth 3 8 * + ; +: transformation-state-label-scratch 4 8 * + ; +: transformation-state-output-metadata 5 8 * + ; +: allocate-transformation-state + 6 8 * allocate + dup transformation-state-saved-here 0 swap ! + dup transformation-state-saved-latest 0 swap ! + dup transformation-state-output-buffer-start 0 swap ! + dup transformation-state-user-stack-depth 0 swap ! + dup transformation-state-label-scratch 0 swap ! + dup transformation-state-output-metadata 0 swap ! ; +allocate-transformation-state s" transformation-state" variable + + +~ When calling the label facility during a transformation, it's necessary +~ to use the real, non-wrapped "here" and "latest". +: swap-transform-variables + here @ transformation-state transformation-state-saved-here @ + here ! transformation-state transformation-state-saved-here ! + latest @ transformation-state transformation-state-saved-latest @ + latest ! transformation-state transformation-state-saved-latest ! ; + +~ We deal with a few address spaces. There's the "host" address space, the +~ space this process performing the compilation is using for itself. There's +~ the "target" address space, the address space that will exist later, when +~ the program we've compiled is running. +~ +~ Then there's "offsets", which are relative to the start of the output +~ buffer. For clarity's sake, we always refer to these as offsets, rather than +~ as addresses. +~ +~ When we define labels for compiled words, we set their values to be +~ offsets pointing to the generated codeword. This is done by +~ label-create-alternate. We then need to convert them either to the host or +~ the target address space, depending on how we're using them. +~ +~ There's no approach here that isn't confusing, but the hope is that by +~ using offsets, so that we always have to convert them regardless of what +~ we're doing with them, we won't miss a spot where conversion needs to +~ happen. +~ +~ (output offset -- target address) +: offset-to-target-address-space + ~ Don't transform null pointers. + dup { swap-transform-variables L@' origin swap-transform-variables + } if ; + +~ (target address -- output offset) +: target-address-space-to-offset + ~ Don't transform null pointers. + dup { swap-transform-variables L@' origin swap-transform-variables - } if ; + +~ (output offset -- host address) +: offset-to-host-address-space + ~ Don't transform null pointers + dup { transformation-state transformation-state-output-buffer-start @ + } if ; + +~ (host address --output offset) +: host-address-space-to-offset + ~ Don't transform null pointers + dup { transformation-state transformation-state-output-buffer-start @ - } if ; + +~ (host address inside the output buffer -- target address) +: host-address-space-to-target + host-address-space-to-offset offset-to-target-address-space ; + +~ (target address -- host address) +: target-address-space-to-host + target-address-space-to-offset offset-to-host-address-space ; + + +: describe-transformation + ." active here " here @ .hex64 space + ." latest " latest @ .hex64 newline + ." saved here " transformation-state transformation-state-saved-here + @ .hex64 space + ." latest " transformation-state transformation-state-saved-latest + @ .hex64 newline + ." output start " + transformation-state transformation-state-output-buffer-start + @ .hex64 newline + ." user stack depth " + transformation-state transformation-state-user-stack-depth + @ .hex64 newline ; + + +~ User stack depth tracking helper +~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~ +~ The transformation facility offers a feature which specific transforms can +~ use, or not, as they prefer, called user stack depth tracking. The log-load +~ transform uses it; the label transform does not. The most difficult part of +~ the tracking is the heler word below; it also relies on the user-stack-depth +~ field of the transformation-state variable. +~ +~ What does user stack depth mean? Well, for every transform there's some +~ notion of running a word "immediately"; for the log-load transform, it means +~ that the word runs at the time the generated log-load routine executes. Each +~ time a word runs, it interacts with the value stack in some way. Some words +~ are neutral; others result in a net increase or decrease in its depth, which +~ we call the delta. The running total of these deltas is what we call the +~ user stack depth. By "user" we mean that these are values which were created +~ by and for the code being transformed, which the code doing the transforming +~ must be careful not to interfere with. +~ +~ Why do we need to do this? Recall that the log-load routine is given the +~ log address as the only value on the stack when it begins, and any time it +~ does something that requires looking at the log, it needs to be able to find +~ that address. The log-load routine runs as part of the warm-start routine +~ (see compilation/execution.e), whose overall job is to get Evocation into a +~ state where it can freely work with dynamically-allocated data on the log +~ whose address isn't known at compile time. Therefore, necessarily, the +~ log-load routine itself cannot assume access to any scratch space; the value +~ stack is the only place it can store anything it needs, most crucially +~ including the log address. +~ +~ Fortunately, it is possible for the transform's own generated code to +~ cooperate with the transformed code so that they can both share the stack, +~ as long as the log transform knows what the transformed code is doing with +~ it. Unfortunately, there's no general way to know that. We achieve it by +~ hard-coding a delta value for every single possible word, and looking up +~ that delta at transform-time, based on the word's name. Yes, that is quite a +~ thing to have to do, but it does mean we don't have to think about this +~ while writing the code we're transforming. +~ +~ It's worth noticing that the delta only applies to things that happen +~ immediately. It does not apply to compiled words because they run later, and +~ it does not apply to alternates because they run too soon. Feel free to +~ contemplate how confusing it is that transformation-time is sooner than +~ "immediate". +~ +~ (name pointer -- delta value) +: transform-get-stack-delta + ~ From language/static/core.e. + dup s" docol" stringcmp 0 = { drop 1 exit } if + dup s" exit" stringcmp 0 = { drop 0 exit } if + dup s" swap" stringcmp 0 = { drop 0 exit } if + dup s" drop" stringcmp 0 = { drop -1 exit } if + dup s" 2drop" stringcmp 0 = { drop -2 exit } if + ~ The following are deliberate omissions: roll, unroll. + dup s" 3roll" stringcmp 0 = { drop 0 exit } if + dup s" 3unroll" stringcmp 0 = { drop 0 exit } if + dup s" dup" stringcmp 0 = { drop 1 exit } if + dup s" 2dup" stringcmp 0 = { drop 2 exit } if + dup s" +" stringcmp 0 = { drop -1 exit } if + dup s" -" stringcmp 0 = { drop -1 exit } if + dup s" *" stringcmp 0 = { drop -1 exit } if + dup s" /%" stringcmp 0 = { drop 0 exit } if + dup s" 1+" stringcmp 0 = { drop 0 exit } if + dup s" 1-" stringcmp 0 = { drop 0 exit } if + dup s" =" stringcmp 0 = { drop -1 exit } if + dup s" !=" stringcmp 0 = { drop -1 exit } if + dup s" >" stringcmp 0 = { drop -1 exit } if + dup s" <" stringcmp 0 = { drop -1 exit } if + dup s" >=" stringcmp 0 = { drop -1 exit } if + dup s" <=" stringcmp 0 = { drop -1 exit } if + dup s" >unsigned" stringcmp 0 = { drop -1 exit } if + dup s" =unsigned" stringcmp 0 = { drop -1 exit } if + dup s" <=unsigned" stringcmp 0 = { drop -1 exit } if + dup s" &" stringcmp 0 = { drop -1 exit } if + dup s" |" stringcmp 0 = { drop -1 exit } if + dup s" xor" stringcmp 0 = { drop -1 exit } if + dup s" invert" stringcmp 0 = { drop 0 exit } if + dup s" lit" stringcmp 0 = { drop 1 exit } if + dup s" litstring" stringcmp 0 = { drop 1 exit } if + dup s" !" stringcmp 0 = { drop -2 exit } if + dup s" @" stringcmp 0 = { drop 0 exit } if + dup s" +!" stringcmp 0 = { drop -2 exit } if + dup s" -!" stringcmp 0 = { drop -2 exit } if + dup s" 8!" stringcmp 0 = { drop -2 exit } if + dup s" 8@" stringcmp 0 = { drop 0 exit } if + dup s" 16!" stringcmp 0 = { drop -2 exit } if + dup s" 16@" stringcmp 0 = { drop 0 exit } if + dup s" 32!" stringcmp 0 = { drop -2 exit } if + dup s" 32@" stringcmp 0 = { drop 0 exit } if + dup s" control!" stringcmp 0 = { drop -1 exit } if + dup s" control@" stringcmp 0 = { drop 1 exit } if + dup s" value!" stringcmp 0 = { drop -1 exit } if + dup s" value@" stringcmp 0 = { drop 1 exit } if + dup s" memcopy" stringcmp 0 = { drop -3 exit } if + dup s" memmove" stringcmp 0 = { drop -3 exit } if + dup s" stringlen" stringcmp 0 = { drop 0 exit } if + dup s" reverse-stringlen" stringcmp 0 = { drop 0 exit } if + dup s" reverse-padding-len" stringcmp 0 = { drop 0 exit } if + dup s" stringcmp" stringcmp 0 = { drop -1 exit } if + dup s" branch" stringcmp 0 = { drop 0 exit } if + dup s" 0branch" stringcmp 0 = { drop 0 exit } if + ~ This is kind of a big assumption. + dup s" execute" stringcmp 0 = { drop -1 exit } if + dup s" entry-to-execution-token" stringcmp 0 = { drop 0 exit } if + dup s" execution-token-to-entry" stringcmp 0 = { drop 0 exit } if + dup s" entry-flags@" stringcmp 0 = { drop 0 exit } if + dup s" entry-flags!" stringcmp 0 = { drop -2 exit } if + dup s" entry-to-name" stringcmp 0 = { drop 0 exit } if + dup s" pack64" stringcmp 0 = { drop -1 exit } if + dup s" pack32" stringcmp 0 = { drop -1 exit } if + dup s" pack16" stringcmp 0 = { drop -1 exit } if + dup s" pack8" stringcmp 0 = { drop -1 exit } if + dup s" packstring" stringcmp 0 = { drop -1 exit } if + dup s" packalign" stringcmp 0 = { drop -1 exit } if + dup s" unpack64" stringcmp 0 = { drop 1 exit } if + dup s" unpack32" stringcmp 0 = { drop 1 exit } if + dup s" unpack16" stringcmp 0 = { drop 1 exit } if + dup s" unpack8" stringcmp 0 = { drop 1 exit } if + dup s" align-size" stringcmp 0 = { drop -1 exit } if + dup s" unpackalign" stringcmp 0 = { drop -1 exit } if + dup s" crash" stringcmp 0 = { drop 0 exit } if + dup s" max" stringcmp 0 = { drop -1 exit } if + dup s" min" stringcmp 0 = { drop -1 exit } if + dup s" over" stringcmp 0 = { drop 1 exit } if + dup s" pick" stringcmp 0 = { drop 0 exit } if + ~ The following are deliberate omissions: ndrop, ndup. + dup s" 3drop" stringcmp 0 = { drop -3 exit } if + dup s" 3dup" stringcmp 0 = { drop 3 exit } if + dup s" &&" stringcmp 0 = { drop -1 exit } if + dup s" ||" stringcmp 0 = { drop -1 exit } if + dup s" not" stringcmp 0 = { drop 0 exit } if + dup s" negate" stringcmp 0 = { drop 0 exit } if + dup s" align-floor" stringcmp 0 = { drop -1 exit } if + + ~ From language/static/linux.e. + dup s" sys-exit" stringcmp 0 = { drop -1 exit } if + dup s" sys-write" stringcmp 0 = { drop -2 exit } if + dup s" sys-read" stringcmp 0 = { drop -2 exit } if + + ~ From language/static/output.e. + dup s" emitstring" stringcmp 0 = { drop -1 exit } if + dup s" space" stringcmp 0 = { drop 0 exit } if + dup s" newline" stringcmp 0 = { drop 0 exit } if + dup s" pow" stringcmp 0 = { drop 2 exit } if + dup s" logfloor" stringcmp 0 = { drop 2 exit } if + dup s" logceil" stringcmp 0 = { drop 2 exit } if + dup s" .base-unsigned" stringcmp 0 = { drop -3 exit } if + dup s" .base" stringcmp 0 = { drop -2 exit } if + dup s" ." stringcmp 0 = { drop -1 exit } if + dup s" .hex" stringcmp 0 = { drop -1 exit } if + dup s" .hex8" stringcmp 0 = { drop -1 exit } if + dup s" .hex16" stringcmp 0 = { drop -1 exit } if + dup s" .hex32" stringcmp 0 = { drop -1 exit } if + dup s" .hex64" stringcmp 0 = { drop -1 exit } if + dup s" .hexn" stringcmp 0 = { drop -2 exit } if + + ~ From language/static/amd64.e. + dup s" :rax" stringcmp 0 = { drop 1 exit } if + dup s" :rcx" stringcmp 0 = { drop 1 exit } if + dup s" :rdx" stringcmp 0 = { drop 1 exit } if + dup s" :rbx" stringcmp 0 = { drop 1 exit } if + dup s" :rsp" stringcmp 0 = { drop 1 exit } if + dup s" :rbp" stringcmp 0 = { drop 1 exit } if + dup s" :rsi" stringcmp 0 = { drop 1 exit } if + dup s" :rdi" stringcmp 0 = { drop 1 exit } if + dup s" :r8" stringcmp 0 = { drop 1 exit } if + dup s" :r9" stringcmp 0 = { drop 1 exit } if + dup s" :r10" stringcmp 0 = { drop 1 exit } if + dup s" :r11" stringcmp 0 = { drop 1 exit } if + dup s" :r12" stringcmp 0 = { drop 1 exit } if + dup s" :r13" stringcmp 0 = { drop 1 exit } if + dup s" :r14" stringcmp 0 = { drop 1 exit } if + dup s" :r15" stringcmp 0 = { drop 1 exit } if + dup s" :eax" stringcmp 0 = { drop 1 exit } if + dup s" :ecx" stringcmp 0 = { drop 1 exit } if + dup s" :edx" stringcmp 0 = { drop 1 exit } if + dup s" :ebx" stringcmp 0 = { drop 1 exit } if + dup s" :esp" stringcmp 0 = { drop 1 exit } if + dup s" :ebp" stringcmp 0 = { drop 1 exit } if + dup s" :esi" stringcmp 0 = { drop 1 exit } if + dup s" :edi" stringcmp 0 = { drop 1 exit } if + dup s" :ax" stringcmp 0 = { drop 1 exit } if + dup s" :cx" stringcmp 0 = { drop 1 exit } if + dup s" :dx" stringcmp 0 = { drop 1 exit } if + dup s" :bx" stringcmp 0 = { drop 1 exit } if + dup s" :sp" stringcmp 0 = { drop 1 exit } if + dup s" :bp" stringcmp 0 = { drop 1 exit } if + dup s" :si" stringcmp 0 = { drop 1 exit } if + dup s" :di" stringcmp 0 = { drop 1 exit } if + dup s" :al" stringcmp 0 = { drop 1 exit } if + dup s" :cl" stringcmp 0 = { drop 1 exit } if + dup s" :dl" stringcmp 0 = { drop 1 exit } if + dup s" :bl" stringcmp 0 = { drop 1 exit } if + dup s" :ah" stringcmp 0 = { drop 1 exit } if + dup s" :ch" stringcmp 0 = { drop 1 exit } if + dup s" :dh" stringcmp 0 = { drop 1 exit } if + dup s" :bh" stringcmp 0 = { drop 1 exit } if + dup s" :cc-overflow" stringcmp 0 = { drop 1 exit } if + dup s" :cc-no-overflow" stringcmp 0 = { drop 1 exit } if + dup s" :cc-below" stringcmp 0 = { drop 1 exit } if + dup s" :cc-above-equal" stringcmp 0 = { drop 1 exit } if + dup s" :cc-equal" stringcmp 0 = { drop 1 exit } if + dup s" :cc-not-equal" stringcmp 0 = { drop 1 exit } if + dup s" :cc-below-equal" stringcmp 0 = { drop 1 exit } if + dup s" :cc-above" stringcmp 0 = { drop 1 exit } if + dup s" :cc-sign" stringcmp 0 = { drop 1 exit } if + dup s" :cc-not-sign" stringcmp 0 = { drop 1 exit } if + dup s" :cc-even" stringcmp 0 = { drop 1 exit } if + dup s" :cc-odd" stringcmp 0 = { drop 1 exit } if + dup s" :cc-less" stringcmp 0 = { drop 1 exit } if + dup s" :cc-greater-equal" stringcmp 0 = { drop 1 exit } if + dup s" :cc-less-equal" stringcmp 0 = { drop 1 exit } if + dup s" :cc-greater" stringcmp 0 = { drop 1 exit } if + dup s" reg64" stringcmp 0 = { drop 0 exit } if + dup s" extrareg64" stringcmp 0 = { drop 0 exit } if + dup s" reg32" stringcmp 0 = { drop 0 exit } if + dup s" reg16" stringcmp 0 = { drop 0 exit } if + dup s" reg8" stringcmp 0 = { drop 0 exit } if + dup s" scalefield" stringcmp 0 = { drop 0 exit } if + dup s" condition-code" stringcmp 0 = { drop 0 exit } if + dup s" rex-0" stringcmp 0 = { drop 0 exit } if + dup s" rex-w" stringcmp 0 = { drop 0 exit } if + dup s" rex-r" stringcmp 0 = { drop 0 exit } if + dup s" rex-x" stringcmp 0 = { drop 0 exit } if + dup s" rex-b" stringcmp 0 = { drop 0 exit } if + dup s" rex-wr" stringcmp 0 = { drop 0 exit } if + dup s" rex-wx" stringcmp 0 = { drop 0 exit } if + dup s" rex-wb" stringcmp 0 = { drop 0 exit } if + dup s" rex-rx" stringcmp 0 = { drop 0 exit } if + dup s" rex-rb" stringcmp 0 = { drop 0 exit } if + dup s" rex-xb" stringcmp 0 = { drop 0 exit } if + dup s" rex-wrx" stringcmp 0 = { drop 0 exit } if + dup s" rex-wrb" stringcmp 0 = { drop 0 exit } if + dup s" rex-wxb" stringcmp 0 = { drop 0 exit } if + dup s" rex-rxb" stringcmp 0 = { drop 0 exit } if + dup s" rex-wrxb" stringcmp 0 = { drop 0 exit } if + dup s" opcodereg" stringcmp 0 = { drop -2 exit } if + dup s" opcodecc" stringcmp 0 = { drop -2 exit } if + dup s" modrm" stringcmp 0 = { drop -3 exit } if + dup s" sib" stringcmp 0 = { drop -3 exit } if + dup s" addressing-reg64" stringcmp 0 = { drop -2 exit } if + dup s" addressing-reg8" stringcmp 0 = { drop -2 exit } if + dup s" addressing-indirect-reg64" stringcmp 0 = { drop -2 exit } if + dup s" addressing-disp8-reg64" stringcmp 0 = { drop -3 exit } if + dup s" addressing-disp32-reg64" stringcmp 0 = { drop -3 exit } if + dup s" addressing-indexed-reg64" stringcmp 0 = { drop -4 exit } if + dup s" addressing-disp8-indexed-reg64" stringcmp 0 = { drop -5 exit } if + dup s" cld" stringcmp 0 = { drop 0 exit } if + dup s" std" stringcmp 0 = { drop 0 exit } if + dup s" syscall" stringcmp 0 = { drop 0 exit } if + dup s" hlt" stringcmp 0 = { drop 0 exit } if + dup s" push-reg64" stringcmp 0 = { drop -1 exit } if + dup s" push-extrareg64" stringcmp 0 = { drop -1 exit } if + dup s" pop-reg64" stringcmp 0 = { drop -1 exit } if + dup s" pop-extrareg64" stringcmp 0 = { drop -1 exit } if + dup s" push-imm32-extended64" stringcmp 0 = { drop -1 exit } if + dup s" lea-reg64-disp8-reg64" stringcmp 0 = { drop -3 exit } if + dup s" lea-reg64-disp32-reg64" stringcmp 0 = { drop -3 exit } if + dup s" lea-reg64-indexed-reg64" stringcmp 0 = { drop -4 exit } if + dup s" lea-reg64-disp8-indexed-reg64" stringcmp 0 = { drop -5 exit } if + dup s" mov-reg64-imm32" stringcmp 0 = { drop -2 exit } if + dup s" mov-reg64-imm64" stringcmp 0 = { drop -2 exit } if + dup s" mov-extrareg64-imm64" stringcmp 0 = { drop -2 exit } if + dup s" mov-reg64-reg64" stringcmp 0 = { drop -2 exit } if + dup s" mov-indirect-reg64-reg64" stringcmp 0 = { drop -2 exit } if + dup s" mov-disp8-reg64-reg64" stringcmp 0 = { drop -3 exit } if + dup s" mov-reg64-indirect-reg64" stringcmp 0 = { drop -2 exit } if + dup s" mov-reg64-disp8-reg64" stringcmp 0 = { drop -3 exit } if + dup s" mov-reg64-disp32-reg64" stringcmp 0 = { drop -3 exit } if + dup s" mov-reg64-indexed-reg64" stringcmp 0 = { drop -4 exit } if + dup s" mov-indexed-reg64-reg64" stringcmp 0 = { drop -4 exit } if + dup s" mov-indirect-reg64-reg32" stringcmp 0 = { drop -2 exit } if + dup s" mov-disp8-reg64-reg32" stringcmp 0 = { drop -3 exit } if + dup s" mov-reg32-indirect-reg64" stringcmp 0 = { drop -2 exit } if + dup s" mov-reg32-disp8-reg64" stringcmp 0 = { drop -3 exit } if + dup s" mov-indirect-reg64-reg16" stringcmp 0 = { drop -2 exit } if + dup s" mov-disp8-reg64-reg16" stringcmp 0 = { drop -3 exit } if + dup s" mov-reg16-indirect-reg64" stringcmp 0 = { drop -2 exit } if + dup s" mov-reg16-disp8-reg64" stringcmp 0 = { drop -3 exit } if + dup s" mov-indirect-reg64-reg8" stringcmp 0 = { drop -2 exit } if + dup s" mov-disp8-reg64-reg8" stringcmp 0 = { drop -3 exit } if + dup s" mov-reg8-indirect-reg64" stringcmp 0 = { drop -2 exit } if + dup s" mov-reg8-disp8-reg64" stringcmp 0 = { drop -3 exit } if + dup s" mov-reg8-reg8" stringcmp 0 = { drop -2 exit } if + dup s" movs8" stringcmp 0 = { drop 0 exit } if + dup s" movs16" stringcmp 0 = { drop 0 exit } if + dup s" movs32" stringcmp 0 = { drop 0 exit } if + dup s" movs64" stringcmp 0 = { drop 0 exit } if + dup s" rep-movs8" stringcmp 0 = { drop 0 exit } if + dup s" rep-movs16" stringcmp 0 = { drop 0 exit } if + dup s" rep-movs32" stringcmp 0 = { drop 0 exit } if + dup s" rep-movs64" stringcmp 0 = { drop 0 exit } if + dup s" lods8" stringcmp 0 = { drop 0 exit } if + dup s" lods16" stringcmp 0 = { drop 0 exit } if + dup s" lods32" stringcmp 0 = { drop 0 exit } if + dup s" lods64" stringcmp 0 = { drop 0 exit } if + dup s" rep-lods8" stringcmp 0 = { drop 0 exit } if + dup s" rep-lods16" stringcmp 0 = { drop 0 exit } if + dup s" rep-lods32" stringcmp 0 = { drop 0 exit } if + dup s" rep-lods64" stringcmp 0 = { drop 0 exit } if + dup s" stos8" stringcmp 0 = { drop 0 exit } if + dup s" stos16" stringcmp 0 = { drop 0 exit } if + dup s" stos32" stringcmp 0 = { drop 0 exit } if + dup s" stos64" stringcmp 0 = { drop 0 exit } if + dup s" rep-stos8" stringcmp 0 = { drop 0 exit } if + dup s" rep-stos16" stringcmp 0 = { drop 0 exit } if + dup s" rep-stos32" stringcmp 0 = { drop 0 exit } if + dup s" rep-stos64" stringcmp 0 = { drop 0 exit } if + dup s" cmps8" stringcmp 0 = { drop 0 exit } if + dup s" cmps16" stringcmp 0 = { drop 0 exit } if + dup s" cmps32" stringcmp 0 = { drop 0 exit } if + dup s" cmps64" stringcmp 0 = { drop 0 exit } if + dup s" repz-cmps8" stringcmp 0 = { drop 0 exit } if + dup s" repz-cmps16" stringcmp 0 = { drop 0 exit } if + dup s" repz-cmps32" stringcmp 0 = { drop 0 exit } if + dup s" repz-cmps64" stringcmp 0 = { drop 0 exit } if + dup s" repnz-cmps8" stringcmp 0 = { drop 0 exit } if + dup s" repnz-cmps16" stringcmp 0 = { drop 0 exit } if + dup s" repnz-cmps32" stringcmp 0 = { drop 0 exit } if + dup s" repnz-cmps64" stringcmp 0 = { drop 0 exit } if + dup s" scas8" stringcmp 0 = { drop 0 exit } if + dup s" scas16" stringcmp 0 = { drop 0 exit } if + dup s" scas32" stringcmp 0 = { drop 0 exit } if + dup s" scas64" stringcmp 0 = { drop 0 exit } if + dup s" repz-scas8" stringcmp 0 = { drop 0 exit } if + dup s" repz-scas16" stringcmp 0 = { drop 0 exit } if + dup s" repz-scas32" stringcmp 0 = { drop 0 exit } if + dup s" repz-scas64" stringcmp 0 = { drop 0 exit } if + dup s" repnz-scas8" stringcmp 0 = { drop 0 exit } if + dup s" repnz-scas16" stringcmp 0 = { drop 0 exit } if + dup s" repnz-scas32" stringcmp 0 = { drop 0 exit } if + dup s" repnz-scas64" stringcmp 0 = { drop 0 exit } if + dup s" add-reg64-reg64" stringcmp 0 = { drop -2 exit } if + dup s" add-indirect-reg64-reg64" stringcmp 0 = { drop -2 exit } if + dup s" add-reg64-indirect-reg64" stringcmp 0 = { drop -2 exit } if + dup s" add-reg64-imm8" stringcmp 0 = { drop -2 exit } if + dup s" sub-reg64-reg64" stringcmp 0 = { drop -2 exit } if + dup s" sub-indirect-reg64-reg64" stringcmp 0 = { drop -2 exit } if + dup s" sub-reg64-imm8" stringcmp 0 = { drop -2 exit } if + dup s" sbb-reg64-imm8" stringcmp 0 = { drop -2 exit } if + dup s" mul-reg64" stringcmp 0 = { drop -1 exit } if + dup s" divmod-reg64" stringcmp 0 = { drop -1 exit } if + dup s" idivmod-reg64" stringcmp 0 = { drop -1 exit } if + dup s" inc-reg64" stringcmp 0 = { drop -1 exit } if + dup s" dec-reg64" stringcmp 0 = { drop -1 exit } if + dup s" and-reg64-reg64" stringcmp 0 = { drop -2 exit } if + dup s" and-reg64-imm8" stringcmp 0 = { drop -2 exit } if + dup s" or-reg64-reg64" stringcmp 0 = { drop -2 exit } if + dup s" or-reg64-imm8" stringcmp 0 = { drop -2 exit } if + dup s" xor-reg64-reg64" stringcmp 0 = { drop -2 exit } if + dup s" not-reg64" stringcmp 0 = { drop -1 exit } if + dup s" cmp-reg64-reg64" stringcmp 0 = { drop -2 exit } if + dup s" test-reg64-reg64" stringcmp 0 = { drop -2 exit } if + dup s" set-reg8-cc" stringcmp 0 = { drop -2 exit } if + dup s" jmp-cc-rel-imm8" stringcmp 0 = { drop -2 exit } if + dup s" jmp-cc-rel-imm32" stringcmp 0 = { drop -2 exit } if + dup s" jmp-abs-indirect-reg64" stringcmp 0 = { drop -1 exit } if + dup s" jmp-rel-imm8" stringcmp 0 = { drop -1 exit } if + dup s" jmp-rel-imm32" stringcmp 0 = { drop -1 exit } if + + ~ From language/static/execution-support.e. + dup s" pack-next" stringcmp 0 = { drop 0 exit } if + dup s" pack-beforenext" stringcmp 0 = { drop -1 exit } if + dup s" pack-pushcontrol" stringcmp 0 = { drop -1 exit } if + dup s" pack-popcontrol" stringcmp 0 = { drop -1 exit } if + + ~ From language/dynamic/dynamic.e. + dup s" stack" stringcmp 0 = { drop 0 exit } if + dup s" stackhex" stringcmp 0 = { drop 0 exit } if + dup s" is-in-log" stringcmp 0 = { drop 0 exit } if + dup s" oldest-entry" stringcmp 0 = { drop 1 exit } if + dup s" next-newer-entry" stringcmp 0 = { drop 0 exit } if + dup s" guess-entry-end" stringcmp 0 = { drop 0 exit } if + dup s" containing-entry" stringcmp 0 = { drop 0 exit } if + dup s" is-assembly-word" stringcmp 0 = { drop 0 exit } if + dup s" is-docol-itself" stringcmp 0 = { drop 0 exit } if + dup s" is-docol-codeword" stringcmp 0 = { drop 0 exit } if + dup s" is-docol-interpreted-word" stringcmp 0 = { drop 0 exit } if + dup s" is-codeword-pointer" stringcmp 0 = { drop 0 exit } if + dup s" indent" stringcmp 0 = { drop -1 exit } if + dup s" word-heading" stringcmp 0 = { drop -1 exit } if + dup s" list-dictionary" stringcmp 0 = { drop 0 exit } if + dup s" hexdump-row" stringcmp 0 = { drop -3 exit } if + dup s" hexdump-between" stringcmp 0 = { drop -2 exit } if + dup s" hexdump-from" stringcmp 0 = { drop -2 exit } if + dup s" hexdump" stringcmp 0 = { drop -1 exit } if + dup s" describe-hex" stringcmp 0 = { drop -1 exit } if + dup s" describe-docol" stringcmp 0 = { drop -1 exit } if + dup s" describe" stringcmp 0 = { drop -1 exit } if + dup s" describe-all" stringcmp 0 = { drop 0 exit } if + dup s" describe-compilation" stringcmp 0 = { drop 0 exit } if + dup s" symbolize-pointer" stringcmp 0 = { drop -1 exit } if + dup s" list-callers" stringcmp 0 = { drop 0 exit } if + dup s" forget" stringcmp 0 = { drop -1 exit } if + dup s" ," stringcmp 0 = { drop -1 exit } if + dup s" make-immediate" stringcmp 0 = { drop 0 exit } if + dup s" make-hidden" stringcmp 0 = { drop 0 exit } if + dup s" make-visible" stringcmp 0 = { drop 0 exit } if + dup s" recurse" stringcmp 0 = { drop 0 exit } if + dup s" recurse" stringcmp 0 = { drop 0 exit } if + dup s" find" stringcmp 0 = { drop 0 exit } if + dup s" allocate" stringcmp 0 = { drop 0 exit } if + dup s" create" stringcmp 0 = { drop -1 exit } if + dup s" self-codeword" stringcmp 0 = { drop 0 exit } if + dup s" variable" stringcmp 0 = { drop -2 exit } if + dup s" keyword" stringcmp 0 = { drop -1 exit } if + dup s" literal" stringcmp 0 = { drop -1 exit } if + + ~ From language/dynamic/input.e. + dup s" buffer-physical-start" stringcmp 0 = { drop 0 exit } if + dup s" buffer-physical-length" stringcmp 0 = { drop 0 exit } if + dup s" buffer-logical-start" stringcmp 0 = { drop 0 exit } if + dup s" buffer-logical-length" stringcmp 0 = { drop 0 exit } if + dup s" input-buffer-refill" stringcmp 0 = { drop 0 exit } if + dup s" input-buffer-next-source" stringcmp 0 = { drop 0 exit } if + dup s" clear-buffer" stringcmp 0 = { drop -1 exit } if + dup s" zero-input-buffer-metadata" stringcmp 0 = { drop -1 exit } if + dup s" allocate-input-buffer-metadata" stringcmp 0 = { drop 1 exit } if + dup s" allocate-input-buffer" stringcmp 0 = { drop 0 exit } if + dup s" attach-string-to-input-buffer" stringcmp 0 = { drop -2 exit } if + dup s" consume-from" stringcmp 0 = { drop -1 exit } if + dup s" peek-from" stringcmp 0 = { drop 0 exit } if + dup s" key-from" stringcmp 0 = { drop 0 exit } if + dup s" normalize-buffer" stringcmp 0 = { drop -1 exit } if + dup s" compute-next-buffer-free-block" stringcmp 0 = { drop 1 exit } if + dup s" refill-input-buffer-from-stdin" stringcmp 0 = { drop -1 exit } if + dup s" main-input-buffer" stringcmp 0 = { drop 1 exit } if + dup s" consume" stringcmp 0 = { drop 0 exit } if + dup s" peek" stringcmp 0 = { drop 1 exit } if + dup s" key" stringcmp 0 = { drop 1 exit } if + + ~ From language/dynamic/interpret.e. + dup s" unroll-past-string" stringcmp 0 = { drop 0 exit } if + dup s" swap-past-string" stringcmp 0 = { drop 0 exit } if + ~ The following are deliberate omissions: dropstring, + ~ dropstring-with-result, accumulate-string. + dup s" is-space" stringcmp 0 = { drop 0 exit } if + dup s" is-alphanumeric" stringcmp 0 = { drop 0 exit } if + ~ The following is a deliberate omission: word. + dup s" generalized-digit-value" stringcmp 0 = { drop 0 exit } if + ~ The following are deliberate omissions: decode-generalized-digit, + ~ read-base-unsigned, read-integer-unsigned, read-integer, read-decimal. + dup s" interpreter-flags" stringcmp 0 = { drop 1 exit } if + dup s" [" stringcmp 0 = { drop 0 exit } if + dup s" ]" stringcmp 0 = { drop 0 exit } if + dup s" :" stringcmp 0 = { drop 0 exit } if + dup s" ;" stringcmp 0 = { drop 0 exit } if + dup s" ;asm" stringcmp 0 = { drop 0 exit } if + dup s" '" stringcmp 0 = { drop 1 exit } if + dup s" relink-main-input-buffer-to-stdin" stringcmp 0 = { drop 0 exit } if + ~ The following is a deliberate omission: interpret. + dup s" quit" stringcmp 0 = { drop 0 exit } if + + ~ From language/dynamic/linux.e. + dup s" allocate-timespec" stringcmp 0 = { drop 1 exit } if + dup s" timespec-seconds" stringcmp 0 = { drop 0 exit } if + dup s" timespec-nanoseconds" stringcmp 0 = { drop 0 exit } if + dup s" nanosleep" stringcmp 0 = { drop 0 exit } if + dup s" sys-sigaction" stringcmp 0 = { drop -2 exit } if + dup s" sys-sigaltstack" stringcmp 0 = { drop -1 exit } if + dup s" allocate-sigaction" stringcmp 0 = { drop 1 exit } if + dup s" sigaction-action" stringcmp 0 = { drop 0 exit } if + dup s" sigaction-flags" stringcmp 0 = { drop 0 exit } if + dup s" sigaction-restorer" stringcmp 0 = { drop 0 exit } if + dup s" sigaction-mask" stringcmp 0 = { drop 0 exit } if + dup s" allocate-sigaltstack" stringcmp 0 = { drop 1 exit } if + dup s" sigaltstack-pointer" stringcmp 0 = { drop 0 exit } if + dup s" sigaltstack-flags" stringcmp 0 = { drop 0 exit } if + dup s" sigaltstack-size" stringcmp 0 = { drop 0 exit } if + dup s" prepare-signal-stack" stringcmp 0 = { drop 0 exit } if + ~ The following is a deliberate omission: signal-return-trampoline. + dup s" wrap-signal-handler" stringcmp 0 = { drop 0 exit } if + dup s" bind-signal" stringcmp 0 = { drop -2 exit } if + ~ The following is a deliberate omission: handle-crash. + dup s" install-crash-handler" stringcmp 0 = { drop 0 exit } if + + ~ Created by warm-start in compilation/execution.e. + dup s" log" stringcmp 0 = { drop 1 exit } if + dup s" s0" stringcmp 0 = { drop 1 exit } if + dup s" r0" stringcmp 0 = { drop 1 exit } if + dup s" latest" stringcmp 0 = { drop 1 exit } if + dup s" here" stringcmp 0 = { drop 1 exit } if + + ~ Word not provided statically, but used during the log-load routine anyway. + dup s" L@'" stringcmp 0 = { drop 1 exit } if + dup s" L!'" stringcmp 0 = { drop -1 exit } if + dup s" labels" stringcmp 0 = { drop 1 exit } if + dup s" allocate-transformation-state" stringcmp 0 = { drop 1 exit } if + + ~ If we get here, that's a problem. Emit an error message to make sure + ~ it's easy to diagnose. We also return a comically large negative value, + ~ to make sure things fail as quickly as possible afterwards. + ." No known stack delta: " emitstring newline + -256 ; + + +~ (delta --) +: transform-apply-stack-delta + transformation-state transformation-state-user-stack-depth @ + + transformation-state transformation-state-user-stack-depth ! ; + + +~ Label transform implementation +~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~ +~ The following code is all part of implementing the label transform. For +~ conceptual overview, see the top of this file. + + +~ This is the alternate version of "create" for use with the label +~ transform. Its code is the same as the regular "create" except as noted +~ below. It is likely to be extremely useful to read and understand "create" +~ in language/dynamic/dynamic.e before attempting to understand +~ label-create-alternate. +: label-create-alternate + ~ While we could put magic comments here, and be guaranteed to catch every + ~ word-defining word under the label transform, there's a subtle thing where + ~ we need to make sure the "indent" command happens in the same places it + ~ does when the same code runs under the log-load transform, and only those + ~ places. So, we do it in the callers of "create" instead. + dup stringlen 1 + + ~ : provide-data + dup 3unroll + here @ 10 + swap memmove + here @ + + ~ This value of "latest" is going into the generated output, so we need + ~ to map it to the target address space. It's stored in the host address + ~ space to make immediate words work as expected, so the appropriate + ~ conversion is host-address-space-to-target. + ~ : 8 #-- (previous entry pointer) + latest @ host-address-space-to-target pack64 + ~ : 1 #-- (entry flags) + 0 pack8 + ~ : bidirectional-null-terminated entry name + 0 pack8 + ~ : 1 data-adjust-output-point + ~ : fresh-line + + + 8 packalign + here @ latest ! + + ~ Now we're immediately after the word header, which is where the codeword + ~ will be. This is the value the label should taken on, so we set it. + dup host-address-space-to-offset + here @ 10 + + swap-transform-variables + intern-label set-label + swap-transform-variables + + here ! ; + + +~ This is the alternate version of ":" for use with the label transform. Its +~ code is the same as the regular "create" except as noted below. It is likely +~ to be extremely useful to read and understand ":" in +~ language/dynamic/dynamic.e before attempting to understand +~ label-colon-alternate. +: label-colon-alternate + ~ This calls label-create-alternate instead of "create". + word value@ + ~ : blank-line + ~ : blank-line + ~ : provide-string-copy + ~ : # is defined here via the label transform; this is its entry header. + ~ : indent + label-create-alternate dropstring + + ~ This looks up "docol" by label. + ~ : 8 #-- (docol codeword) + swap-transform-variables + L@' docol-codeword-value + L@' origin + swap-transform-variables + + , + ~ : blank-line + + latest @ dup entry-flags@ 0x80 | swap entry-flags! + ] ; + + +~ So, the way docol's user-facing entry is set up in language/static/core.e, +~ it really wants to reference "here" to create its codeword, but "here" is in +~ the host address space. We solve this by having it call the word +~ "self-codeword" and then patching around the address space issue with +~ alternates. +~ +~ If we didn't have this fix, everything would appear to work fine but the +~ statically generated user-facing docol from the label transform would have +~ a wrong codeword, varying based on the host's memory usage. It's never +~ actually called by evoke itself (it would crash horribly), though it may +~ be from programs that use the facilities differently. It's a binary +~ reproducibility issue - our only known one, in fact - so it's worth fixing. +: label-self-codeword-alternate + here @ dup 8 + host-address-space-to-target pack64 here ! + ; make-immediate + + +~ This is the alternate version of ";" for use with the label transform. Its +~ code is the same as the regular "create" except as noted below. It is likely +~ to be extremely useful to read and understand ";" in +~ language/dynamic/dynamic.e before attempting to understand +~ label-semicolon-alternate. +: label-semicolon-alternate + ~ This looks up "exit" by label. + swap-transform-variables + L@' exit + swap-transform-variables + offset-to-target-address-space , ~ : -8 exit #-- (codeword pointer) + ~ : deindent + + latest @ dup entry-flags@ 0x80 invert & swap entry-flags! + + ~ Since [ is an immediate word, we have to go to extra trouble to compile + ~ it as part of ;. + [ ' [ entry-to-execution-token , ] + ; make-immediate + + +~ This is the alternate version of ";asm" for use with the label transform. +~ Its code is the same as the regular "create" except as noted below. It is +~ likely to be extremely useful to read and understand ";asm" in +~ language/dynamic/dynamic.e before attempting to understand +~ label-semicolon-assembly-alternate. +: label-semicolon-assembly-alternate + here @ pack-next 8 packalign + ~ : provide-data + here ! + ~ : deindent + + latest @ dup dup entry-flags@ 0x80 invert & swap entry-flags! + entry-to-execution-token + ~ The codeword needs to be transformed to the target address space. + ~ : provide-data + ~ : data-set-output-point + ~ : delete-last-comment-at-output-point + ~ : 8 #-- (self codeword) + dup 8 + host-address-space-to-target + swap ! + ~ : data-set-output-point + + ~ Since [ is an immediate word, we have to go to extra trouble to compile + ~ it as part of ;asm. + [ ' [ entry-to-execution-token , ] + ; make-immediate + + +~ TODO there should really be an actual word that this alternate is replacing +~ +~ If it existed, that word would be (string pointer --). +: label-keyword-alternate + ~ : blank-line + ~ : blank-line + ~ : provide-string-copy + ~ : # is defined here via the label transform; this is its entry header. + ~ : indent + label-create-alternate + + here @ dup + ~ (self execution token, output point) + dup 8 + host-address-space-to-target pack64 + swap host-address-space-to-target :rax mov-reg64-imm64 + ~ (output point) + :rax push-reg64 + pack-next + 8 packalign + here ! + ~ : deindent + ; make-immediate + + +~ It's convenient to be able to work with strings in label-transformed +~ code. If we don't provide an alternate for it, we get codeword pointers in +~ the host address space, which therefore don't work. +: label-string-alternate + ~ Something subtle here: s" is state-dependent. That is, it does different + ~ things depending on the interpreter flags. We would really rather know + ~ which version we're getting, and also it would be best if it didn't + ~ scribble on the output buffer. Fortunately we can achieve both of these, + ~ by coercing things into a known state while calling it. + ~ + ~ We could choose either version of s", but the interpreted one is more + ~ convenient because it doesn't mess with a spurious litstring invocation, + ~ just scribbled into scratch space after "here". Of course, that raises the + ~ additional concern that we have the wrong "here", but we can just swap + ~ that around, too. + ~ + ~ This is all worth it to avoid reimplementing s". If we had two + ~ implementations, they'd have to be kept in sync, and it's an important + ~ user-facing word with semantics that are likely to improve over time. + interpreter-flags @ + ' s" entry-to-execution-token + swap-transform-variables + ~ Since [ is an immediate word, we have to go to extra trouble to compile + ~ it as part of the alternate. + [ ' [ entry-to-execution-token , ] + execute + swap-transform-variables + swap interpreter-flags ! + + ~ Whew. What a mouthful. + ~ + ~ Having done this, we have the string pointer on the stack at transform + ~ time, which with the label transform is the same as immediate time. If + ~ we're in immediate mode, therefore, we've left the string on the stack and + ~ that's all we wanted. If we're in compile mode, it's our job to output a + ~ litstring invocation. + + interpreter-flags @ 0x01 & { + swap-transform-variables + L@' litstring + swap-transform-variables + + offset-to-target-address-space , ~ : -8 litstring #-- (codeword pointer) + here @ swap packstring 8 packalign here ! + } if + ; make-immediate + + +: label-dot-string-alternate + ' label-string-alternate entry-to-execution-token execute + + interpreter-flags @ 0x01 & { + swap-transform-variables + L@' emitstring + swap-transform-variables + + offset-to-target-address-space , ~ : -8 emitstring #-- (codeword pointer) + } { emitstring } if-else + ; make-immediate + + +~ Because docol requires it, we provide a special mini-version of the label +~ system. We only do L@' and L!', because that's all we need. These are real +~ labels; there can be arbitrarily many of them, and they can have forward +~ references. +~ +~ The value that's accepted is in the host address space; the label is set +~ to an offset; and the value that's returned is in the target address space. +~ +~ (-- value) +: label-L@'-alternate + word value@ + + swap-transform-variables + intern-label + use-label + swap-transform-variables + + dropstring-with-result + + offset-to-target-address-space + ; make-immediate + + +~ (value --) +: label-L!'-alternate + host-address-space-to-offset + + word value@ + + swap-transform-variables + intern-label + swap-transform-variables + + dropstring-with-result + + swap-transform-variables + set-label + swap-transform-variables + ; make-immediate + + +~ We implement alternates of all the high-level flow-control words for the +~ label transform: if, unless, if-else, forever, and while. We leave { and } +~ as their usual implementations. The values { and } leave on the stack will +~ be in the host address space, which is convenient anyway. Those values don't +~ make their way into the output, since everything is relative, and the +~ relative offsets are correct regardless. What we do need to change, though, +~ is the addresses of branch and 0branch. We need to resolve these via label. +~ +~ (start pointer, length --) +: label-if-alternate + 2dup swap dup 5 8 * + 3roll + ~ (start pointer, length, adjusted start pointer, start pointer, length) + memmove + ~ (start pointer, length) + swap here @ swap here ! swap + ~ (old here, length) + swap-transform-variables + L@' 0branch + L@' != + L@' lit + swap-transform-variables + offset-to-target-address-space , ~ : -8 lit #-- (codeword pointer) + 0 ~ : provide-hex + , ~ : -8 # #-- (integer literal) + offset-to-target-address-space , ~ : -8 != #-- (codeword pointer) + ~ The branch length needs to be one word longer than the block length, + ~ because the length field itself is part of the scope of the branch. + offset-to-target-address-space , ~ : -8 0branch #-- (codeword pointer) + dup 8 + ~ : provide-hex + ~ : 8 # #-- (branch offset) + , + ~ (old here, length) + drop 5 8 * + here ! + ; make-immediate + +~ (start pointer, length) +: label-unless-alternate + 2dup swap dup 5 8 * + 3roll + ~ (start pointer, length, adjusted start pointer, start pointer, length) + memmove + ~ (start pointer, length) + swap here @ swap here ! swap + ~ (old here, length) + swap-transform-variables + L@' 0branch + L@' = + L@' lit + swap-transform-variables + offset-to-target-address-space , ~ : -8 lit #-- (codeword pointer) + 0 ~ : provide-hex + , ~ : -8 # #-- (integer literal) + offset-to-target-address-space , ~ : -8 = #-- (codeword pointer) + ~ The branch length needs to be one word longer than the block length, + ~ because the length field itself is part of the scope of the branch. + offset-to-target-address-space , ~ : -8 0branch #-- (codeword pointer) + dup 8 + ~ : provide-hex + ~ : 8 # #-- (branch offset) + , + ~ (old here, length) + drop 5 8 * + here ! + ; make-immediate + +~ (true start, true length, false start, false length) +: label-if-else-alternate + ~ First we slide the false-block forward, then the true-block. We slide + ~ them both directly into their final positions, leaving space at the start + ~ for a test and branch, and space in between for an unconditional branch. + ~ Those spaces will take five words, and two words, respectively. So the + ~ false-block gets moved by seven words, and the true-block gets moved by + ~ five words. + 2dup swap dup 7 8 * + 3roll memmove + 3 pick dup 5 8 * + 4 pick memmove + ~ (true start, true length, false start, false length) + + ~ Now we write out the initial test-and-branch. + 4 roll dup 5 unroll here @ 6 unroll here ! + ~ (old here, true start, true length, false start, false length) + swap-transform-variables + L@' 0branch + L@' != + L@' lit + swap-transform-variables + offset-to-target-address-space , ~ : -8 lit #-- (codeword pointer) + 0 ~ : provide-hex + , ~ : -8 # #-- (integer literal) + offset-to-target-address-space , ~ : -8 != #-- (codeword pointer) + ~ Branch past the length field, the true-block, and the unconditional + ~ branch in the middle. + offset-to-target-address-space , ~ : -8 0branch #-- (codeword pointer) + 3roll dup 4 unroll 3 8 * + ~ : provide-hex + ~ : 8 # #-- (branch offset) + , + + ~ Next, write out the unconditional branch in the middle. + swap dup 3unroll 5 8 * + here ! + swap-transform-variables + L@' branch + swap-transform-variables + offset-to-target-address-space , ~ : -8 branch #-- (codeword pointer) + ~ Branch past the length field and the false-block. + dup 8 + ~ : provide-hex + ~ : 8 # #-- (branch offset) + , + + ~ Set "here" to point to the true end. + drop drop drop drop 7 8 * + here ! + ; make-immediate + +~ (start, length --) +: label-forever-alternate + swap-transform-variables + L@' branch + swap-transform-variables + offset-to-target-address-space , ~ : -8 branch #-- (codeword pointer) + 8 + -1 * ~ : provide-hex + ~ : 8 # #-- (branch offset) + , drop + ; make-immediate + +~ (test start, test length, body start, body length --) +: label-while-alternate + ~ The conditional branch needs five words. + 2dup swap dup 5 8 * + 3roll memmove + here @ 5 unroll swap dup 3unroll here ! + ~ (old here, test start, test length, body start, body length) + swap-transform-variables + L@' 0branch + L@' != + L@' lit + swap-transform-variables + offset-to-target-address-space , ~ : -8 lit #-- (codeword pointer) + 0 ~ : provide-hex + , ~ : -8 # #-- (integer literal) + offset-to-target-address-space , ~ : -8 != #-- (codeword pointer) + ~ Branch past the length field, the body, and the unconditional branch. + offset-to-target-address-space , ~ : -8 0branch #-- (codeword pointer) + dup 3 8 * + ~ : provide-hex + ~ : 8 # #-- (branch offset) + , + ~ Set "here" to the new end. + 5 8 * 6 roll + here ! + ~ (test start, test length, body start, body length) + ~ Unconditionally branch backwards past the branch word, the body, the + ~ conditional branch, and the test. + swap-transform-variables + L@' branch + swap-transform-variables + offset-to-target-address-space , ~ : -8 branch #-- (codeword pointer) + 6 8 * + swap drop + swap drop -1 * ~ : provide-hex + ~ : 8 # #-- (branch offset) + , + ; make-immediate + + +~ This implements the label transform for a single word. It is directly +~ analogous to "interpret", and reading language/dynamic/dynamic.e may help in +~ understanding it, though it's meant to still make sense on its own. +~ +~ It expects to be called from "label-transform", below, which loops. +~ +~ (-- done) +: label-transform-one + word + + ~ If no word was returned, end the transformation. + dup 0 = { drop 1 exit } if + + ~ The string is on the top of the stack, so to get a pointer to it we get + ~ the stack address. + ~ (string) + value@ + + ~ Check whether it's one of the words we have alternates for, and look up + ~ the alternate if so. + dup 0 swap + ~ (name as stack string, name pointer, placeholder, name pointer) + dup s" create" stringcmp 0 = { swap drop ' label-create-alternate swap } if + dup s" :" stringcmp 0 = { swap drop ' label-colon-alternate swap } if + dup s" self-codeword" stringcmp 0 = { + swap drop ' label-self-codeword-alternate swap } if + dup s" ;" stringcmp 0 = { swap drop ' label-semicolon-alternate swap } if + dup s" ;asm" stringcmp 0 = { + swap drop ' label-semicolon-assembly-alternate swap } if + ~ It is nontrivial to construct a string with a double-quote in it. + dup ' s" entry-to-name stringcmp 0 = { + swap drop ' label-string-alternate swap } if + dup ' ." entry-to-name stringcmp 0 = { + swap drop ' label-dot-string-alternate swap } if + dup s" L@'" stringcmp 0 = { swap drop ' label-L@'-alternate swap } if + dup s" L!'" stringcmp 0 = { swap drop ' label-L!'-alternate swap } if + dup s" keyword" stringcmp 0 = { + swap drop ' label-keyword-alternate swap } if + dup s" if" stringcmp 0 = { swap drop ' label-if-alternate swap } if + dup s" unless" stringcmp 0 = { swap drop ' label-unless-alternate swap } if + dup s" if-else" stringcmp 0 = { + swap drop ' label-if-else-alternate swap } if + dup s" forever" stringcmp 0 = { + swap drop ' label-forever-alternate swap } if + dup s" while" stringcmp 0 = { swap drop ' label-while-alternate swap } if + drop swap + ~ (name as stack string, 0 or alternate entry pointer, name pointer) + + ~ If an alternate was found, the alternate will be used in immediate mode. + ~ If not, we look up the word in the regular, non-transformed dictionary + ~ and use that for immediate mode. + over { dup + transformation-state transformation-state-saved-latest @ swap find-in + 3roll drop swap } unless + ~ (name as stack string, immediate entry pointer, name pointer) + + ~ In regular "interpret", we would check whether we found the word before + ~ checking the mode. However, we have three different places words could + ~ come from, so that's not a simple notion. So, we check the mode first. + interpreter-flags @ 0x01 & { + ~ If we're in compile mode, there's still a chance it's an immediate + ~ word. First check whether we have an immediate entry, then if so, check + ~ that entry's flags. Notice that this means the generated code can't + ~ override an immediate word with a non-immediate word of the same name. + over dup { entry-flags@ 0x01 & not } { not } if-else + + { + ~ Either there was no immediate entry, or the immediate entry wasn't + ~ flagged as an immediate word. So we check whether this could be a + ~ compilation. + ~ + ~ To do this, we need to look the word up in the output buffer. We + ~ can't easily traverse the next-entry pointers in the output buffer's + ~ dictionary, so we check the label. Since we don't know the word's name + ~ statically, this is a rare scenario where we can't use the abbreviated + ~ label syntax, but that's easy enough. + ~ + ~ Even though we've ruled out the possibility that the word is only + ~ ever used immediately, it is still possible that there's some reason + ~ the word doesn't exist. In particular, it could be an integer literal. + ~ If we were to call use-label first, that would count as a requirement + ~ that the label must eventually be set. We don't want to require that + ~ quite yet, so we call find-label. + ~ + ~ This check is the means by which forward references are disallowed: + ~ On the very first pass, a forward-referenced label won't exist yet, so + ~ transform will give a "no such word" error, which in an ideal world + ~ would prevent there from being a subsequent pass, but at the very + ~ least it will ensure the output isn't a valid ELF. + dup + swap-transform-variables + find-label + swap-transform-variables + { + ~ It exists, so we declare our use of it (that's also the only way to + ~ get a value for it). + ~ : provide-string-copy + swap-transform-variables + intern-label use-label + swap-transform-variables + + ~ Labels point to codewords (because that's what + ~ label-create-alternate does), which is already what we want to + ~ output. + ~ + ~ An important caveat: Though it would require something weird to be + ~ happening, such as a forced forward reference, the label may be + ~ zero! We need to allow for that possibility by not examining the + ~ contents of a nonexistent entry. + ~ + ~ Fortunately we don't have to look at it, just append it to the log + ~ and clean up. + ~ : 8 # #-- (codeword pointer) + offset-to-target-address-space , drop dropstring 0 exit + } if + + ~ If we got here, we're in compile mode, no label was found, and even + ~ if there was a candidate for an immediate word it wasn't flagged as + ~ immediate. There are two possibilities: It's genuinely missing, or it's + ~ an integer literal. We decline to run the candidate immediate entry, + ~ even if it exists, because that's not the correct semantics. + ~ + ~ If the word is genuinely missing, we want to make sure we make it + ~ all the way to the not-found error-handling code at the end, because + ~ that will be way easier to debug than doing the wrong thing will. Way, + ~ way easier. Far less staring at numbers. + ~ + ~ Anyway, we no longer need the immediate entry pointer, so we drop + ~ it. + drop drop + } { + ~ If we get here, we're in compile mode, but there was a candidate + ~ entry for an immediate word, and it was indeed flagged as immediate. + ~ So, we run it and exit. + drop dropstring-with-result entry-to-execution-token execute + 0 exit + } if-else + + ~ This is the end of the compile-mode branch. As you can see by tracing + ~ through all the above cases, if we got here, the two possibilities are + ~ that the word is genuinely missing, or it's an integer literal. + ~ + ~ Please notice that these are the same two possibilities remaining at + ~ the end of the immediate-mode branch, below. + } { + ~ If we got here, we're in interpret mode. There are three + ~ possibilities: there's an immediate word which we should run; it's an + ~ integer literal; or the word is genuinely missing. + ~ + ~ If the immediate entry pointer is non-zero, run it and exit. + over { + drop dropstring-with-result entry-to-execution-token execute + 0 exit + } if + + ~ There was no immediate word, so either it's an integer literal or + ~ the word is genuinely missing. Please notice that these are the same two + ~ possibilities remaining at the end of the compile-mode branch, above. + ~ + ~ We no longer need the immediate-mode pointer, so drop it. + drop drop + } if-else + ~ (name as stack string) + + ~ If we got here, one of two things is true: the word is an integer + ~ literal, or it's genuinely missing. We know this because both the mode + ~ cases above end with these as the only two remaining possibilities. So + ~ + ~ Check whether it's an integer literal. As before, we get the stack + ~ address and use it as a string pointer. + value@ read-integer 0 = { + ~ It's a number. + interpreter-flags @ 0x01 & { + ~ We're in compile mode; append first "lit", then the number, to the + ~ log. The version of "lit" we use is found by label, so it'll be the + ~ one that exists when this code is ultimately run. + dropstring-with-result + + ~ We look up "lit" as a label. + swap-transform-variables L@' lit swap-transform-variables + offset-to-target-address-space + ~ : 8 lit #-- (codeword pointer) + , + ~ : provide-hex + ~ : 8 # #-- (integer literal) + , + 0 exit + } if + + ~ We're in interpret mode; push the number to the stack. Or at least, that's + ~ what the code we're interpreting will see. Really it's already on the + ~ stack, just clean everything else up and leave it there. + dropstring-with-result + 0 exit + } if + + ~ If it's neither in the dictionary nor a number, just print an error. + ~ + ~ It's really important, when maintaining this code, to make sure that all + ~ the possible ways the word can fail to exist, end up here. Doing anything + ~ else is going to result in many hours of trying to untangle the + ~ consequences of incorrect behavior, after-the-fact. + s" No such word: " emitstring value@ emitstring newline dropstring 0 ; + + +~ This implements the label transform for all words in a region given as an +~ input string. It is directly analogous to "quit", in interpet.e, but is far +~ more complex. +~ +~ (output buffer start, output point, input string pointer +~ -- output buffer start, output point) +: label-transform + ~ : blank-line + ~ : blank-line + ~ : ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ + ~ : ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ + ~ : ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ + ~ : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + ~ : blank-line + ~ : This is the start of a long block of code which has been processed by + ~ : the label transform, meaning it consists of ready-to-run Forth word + ~ : definitions that reference each other by address, suitable for direct + ~ : execution as soon as the cold-start routine has set up the basic + ~ : registers, flags, and memory areas that form Evocation's internal ABI. + ~ : + ~ : Each word definition consists of a dictionary entry header, a + ~ : codeword, and a word body. For words implemented as Forth code, the + ~ : codeword points to the "docol" routine, and the word body is an array of + ~ : pointers to the codewords of other words, occasionally interspersed with + ~ : literal values. For words implemented as machine code, the codeword + ~ : points to the start of the body, and the body is the machine code. + ~ : + ~ : All the word entry headers in this code link to each other, from last + ~ : backwards to first, forming an Evocation dictionary. + ~ : + ~ : The warm-start routine relies on words defined by the label transform, + ~ : as does the log-load routine. + ~ : blank-line + + main-input-buffer dup push-input-buffer + swap attach-string-to-input-buffer + + ~ Save the old values of "here" and "latest", and set the initial values + ~ of the internal ones. These values need to persist across iterations, + ~ since client code will make its own updates to them and then rely on those + ~ updates having taken effect. So we do the swap just once, here outside the + ~ loop, and set it back when the loop ends. + here @ transformation-state transformation-state-saved-here ! + latest @ transformation-state transformation-state-saved-latest ! + swap transformation-state transformation-state-output-buffer-start ! + here ! + 0 latest ! + ~ Now the stack has nothing of ours on it, so client code can do its thing. + + ~ It's important that the stack has nothing of ours on it that persists + ~ across iterations, so that client code can add and remove stuff there as + ~ it sees fit. + { label-transform-one + ~ (..., done) + + ~ When the loop is done, get the real values of "here" and "latest" + ~ back. The internal "here" is also the output point, and will become our + ~ return value. The internal "latest" is discarded. + { transformation-state transformation-state-output-buffer-start @ + here @ + transformation-state transformation-state-saved-here @ here ! + transformation-state transformation-state-saved-latest @ latest ! + ~ (output buffer start, output point) + + ~ Though we don't actually use transformation-state outside of this + ~ invocation, for tidiness we zero it out. + 0 transformation-state transformation-state-saved-here ! + 0 transformation-state transformation-state-saved-latest ! + 0 transformation-state transformation-state-output-buffer-start ! + + ~ Also put the input source back how it was. + main-input-buffer pop-input-buffer + + exit } if } forever ; + + +~ Log-load transform implementation +~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~ +~ The following code is all part of implementing the log-load transform. +~ For conceptual overview, see the top of this file. + + +~ This outputs code that performs a roll of a given size, for use by the +~ words that rely on the tracked user stack depth. +~ +~ (amount to roll by --) +: log-load-roll + ~ A roll of size 1 is an nop, and a roll of size 0 crashes. + dup 2 > { drop exit } if + + ~ A roll of size 2 is equivalent to a swap, and we compile it as one for + ~ clarity when reading the hexdump. + ~ + ~ Yes, clarity when reading the hexdump. + dup 2 = { + drop + + swap-transform-variables + L@' swap + swap-transform-variables + + offset-to-target-address-space , ~ : -8 swap #-- (codeword pointer) + + exit + } if + + swap-transform-variables + L@' roll + L@' lit + swap-transform-variables + + offset-to-target-address-space , ~ : -8 lit #-- (codeword pointer) + swap ~ : provide-hex + , ~ : -8 # #-- (integer literal) + offset-to-target-address-space , ~ : -8 roll #-- (codeword pointer) + ; + + +~ This outputs code that perfoms an unroll of a given size, for use by the +~ words that rely on the tracked user stack depth. + +~ (amount to unroll by --) +: log-load-unroll + ~ An unroll of size 1 is an nop, and an unroll of size 0 crashes. + dup 2 > { drop exit } if + + ~ Yes, clarity when reading the hexdump! What's so weird about that? :) + dup 2 = { + drop + + swap-transform-variables + L@' swap + swap-transform-variables + + offset-to-target-address-space , ~ : -8 swap #-- (codeword pointer) + + exit + } if + + swap-transform-variables + L@' unroll + L@' lit + swap-transform-variables + + offset-to-target-address-space , ~ : -8 lit #-- (codeword pointer) + swap ~ : provide-hex + , ~ : -8 # #-- (integer literal) + offset-to-target-address-space , ~ : -8 unroll #-- (codeword pointer) + ; + + +~ This checks the tracked user stack depth and outputs an appropriate roll +~ to put the log address on top, as preparation for calling something that +~ relies on it. +: log-load-roll-log-address + transformation-state transformation-state-user-stack-depth @ + ~ The amount we're rolling includes the log address, which is not a user + ~ value, so we add 1. + 1 + + log-load-roll ; + + +~ This checks the tracked user stack depth and outputs an appropriate unroll +~ to put the log address back at the bottom, as cleanup after calling +~ something that relies on it. +: log-load-unroll-log-address + transformation-state transformation-state-user-stack-depth @ + ~ The amount we're rolling includes the log address, which is not a user + ~ value, so we add 1. + 1 + + log-load-unroll ; + + +~ This checks the tracked user stack depth and outputs an appropriate roll +~ to put the saved label value on top, as preparation for calling something +~ that consumes it. +: log-load-roll-saved-label + transformation-state transformation-state-user-stack-depth @ + ~ The amount we're rolling includes both the log address and the label + ~ value, neither of which are user values, so we add 2. + 2 + + log-load-roll ; + + +~ This checks the tracked user stack depth and outputs an appropriate unroll +~ to put a saved label value on the bottom, as a way of safely storing it away +~ when it's newly generated. +: log-load-unroll-saved-label + transformation-state transformation-state-user-stack-depth @ + ~ The amount we're rolling includes both the log address and the label + ~ value, neither of which are user values, so we add 2. + 2 + + log-load-unroll ; + + +~ (name pointer --) +: log-load-compile-dynamic-word + log-load-roll-log-address + + swap-transform-variables + ~ Looking these up in reverse order saves us some stack juggling. Does it + ~ help readability, or hurt it? Who can say... + L@' log-load-comma + L@' log-load-find-execution-token + L@' litstring + swap-transform-variables + ~ (name pointer, log-load-comma, log-load-find-execution-token, litstring) + + ~ The overall stack delta of this sequence is 0. + offset-to-target-address-space , ~ : -8 litstring #-- (codeword pointer) + 3roll here @ swap packstring 8 packalign here ! + offset-to-target-address-space , + ~ : -8 log-load-find-execution-token #-- (codeword pointer) + offset-to-target-address-space , + ~ : -8 log-load-comma #-- (codeword pointer) + + log-load-unroll-log-address ; + + +~ When we want the log-load routine to run a word that wasn't statically +~ compiled-in to the target executable, we output code that looks up the word +~ by name on the log, then calls it. +~ +~ We update the user stack depth as we go, to account for our internal +~ needs. The last thing our generated code does is execute the word we were +~ asked to, and we have no way of knowing what its stack delta will be, so +~ that final delta is our caller's responsibility. The caller should only +~ consider the delta of the word it requested itself; we handle everything +~ pertaining to retrieving, and then consuming, its execution token. +~ +~ (name pointer --) +: log-load-execute-dynamic-word + log-load-roll-log-address + + swap-transform-variables + ~ This is reverse order again. + L@' swap + L@' log-load-find-execution-token + L@' litstring + swap-transform-variables + ~ (name pointer, swap, log-load-find-execution-token, litstring) + + offset-to-target-address-space , ~ : -8 litstring #-- (codeword pointer) + 3roll here @ swap packstring 8 packalign here ! + offset-to-target-address-space , + ~ : -8 log-load-find-execution-token #-- (codeword pointer) + offset-to-target-address-space , ~ : -8 swap #-- (codeword pointer) + + ~ Now the execution token is on the stack, immediately below the log + ~ address, so we apply a delta for it. + 1 transform-apply-stack-delta + + log-load-unroll-log-address + + swap-transform-variables + L@' execute + swap-transform-variables + + offset-to-target-address-space , ~ : -8 execute #-- (codeword pointer) + + ~ Invoking execute consumes the execution token, which is a delta of -1. + ~ Any additional delta is our caller's responsibility, per above. + -1 transform-apply-stack-delta ; + + +~ We generate code that looks up "docol" by name, runs it to get the +~ codeword pointer, then finally appends it to the entry. +~ +~ This one's unusual in that it first executes the word, then compiles the +~ result. To avoid rolling and unrolling repeatedly, we implement it as a +~ special case. For clarity's sake we put the code here, with the other +~ dynamic word stuff. +~ +: log-load-compile-docol + log-load-roll-log-address + + swap-transform-variables + ~ As usual, we do these in reverse. + L@' log-load-comma + L@' execute + L@' log-load-find-execution-token + L@' litstring + swap-transform-variables + + ~ The overall stack delta of this sequence is 0. + offset-to-target-address-space , ~ : -8 litstring #-- (codeword pointer) + here @ s" docol" packstring 8 packalign here ! + offset-to-target-address-space , + ~ : -8 log-load-find-execution-token #-- (codeword pointer) + offset-to-target-address-space , ~ : -8 execute #-- (codeword pointer) + offset-to-target-address-space , + ~ : -8 log-load-comma #-- (codeword pointer) + + log-load-unroll-log-address ; + + +~ There's one more compilation case, where we wish to compile an integer +~ literal. Again, for clarity, we do it here. +~ +~ (integer value --) +: log-load-compile-literal + log-load-roll-log-address + + swap-transform-variables + ~ As usual, we do these in reverse. + L@' log-load-comma + L@' lit + L@' log-load-comma + L@' log-load-find-execution-token + L@' litstring + swap-transform-variables + + ~ The overall stack delta of this sequence is 0. + offset-to-target-address-space , ~ : -8 litstring #-- (codeword pointer) + here @ s" lit" packstring 8 packalign here ! + offset-to-target-address-space , + ~ : -8 log-load-find-execution-token #-- (codeword pointer) + offset-to-target-address-space , + ~ : -8 log-load-comma #-- (codeword pointer) + offset-to-target-address-space , ~ : -8 lit #-- (codeword pointer) + swap ~ : provide-hex + , ~ : -8 # #-- (integer literal) + offset-to-target-address-space , + ~ : -8 log-load-comma #-- (codeword pointer) + + log-load-unroll-log-address ; + + +~ This is the alternate version of "create" for use with the log-load +~ transform. This one is quite unlike the regular "create"; rather than +~ creating an entry on the log directly, its job is to output words which, +~ when they're later executed, will do create's job. +~ +~ In practice that means outputting a codeword pointer to run a +~ statically-compiled word that does the work. The implementation of +~ log-load-create is in language/static/log-load.e. +~ +~ It's worth keeping in mind that this alternate only gets called for +~ manual invocations of "create". It isn't called from the colon alternate. +: log-load-create-alternate + ~ In immediate mode, we have special behavior because this is fundamental + ~ to bootstrapping the log. In compile mode, we compile a dynamic call, + ~ which is the same thing that would happen if we didn't have an alternate + ~ at all. + interpreter-flags @ 0x01 & { + ~ Notice that when we're in compile mode, we aren't actually defining a + ~ word at the time the log-load routine runs, so we don't have magic + ~ comments. + s" create" log-load-compile-dynamic-word + } { + ~ : blank-line + ~ : create + log-load-roll-log-address + + swap-transform-variables + L@' log-load-create + L@' swap + swap-transform-variables + + ~ The overall stack delta of this sequence is 0. + offset-to-target-address-space , ~ : -8 swap #-- (codeword pointer) + offset-to-target-address-space , + ~ : -8 log-load-create #-- (codeword pointer) + + ~ We've consumed a string pointer from the stack, so that's a delta of -1. + -1 transform-apply-stack-delta + + log-load-unroll-log-address + ~ : blank-line + } if-else ; + + +~ This is the alternate version of ":" for use with the log-load transform. +~ Its code is the same as the regular ":" except as noted below. It is likely +~ to be extremely useful to read and understand ":" in +~ language/dynamic/dynamic.e before attempting to understand +~ "log-load-colon-alternate". +: log-load-colon-alternate + word value@ + ~ : blank-line + ~ : blank-line + ~ : provide-string-copy + ~ : # is defined here via the log-load transform. + ~ : indent + + ~ Calling log-load-create-alternate would result in some redundant rolling + ~ and unrolling, so we do it together like this instead. + log-load-roll-log-address + + swap-transform-variables + L@' log-load-create + L@' litstring + swap-transform-variables + + ~ The overall stack delta of this sequence is 0. + offset-to-target-address-space , ~ : -8 litstring #-- (codeword pointer) + swap here @ swap packstring 8 packalign here ! + offset-to-target-address-space , + ~ : -8 log-load-create #-- (codeword pointer) + + log-load-unroll-log-address + + dropstring + + ~ We generate code that looks up "docol" by name, runs it to get the + ~ codeword pointer, then finally appends it to the entry. + log-load-compile-docol + + ~ This is where we would mark the entry hidden, but we don't do that. It + ~ won't shadow anything and it won't be called until the entire log-load + ~ routine has finished. + + ~ Switching between immediate and compile mode is one of the very few + ~ things that happens NOW, while the log-load transform is actually running. + ] + ; + + +~ See notes on label-self-codeword-alternate for what problem this is +~ solving. +~ +~ Since self-codeword's real definition is in language/dynamic/dynamic.e, +~ which absolutely cannot be loaded yet at the time we output docol at the +~ very beginning of language/static/core.e, we need to provide an alternate +~ for the log-load transform as well, even though only the label transform has +~ the reproducibility issue. +: log-load-self-codeword-alternate + ~ It's important to remember, this is not an immediate word! It needs to + ~ produce correct compiled output, when invoked in compile mode. Since it's + ~ also a regular word that exists, correct output is a reference to the + ~ non-alternate version. + interpreter-flags @ 0x01 & { + s" self-codeword" log-load-compile-dynamic-word + } { + log-load-roll-log-address + + swap-transform-variables + L@' log-load-self-codeword + swap-transform-variables + + offset-to-target-address-space , + ~ : -8 log-load-self-codeword #-- (codeword pointer) + + log-load-unroll-log-address + } if-else + ; make-immediate + + +~ This is the alternate version of ";" for use with the log-load transform. +~ Its code is the same as the regular ";" except as noted below. It is +~ likely to be extremely useful to read and understand ";" in +~ language/dynamic/dynamic.e before attempting to understand +~ "log-load-semicolon-alternate". +: log-load-semicolon-alternate + ~ We generate code that looks up "exit" by name and appends it to the + ~ entry. + log-load-roll-log-address + + swap-transform-variables + ~ As usual, we do these in reverse. + L@' log-load-comma + L@' log-load-find-execution-token + L@' litstring + swap-transform-variables + + ~ The overall stack delta of this sequence is 0. + offset-to-target-address-space , ~ : -8 litstring #-- (codeword pointer) + here @ s" exit" packstring 8 packalign here ! + offset-to-target-address-space , + ~ : -8 log-load-find-execution-token #-- (codeword pointer) + offset-to-target-address-space , + ~ : -8 log-load-comma #-- (codeword pointer) + + log-load-unroll-log-address + + ~ : deindent + + ~ This is where we would unhide the entry, but again, we don't do that. + + ~ Since [ is an immediate word, we have to go to extra trouble to compile + ~ it as part of ;. + [ ' [ entry-to-execution-token , ] + ; make-immediate + + +~ This is the alternate version of ";asm" for use with the log-load +~ transform. Its code is the same as the regular "create" except as noted +~ below. It is likely to be extremely useful to read and understand ";asm" in +~ language/dynamic/dynamic.e before attempting to understand "log-load;asm". +: log-load-semicolon-assembly-alternate + ~ We generate code that statically invokes log-load-semicolon-assembly, + ~ which does all the actual work. There's quite a few steps, so it makes + ~ sense to have a word that bundles them up. + log-load-roll-log-address + + swap-transform-variables + ~ As usual, we do these in reverse. + L@' log-load-semicolon-assembly + swap-transform-variables + + ~ The stack delta of this call is 0. + offset-to-target-address-space , + ~ : -8 log-load-semicolon-assembly #-- (codeword pointer) + + log-load-unroll-log-address + + ~ : deindent + + ~ This is where we would unhide the entry, but again, we don't do that. + + ~ Since [ is an immediate word, we have to go to extra trouble to compile + ~ it as part of ;. + [ ' [ entry-to-execution-token , ] + ; make-immediate + + +~ This just does the same thing [ always does, but having it as an alternate +~ means it happens at transformation time, which is sooner than "immediate" +~ time. The log load transform is weird like that, it has three different +~ times things can happen, rather than the usual two. +: log-load-left-square-brace-alternate + ~ Since [ is an immediate word, we have to go to extra trouble to compile + ~ it as part of the alternate. + [ ' [ entry-to-execution-token , ] + ; make-immediate + +~ We need this one, too. It's not even an immediate word normally! +: log-load-right-square-brace-alternate ] ; make-immediate + + +: log-load-comma-alternate + ~ In immediate mode, we have special behavior because of the "here" magic. + ~ In compile mode, we compile a dynamic call, which is the same thing that + ~ would happen if we didn't have an alternate at all. + interpreter-flags @ 0x01 & { + s" ," log-load-compile-dynamic-word + } { + log-load-roll-log-address + + swap-transform-variables + L@' log-load-comma + L@' swap + swap-transform-variables + + offset-to-target-address-space , ~ : -8 swap #-- (codeword pointer) + offset-to-target-address-space , + ~ : -8 log-load-comma #-- (codeword pointer) + + ~ We consumed the value, so we apply a delta. + -1 transform-apply-stack-delta + + log-load-unroll-log-address + } if-else + ; make-immediate + + +: log-load-variable-alternate + ~ In immediate mode, we have special behavior because of the "here" magic. + ~ In compile mode, we compile a dynamic call, which is the same thing that + ~ would happen if we didn't have an alternate at all. + interpreter-flags @ 0x01 & { + s" variable" log-load-compile-dynamic-word + } { + ~ Calling log-load-create-alternate would result in some redundant rolling + ~ and unrolling, so we do it together like this instead. + log-load-roll-log-address + + swap-transform-variables + L@' log-load-variable + L@' 3unroll + swap-transform-variables + + offset-to-target-address-space , ~ : -8 3unroll #-- (codeword pointer) + offset-to-target-address-space , + ~ : -8 log-load-variable #-- (codeword pointer) + + ~ We consumed the string and address, so we apply a delta. + -2 transform-apply-stack-delta + + log-load-unroll-log-address + } if-else + ; make-immediate + + +: log-load-keyword-alternate + ~ In immediate mode, we have special behavior because of the "here" magic. + ~ In compile mode, we compile a dynamic call, which is the same thing that + ~ would happen if we didn't have an alternate at all. + interpreter-flags @ 0x01 & { + s" keyword" log-load-compile-dynamic-word + } { + ~ Calling log-load-create-alternate would result in some redundant rolling + ~ and unrolling, so we do it together like this instead. + log-load-roll-log-address + + swap-transform-variables + L@' log-load-keyword + L@' swap + swap-transform-variables + + offset-to-target-address-space , ~ : -8 swap #-- (codeword pointer) + offset-to-target-address-space , + ~ : -8 log-load-keyword #-- (codeword pointer) + + ~ We consumed the string, so we apply a delta. + -1 transform-apply-stack-delta + + log-load-unroll-log-address + } if-else + ; make-immediate + + +~ Yeah comments have to go. +: log-load-tilde-alternate + ' ~ entry-to-execution-token execute + ; make-immediate + + +~ Strings are important and must happen now, now, now. +: log-load-string-alternate + ~ Wrapping s" is surprisingly difficult; see label-string-alternate + ~ for detailed notes on how that works. Once we've finished the + ~ transform-time part, our implementations will diverge, but this first bit + ~ is the same. + interpreter-flags @ + ' s" entry-to-execution-token + swap-transform-variables + ~ Since [ is an immediate word, we have to go to extra trouble to compile + ~ it as part of the alternate. + [ ' [ entry-to-execution-token , ] + execute + swap-transform-variables + swap interpreter-flags ! + + ~ Nope, doing it twice didn't make that any less of a mouthful. + ~ + ~ Having done this, we have the string pointer on the stack at transform + ~ time (not, mind you, at log-load time, as yet). Now it is our job to + ~ either compile it, or invoke it "immediately". Recall that the log-load + ~ transform's "immediate" time, which we also call log-load time, is + ~ analogous to the label transform's "compile" time. The log-load + ~ transform's "compile" time is an additional layer of indirection. + ~ + ~ No matter which mode we're in, the first thing we want to do is get the + ~ string onto the value stack at log-load time. So we compile code to do + ~ that, first. We do that using the static "litstring", which we find by + ~ label. + ~ + ~ If this is all too twisty, it might be helpful to go re-read the + ~ definition of log-load-compile-literal, which does a simpler version of + ~ this same task (because integers are easier than strings). + + swap-transform-variables + L@' litstring + swap-transform-variables + + offset-to-target-address-space , ~ : -8 litstring #-- (codeword pointer) + here @ swap packstring 8 packalign here ! + + ~ Now the string pointer is on the stack at log-load time, so we apply a + ~ delta for it. + 1 transform-apply-stack-delta + + ~ If we're in immediate mode, we're done! Yay! Putting the string on the + ~ stack at log-load time is what we wanted. If not... + interpreter-flags @ 0x01 & { + ~ We're in compile mode, so we need to look up the ultimate, dynamic, + ~ log-based, super-atomic space rocket copy of "litstring" and use that. + ~ We could call log-load-compile-dynamic-word here, but, as usual, it + ~ would produce some excessive rolling and unrolling, so we do its job + ~ ourselves instead. + log-load-roll-log-address + + swap-transform-variables + ~ As usual, we do these in reverse. + L@' log-load-comma-string + L@' swap + L@' log-load-comma + L@' log-load-find-execution-token + L@' litstring + swap-transform-variables + + ~ The overall stack delta of this sequence is 0. + offset-to-target-address-space , + ~ : -8 litstring #-- (codeword pointer) + here @ s" litstring" packstring 8 packalign here ! + offset-to-target-address-space , + ~ : -8 log-load-find-execution-token #-- (codeword pointer) + offset-to-target-address-space , + ~ : -8 log-load-comma #-- (codeword pointer) + offset-to-target-address-space , ~ : -8 swap #-- (codeword pointer) + offset-to-target-address-space , + ~ : -8 log-load-comma-string #-- (codeword pointer) + + ~ We consumed the string pointer at log-load time, so we apply a delta + ~ for it. + ~ + ~ Notice by the way how the nearby calls to log-load-unroll-log-address + ~ rely on this being split into two deltas like this, even though they + ~ net out to zero; we aren't doing it just for kicks. + -1 transform-apply-stack-delta + + log-load-unroll-log-address + } if + ; make-immediate + + +: log-load-dot-string-alternate + ' log-load-string-alternate entry-to-execution-token execute + + interpreter-flags @ 0x01 & { + ~ Hey look, it's the one time we're doing something simple enough that + ~ we can actually benefit from log-load-compile-dynamic-word as something + ~ more than an example to start from. :) + s" emitstring" log-load-compile-dynamic-word + } { + swap-transform-variables + L@' emitstring + swap-transform-variables + + offset-to-target-address-space , + ~ : -8 emitstring #-- (codeword pointer) + } if-else + ; make-immediate + + +~ No matter what mode we're in, we read a name at transform time. Then we +~ look up the name at either log-load time or runtime, depending. +: log-load-tick-alternate + ~ Read the name. + word value@ + + ~ Get the name onto the stack at log-load time. + swap-transform-variables + L@' litstring + swap-transform-variables + + offset-to-target-address-space , ~ : -8 litstring #-- (codeword pointer) + here @ swap packstring 8 packalign here ! + + ~ Apply a delta for the string pointer. + 1 transform-apply-stack-delta + + dropstring + + ~ Now break out the remaining behavior based on mode. + interpreter-flags @ 0x01 & { + log-load-roll-log-address + + swap-transform-variables + L@' log-load-comma + L@' log-load-find-execution-token + L@' litstring + L@' log-load-comma-string + L@' swap + L@' log-load-comma + L@' log-load-find-execution-token + L@' litstring + swap-transform-variables + + ~ The overall stack delta of this sequence is 0. + offset-to-target-address-space , + ~ : -8 litstring #-- (codeword pointer) + here @ s" litstring" packstring 8 packalign here ! + offset-to-target-address-space , + ~ : -8 log-load-find-execution-token #-- (codeword pointer) + offset-to-target-address-space , + ~ : -8 log-load-comma #-- (codeword pointer) + offset-to-target-address-space , ~ : -8 swap #-- (codeword pointer) + offset-to-target-address-space , + ~ : -8 log-load-comma-string #-- (codeword pointer) + + offset-to-target-address-space , + ~ : -8 litstring #-- (codeword pointer) + here @ s" find" packstring 8 packalign here ! + offset-to-target-address-space , + ~ : -8 log-load-find-execution-token #-- (codeword pointer) + offset-to-target-address-space , + ~ : -8 log-load-comma #-- (codeword pointer) + + ~ We consumed the string pointer at log-load time, so we apply a delta + ~ for it. As in log-load-string-alternate, this needs to be split into + ~ two parts like this. + -1 transform-apply-stack-delta + + log-load-unroll-log-address + } { + log-load-roll-log-address + + swap-transform-variables + L@' log-load-find + swap-transform-variables + + offset-to-target-address-space , + ~ : -8 log-load-find #-- (codeword pointer) + + ~ We leave an execution token on the log-load-time stack, so no delta + ~ needed. + + log-load-unroll-log-address + } if-else + ; make-immediate + + +~ Because docol requires it, we provide a special mini-version of the label +~ system. We only do L@' and L!', because that's all we need. Unlike the +~ version of this feature for the label transform, for the log-load transform, +~ we heavily restrict the use-case. +~ +~ The implementation strategy is that we ignore the label name, and store +~ the value on the stack when the generated log-load routine runs. So, each +~ instance of L@' must be closely followed by a matching instance of L!'. Each +~ label can only ever be used exactly once, and it must be a backward +~ reference. Furthermore, the stack is used in a very specific way, which the +~ transformed code must be compatible with. The easiest way to explain it is +~ by showing the interface of these words from the transformed code's +~ perspective: +~ +~ L!' is (preserved values, ..., ..., value of label +~ -- value of label, preserved values, ..., ...) +~ L@' is (value of label, preserved values, ..., ...) +~ -- preserved values, ..., ..., value of label) +~ +~ The preserved values are simply more items on the stack, which the +~ alternates take pains not to interfere with. The alternates output a roll or +~ unroll of an appropriate size. Hopefully, in reading this, you have the +~ question: How can the alternates possibly know what size is appropriate? +~ The answer is that the log-load transform carefully tracks how many items +~ the transformed code is expected to have on the stack, based on its +~ hardcoded understanding of each word that's expected to be relevant, looked +~ up by the word's name. The number of items is stored in the user-stack-depth +~ field of transformation-state. +~ +~ There is no adjustment done on the saved value, since it's created in the +~ target address space and then also used in the target address space. It +~ wouldn't actually be necessary to use this at all, since checking "here" +~ would be sufficient, but then the code would have to do something different +~ depending on which transform it's running under, and there'd have to be a +~ mechanism for that. +~ +~ This is sufficient to implement docol, and that's probably the only thing +~ it should be used for. +: log-load-L@'-alternate + word dropstring + log-load-roll-saved-label + ~ We now begin thinking of the label value as user data, so we need to + ~ notate that appropriately. + 1 transform-apply-stack-delta + ; make-immediate + +: log-load-L!'-alternate + word dropstring + ~ We cease to think of the label value as user data, so we need to notate + ~ that appropriately. + -1 transform-apply-stack-delta + log-load-unroll-saved-label + ; make-immediate + + +~ Not to be outdone by the label transform, the log-load transform also +~ offers alternates of all the high-level flow-control words: if, unless, +~ if-else, forever, and while. Unlike the label transform, we also need to +~ implement alternates of { and }, because their values need to be on the +~ stack at log-load "immediate" time, not at transform time. +~ +~ Anyway, most of the implementation is deferred to pre-packaged words in +~ language/static/log-load.e. +: log-load-left-curly-brace-alternate + log-load-roll-log-address + + swap-transform-variables + L@' log-load-left-curly-brace + swap-transform-variables + + offset-to-target-address-space , + ~ : -8 log-load-left-curly-brace #-- (codeword pointer) + + ~ It pushes a start pointer onto the stack. + 1 transform-apply-stack-delta + + log-load-unroll-log-address + ~ : { + ~ : indent + ; make-immediate + + +: log-load-right-curly-brace-alternate + ~ : deindent + ~ : } + log-load-roll-log-address + + swap-transform-variables + L@' log-load-right-curly-brace + swap-transform-variables + + offset-to-target-address-space , + ~ : -8 log-load-right-curly-brace #-- (codeword pointer) + + ~ It pushes a length value onto the stack. + 1 transform-apply-stack-delta + + log-load-unroll-log-address + ; make-immediate + + +~ (start pointer, length --) +: log-load-if-alternate + ~ : if + log-load-roll-log-address + + swap-transform-variables + L@' log-load-if + swap-transform-variables + + offset-to-target-address-space , + ~ : -8 log-load-if #-- (codeword pointer) + + ~ It pops the start and length. + -2 transform-apply-stack-delta + + log-load-unroll-log-address + ~ : blank-line + ; make-immediate + + +~ (start pointer, length) +: log-load-unless-alternate + ~ : unless + log-load-roll-log-address + + swap-transform-variables + L@' log-load-unless + swap-transform-variables + + offset-to-target-address-space , + ~ : -8 log-load-unless #-- (codeword pointer) + + ~ It pops the start and length. + -2 transform-apply-stack-delta + + log-load-unroll-log-address + ~ : blank-line + ; make-immediate + + +~ (true start, true length, false start, false length) +: log-load-if-else-alternate + ~ : if-else + log-load-roll-log-address + + swap-transform-variables + L@' log-load-if-else + swap-transform-variables + + offset-to-target-address-space , + ~ : -8 log-load-if-else #-- (codeword pointer) + + ~ It pops two pairs of start and length. + -4 transform-apply-stack-delta + + log-load-unroll-log-address + ~ : blank-line + ; make-immediate + + +~ (start, length --) +: log-load-forever-alternate + ~ : forever + log-load-roll-log-address + + swap-transform-variables + L@' log-load-forever + swap-transform-variables + + offset-to-target-address-space , + ~ : -8 log-load-forever #-- (codeword pointer) + + ~ It pops the start and length. + -2 transform-apply-stack-delta + + log-load-unroll-log-address + ~ : blank-line + ; make-immediate + + +~ (test start, test length, body start, body length --) +: log-load-while-alternate + ~ : while + log-load-roll-log-address + + swap-transform-variables + L@' log-load-while + swap-transform-variables + + offset-to-target-address-space , + ~ : -8 log-load-while #-- (codeword pointer) + + ~ It pops two pairs of start and length. + -4 transform-apply-stack-delta + + log-load-unroll-log-address + ~ : blank-line + ; make-immediate + + +~ This implements the log-load transform for a single word. It is directly +~ analogous to "interpret", and reading language/dynamic/interpret.e may help +~ in understanding it, though it's meant to still make sense on its own. +~ +~ It expects to be called from "log-load-transform", below, which loops. +~ +~ (-- done) +: log-load-transform-one + word + + ~ If no word was returned, end the transformation. + dup 0 = { drop 1 exit } if + + ~ The string is on the top of the stack, so to get a pointer to it we get + ~ the stack address. + ~ (string) + value@ + + ~ Check whether it's one of the words we have alternates for, and look up + ~ the alternate if so. + 0 swap + ~ (name as stack string, placeholder, name pointer) + dup s" create" stringcmp 0 = { + swap drop ' log-load-create-alternate swap } if + dup s" :" stringcmp 0 = { + swap drop ' log-load-colon-alternate swap } if + dup s" self-codeword" stringcmp 0 = { + swap drop ' log-load-self-codeword-alternate swap } if + dup s" ;" stringcmp 0 = { + swap drop ' log-load-semicolon-alternate swap } if + dup s" ;asm" stringcmp 0 = { + swap drop ' log-load-semicolon-assembly-alternate swap } if + dup s" [" stringcmp 0 = { + swap drop ' log-load-left-square-brace-alternate swap } if + dup s" ]" stringcmp 0 = { + swap drop ' log-load-right-square-brace-alternate swap } if + dup s" ," stringcmp 0 = { swap drop ' log-load-comma-alternate swap } if + dup s" variable" stringcmp 0 = { + swap drop ' log-load-variable-alternate swap } if + dup s" keyword" stringcmp 0 = { + swap drop ' log-load-keyword-alternate swap } if + dup s" ~" stringcmp 0 = { + swap drop ' log-load-tilde-alternate swap } if + ~ It is nontrivial to construct a string with a double-quote in it. + dup ' s" entry-to-name stringcmp 0 = { + swap drop ' log-load-string-alternate swap } if + dup ' ." entry-to-name stringcmp 0 = { + swap drop ' log-load-dot-string-alternate swap } if + dup ' ' entry-to-name stringcmp 0 = { + swap drop ' log-load-tick-alternate swap } if + dup s" L@'" stringcmp 0 = { swap drop ' log-load-L@'-alternate swap } if + dup s" L!'" stringcmp 0 = { swap drop ' log-load-L!'-alternate swap } if + dup s" {" stringcmp 0 = { + swap drop ' log-load-left-curly-brace-alternate swap } if + dup s" }" stringcmp 0 = { + swap drop ' log-load-right-curly-brace-alternate swap } if + dup s" if" stringcmp 0 = { swap drop ' log-load-if-alternate swap } if + dup s" unless" stringcmp 0 = { + swap drop ' log-load-unless-alternate swap } if + dup s" if-else" stringcmp 0 = { + swap drop ' log-load-if-else-alternate swap } if + dup s" forever" stringcmp 0 = { + swap drop ' log-load-forever-alternate swap } if + dup s" while" stringcmp 0 = { swap drop ' log-load-while-alternate swap } if + ~ (name as stack string, 0 or alternate entry pointer, name pointer) + + ~ If we have an alternate, we want to run that now, regardless of what + ~ mode we're in. They're all flagged as immediate, but we don't even bother + ~ checking, because it doesn't fully describe their behavior anyway. With + ~ this transform there's three potential times at which we might execute + ~ things, not two. The alternates are more immediate than immediate; they + ~ run NOW, during the transformation. + over { + drop dropstring-with-result + entry-to-execution-token execute + 0 exit + } if + drop drop + ~ (name as stack string) + + ~ Now we might have a compiled word, an immediate word, or an integer + ~ literal. Recall that the word won't actually be looked up until the + ~ routine we're producing is run - that's the whole point - so there's no + ~ check we can perform now that will tell us whether the word we have exists + ~ in the eventual log. Instead, we invert the usual fallback order and + ~ check whether the word could be an integer literal. If it is, we'll + ~ handle that; if not, we'll assume it'll eventually exist. + ~ + ~ This means that code that's run with the log-load transform can't + ~ shadow an integer literal with a word definition. Oh, so limiting. + value@ read-integer 0 = { + ~ It's a number. + ~ + ~ (name as stack string, integer value) + dropstring-with-result + ~ (integer value) + + interpreter-flags @ 0x01 & { + ~ We're in compile mode, so we want to generate code which will compile + ~ the number. + log-load-compile-literal + 0 exit + } if + + ~ We're in interpret mode, so we want to generate code which will push + ~ the number to the stack. + ~ + ~ This is an immediate effect, so once we've done it, we update the user + ~ stack depth. An integer literal is a stack-depth delta of 1. + swap-transform-variables L@' lit swap-transform-variables + offset-to-target-address-space , ~ : -8 lit #-- (codeword pointer) + ~ : provide-hex + , ~ : -8 # #-- (integer literal) + 1 transform-apply-stack-delta + 0 exit + } if + ~ (name as stack string) + + ~ We know it's a regular word, and we're assuming it will exist at + ~ runtime. We of course have no way to check what flags it will have, which + ~ means immediate words don't work with this transform. We still treat it + ~ differently based on whether we're in compile mode. + interpreter-flags @ 0x01 & { + ~ We're in compile mode. We compile code that compiles the word. + ~ + ~ Since that's not an immediate effect, we don't need to update the + ~ user stack depth. Instead, this is a key spot where we rely on that + ~ knowledge, to roll and unroll the log address. + value@ log-load-compile-dynamic-word + dropstring 0 exit + } if + ~ (name as stack string) + + ~ At this point we know we're in immediate mode and have a regular word. + ~ While we still have the name pointer handy, we check what delta this word + ~ will cause for the user stack depth, when run. See + ~ transform-get-stack-delta for more explanation of this. We keep track of + ~ the proposed delta until we actually process the word, down below. + value@ dup transform-get-stack-delta swap + ~ (name as stack string, proposed stack depth delta, name pointer) + + ~ We're in immediate mode. We compile code that runs the word immediately. + ~ We check whether there's a label for the word; if there is, we'll output + ~ that. Otherwise we'll output code that looks it up in the log and runs it. + ~ + ~ Just like in label-transform, we use find-label to check whether a label + ~ exists without declaring a dependency on it, then if it does, we do + ~ use-label to ask for its value. + ~ + ~ There's one additional wrinkle to remember here: We're running inside + ~ the label loop, and warm-start appears before all the normal words in the + ~ executable. So all the labels we'll be checking are forwared references, + ~ and on the very first pass they definitely won't be defined. That's fine + ~ though, they will exist on all subsequent passes, so things will + ~ definitely still converge. + ~ + ~ The first pass will never accidentally think it succeeded, because even + ~ the reference to L' cold-start from the ELF header is a forward reference + ~ and won't exist on the first pass. + dup + swap-transform-variables + find-label + swap-transform-variables + { + ~ Again just like in label-transform, we declare our use of the label + ~ and get a value for it. + ~ : provide-string-copy + swap-transform-variables + intern-label use-label + swap-transform-variables + ~ (name as stack string, proposed stack depth delta, label value) + + ~ Like in label-transform, this is a codeword pointer, so we just output + ~ it directly. Also as before, because we don't have to examine it, we + ~ don't have to do anything special in the case where it's zero due to the + ~ way the label loop works. + ~ + ~ This is an immediate effect, so once we've done that, we update the + ~ user stack depth. + offset-to-target-address-space , + ~ : -8 # #-- (codeword pointer) + transform-apply-stack-delta + dropstring 0 exit + } if + ~ (name as stack string, proposed stack depth delta, name pointer) + + ~ There's no label for the word; that means it wasn't statically + ~ compiled-in to the target executable. So we output code that looks up the + ~ word by name on the log, then calls it. + ~ + ~ This is also an immediate effect, despite being distinct from the above + ~ case, so once we've done that, we update the user stack depth. + ~ + ~ There's no such thing as not finding the word, with this transform. So + ~ once we're done, we just exit. + log-load-execute-dynamic-word + transform-apply-stack-delta + dropstring 0 ; + + +~ This implements the log-load transform for all words in a region given as +~ an input string. It is directly analogous to "quit", in +~ language/dynamic/interpret.e, but is far more complex. +~ +~ (output buffer start, output point, input string pointer +~ -- output buffer start, output point) +: log-load-transform + ~ : blank-line + ~ : This is the start of the log-load routine, which is a long block of + ~ : code that has been processed by the log-load transform. It is part of + ~ : the warm-start routine, which does other things before and after it. + ~ : For clarity's sake, to avoid weird nesting, the section header above + ~ : calls this the warm-start section of the executable, but in a sense it + ~ : might be more accurate to call it the log-load section. + ~ : + ~ : The routine is interpreted Forth code, just as the warm-start routine + ~ : as a whole is; see above. + ~ : + ~ : The job of the log-load routine is to create runnable code in the + ~ : dynamic log, at runtime. The normal way to create a Forth executable + ~ : is to use label-transformed code to implement a minimal set of + ~ : functionality needed for the log-load routine to run, then invoke the + ~ : log-load routine to copy a more complete set of functionality into + ~ : memory. + ~ : + ~ : The log-load routine is super important, but also extremely verbose, + ~ : which necessarily makes it hard to read. If you're skimming this file + ~ : for the first time to get a sense of what it contains, you might want + ~ : to search for the words "label transform" to get to the stuff that makes + ~ : better introductory reading. + ~ : blank-line + + main-input-buffer dup push-input-buffer + swap attach-string-to-input-buffer + + ~ Save the old values of "here" and "latest", and set the initial values + ~ of the internal ones. These values need to persist across iterations, + ~ since client code will make its own updates to them and then rely on those + ~ updates having taken effect. So we do the swap just once, here outside the + ~ loop, and set it back when the loop ends. + ~ + ~ We also take this opportunity to initialize the output-buffer-start and + ~ user-stack-depth fields of transformation-state. + here @ transformation-state transformation-state-saved-here ! + latest @ transformation-state transformation-state-saved-latest ! + swap transformation-state transformation-state-output-buffer-start ! + 0 transformation-state transformation-state-user-stack-depth ! + here ! + 0 latest ! + ~ Now the stack has nothing of ours on it, so client code can do its thing. + + ~ It's important that the stack has nothing of ours on it that persists + ~ across iterations, so that client code can add and remove stuff there as + ~ it sees fit. + { log-load-transform-one + ~ (..., done) + + ~ When the loop is done, get the real values of "here" and "latest" + ~ back. The internal "here" is also the output point, and will become our + ~ return value. The internal "latest" is discarded. + { transformation-state transformation-state-output-buffer-start @ + here @ + transformation-state transformation-state-saved-here @ here ! + transformation-state transformation-state-saved-latest @ latest ! + ~ (output buffer start, output point) + + ~ Though we don't actually use transformation-state outside of this + ~ invocation, for tidiness we zero it out. + 0 transformation-state transformation-state-saved-here ! + 0 transformation-state transformation-state-saved-latest ! + 0 transformation-state transformation-state-output-buffer-start ! + 0 transformation-state transformation-state-user-stack-depth ! + + ~ Also put the input source back how it was. + main-input-buffer pop-input-buffer + + exit } if } forever ; + + +~ Hex transform implementation +~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~ +~ The following code is all part of implementing the hex transform. For +~ conceptual overview, see the top of this file. +~ +~ The hex transform operates under some very complex memory-management +~ assumptions, even compared to the other transforms, so it's worth going over +~ all that and studying how it comes together, even though much of it is +~ material that has been explained elsewhere. This explanation is here with +~ the implementation details rather than at the top of the file, because it's +~ only necessary to understanding how to modify the transform, not how to use +~ it. +~ +~ This is the bit of the hex transform where all the memory-management +~ assumptions are in play at once, so, a quick refresher. The transform +~ itself is running in the outer context; hex-trace and its siblings are +~ part of Evocation's main log and dictionary. This is also true for the +~ label and log-load transforms. Evocation has two global variables, +~ "here" and "latest", which are the root of all references to the log and +~ dictionary, respectively. Remember, the log is the structure used for +~ memory allocation, and the dictionary, which forms the majority of the +~ log, is the structure used for looking up words by name. +~ +~ Every transform, in its various ways, is changing what it means to +~ "execute" and "compile" code; therefore, when the transforms are invoked, +~ they are passed an "output buffer" which is used as the destination for +~ any compilation that occurs, and any scratch space that is needed. When +~ more than one transform is running at once (such as when the hex transform +~ operates on code that is itself running the label transform), each +~ transform has its own separate output buffer. +~ +~ Furthermore, the transformation facility helps each transform to swap out +~ the values of "here" and "latest", and restore them later, so that the code +~ being transformed will see the output buffer as if it is the "real" log, and +~ will see the separate dictionary stored within the output buffer as if it is +~ the "real" dictionary. +~ +~ All this stuff about the output buffer applies to every transform. The +~ hex transform takes it one step further, and as such there are two +~ additional memory regions that are important to understanding it. The first +~ is the label-loop buffer, which is allocated out of the inner log, and used +~ to hold draft versions of the binary output that the inner compilation +~ process would eventually emit to the standard output stream, if it were +~ being allowed to emit output in the normal way. The second is the output +~ metadata buffer, which is allocated out of the outer log, and used to hold +~ metadata such as comments and formatting pertaining to code structure, which +~ will need to be referenced when actually outputting the final hex dump. +~ +~ So, the label-loop buffer can be thought of as scratch space for the +~ compiled binary output from the code being transformed, and the output +~ metadata buffer can be thought of as the scratch space for the formatted hex +~ dump that will be made out of that output. It's important not to confuse +~ either of them with the output buffer, which is the scratch space used as +~ part of the compilation process. +~ +~ Confusingly, the output buffer and the label-loop buffer both contain +~ logs and dictionaries, at least potentially, if the code being compiled is +~ Forth code. It is also possible to use the hex transform on assembly code, +~ in which case the label-loop buffer won't have a log in it but the output +~ buffer still will. In practice, the hex transform frequently winds up +~ needing to be careful about the log and dictionary in the output buffer, but +~ rarely does anything that directly touches the log and dictionary in the +~ label-loop buffer, so you can usually ignore the latter when reading this +~ code. +~ +~ Allocating the output buffer and output metadata buffer is the +~ responsibility of the caller of hex-transform. Allocating the label-loop +~ buffer is the responsibility of label-loop, which is called by the code +~ being transformed, but the hex transform intervenes in that allocation to +~ save information about it. +~ +~ The output buffer is referenced during the transform via the +~ transformation-state-output-buffer-start field of the global +~ "transformation-state" variable, and the output metadata buffer is +~ referenced via the transformation-state-output-metadata field of that same +~ variable. The label-loop buffer is referenced from the +~ hex-output-metadata-label-loop-buffer-start and -length fields of the +~ output metadata's top-level header, and the code being transformed passes +~ around its own reference to that same buffer. +~ +~ Finally, recall that, when Evocation itself is the thing being +~ transformed, the code under transformation includes its own copy of the +~ transformation facility, along with all the rest of Evocation. When the code +~ under transformation invokes the label and log-load transforms, it will do +~ so using the copies of those transforms which have been transformed by the +~ hex transform. The same goes for most Evocation internals invoked by the +~ compilation process, except for whatever specific exceptions the hex +~ transform creates. Do not confuse the inner copies of anything with the +~ outer copies. Also try not to worry too much about the inner-inner copies, +~ let the inner transforms deal with them. +~ +~ Got all that background fresh in your mind? Okay, time to get concrete! +~ The output metadata is a header structure followed by an array of entry +~ structures, as follows: +~ +~ (header start) +~ 0x00 - 0x07 Label-loop buffer start +~ 0x08 - 0x0f Label-loop buffer length +~ (entry start) +~ ... + 0x00 - ... + 0x07 Data pointer +~ ... + 0x08 - ... + 0x0f Referenced data length +~ ... + 0x10 - ... + 0x17 String pointer +~ (entry fields repeat at successive offsets, until...) +~ ... + 0x00 - ... + 0x07 (end) Zero, used as a delimiter +~ +~ +~ Magic comment syntax +~ ~~~~~~~~~~~~~~~~~~~~ +~ +~ How, the sufficiently determined reader may ask, does all this output +~ metadata get created? Surely, the reader may rightly think, creating +~ metadata entries which fully describe a compilation process must entail +~ gathering and synthesizing information from disparate stages of that +~ process, which seems quite involved. +~ +~ That is correct: It is quite involved. It would be nice if there were a +~ simple trick, but this appears to be essential complexity. Several +~ approaches were considered. In principle it would be possible to have a +~ specific trap defined here in this file for every word that generates +~ nontrivial output, and keep knowledge of what information to gather from +~ those words in the trap definitions. However, that would bind the important +~ semantics quite closely to the already-overwhelming topic of the +~ transformation data structures, and make it all but impossible to port +~ Evocation to a new architecture. The knowledge we wish to capture is really +~ knowledge about an instruction set architecture, not about data structures. +~ +~ Instead, following the principle that different kinds of knowledge on the +~ same topic should live close together, we've defined a magic-comment syntax +~ which allows the assembly-language words in language/static/amd64.e to +~ describe their own metadata, and is versatile enough to do the same for +~ future architectures. +~ +~ When the compiler operates normally, without the hex transform, these +~ magic comments are simply regular comments, and are discarded as the +~ interpreter finds them. When the compiler runs under the hex transform, they +~ are instead recognized as commands to be processed by the hex transform. +~ Magic comments can only appear in the body of a word being compiled, not at +~ the top level. The hex transform intervenes in the compilation process to +~ insert calls to its own internal words, which are executed, not when the +~ comment is read, but later, when the word it was compiled into is called. +~ When the word in question is itself part of the compiler, the magic comments +~ will be able to see and work with all the state that exists as it runs, and +~ gather it as they wish. +~ +~ There are several kinds of magic comment. All of them begin with the +~ characters : and space as the first two characters of the comment body (and +~ don't forget that there's a space after ~, as well). The syntax on the rest +~ of the line determines what type of command it is. +~ +~ Here are some examples: +~ +~ ~ : Intriguing commentary goes here. +~ ~ : 5 This comment describes the next five bytes of output. +~ 777 pack32 ~ : -4 This comment describes the previous four bytes. +~ +~ ~ : indent +~ ~ : deindent +~ ~ : fresh-line +~ ~ : blank-line +~ +~ ~ : 1 adjust-length +~ ~ : -10 adjust-start +~ ~ : 1 data-adjust-length +~ ~ : -1 data-adjust-start +~ +~ ~ : 0x1000 set-output-point +~ ~ : 8 adjust-output-point +~ ~ : data-set-output-point +~ ~ : 1 data-adjust-output-point +~ +~ ~ : 1 suppress +~ ~ : delete-first-comment-at-output-point +~ ~ : delete-last-comment-at-output-point +~ +~ ~ : Do a thing with # and # ? +~ ~ : provide-decimal +~ ~ : provide-hex +~ ~ : provide-hex8 +~ ~ : provide-hex16 +~ ~ : provide-hex32 +~ ~ : provide-hex64 +~ ~ : provide-string +~ ~ : provide-string-copy +~ ~ : provide-keyword +~ ~ : provide-data +~ ~ : drop-subitem +~ ~ : swap-subitems +~ ~ : 3 roll-subitems +~ +~ By the way, the reason it's possible to write these examples is that +~ they're all embedded within an outer comment, the text you're reading, so +~ they don't look like real magic comments to the hex transform. Hopefully +~ that's not too confusing; there isn't really another way. +~ +~ So. You'll have noticed that some magic comments seem to be descriptive +~ text, and others seem to be commands. Every magic comment, in one way or +~ another, creates or manipulates the metadata entries, which are stored in +~ an array inside the output metadata as global mutable state. There are +~ several phases of processing, and if you've gotten this far, you likely need +~ to understand what they are and what order they happen in. +~ +~ The simplest kind of comment is the "line comment", the first example +~ above. Any comment that doesn't match some more-specific syntax is treated +~ as a line comment. It has not escaped Irenes' notice that this means your +~ syntax errors will seem to half-work; sorry about that. When a line comment +~ is executed, it creates a metadata entry at the end of the array of entries, +~ of -entry-type-line-comment, attached to the latest output point. +~ +~ This is the general pattern: Any magic comment that creates a metadata +~ entry will always create it at the end of the entry array, and will always +~ attach it to the latest output point as of the time the comment executes. +~ Notice that, although the entry array will eventually be sorted by the +~ addresses entries are attached to, at the time entries are created they're +~ simply appended. This is rarely an important distinction, but when it is, it +~ is. +~ +~ What is the latest output point? Well, the hex transform has trapped all +~ the words from core that compilation processes use to directly append bytes +~ to the label-loop buffer, including pack8, packstring, and the variants +~ thereof. The traps are able to tell when the destination is in the +~ label-loop buffer, and they ignore any writes that aren't. The traps keep +~ track of the next address AFTER whatever was just written, and store that +~ in the output metadata header so magic comments can make use of it. +~ +~ It's necessary to take this sneaky approach, because the compiler is free +~ to write things in any order it wants; the label loop goes back to the start +~ every so often, and the high-level flow-control words even slide things +~ around after the fact. The compiler is also free to pass around addresses +~ however it wants within itself, and without restricting that, there's no way +~ for the hex transform to know which value represents the place stuff MIGHT +~ be written to. +~ +~ The semantics of the latest output point are simple enough that they're +~ easy to keep in mind while writing magic comments. In practice, the output +~ point is almost never the cause of confusion about the behavior of magic +~ comments, but it's important to know about since it's the input the whole +~ thing starts with. +~ +~ Now, one thing to keep in mind is that every metadata entry has both an +~ address in the label-loop buffer that it attaches to, and a length field. +~ Line comments insert themselves in between lines of output in the hex dump, +~ and don't otherwise affect the surrounding formatting, so they don't need to +~ make use of the length and it's always set to zero. +~ +~ Now the second example above, which starts with the number 5 before its +~ descriptive text. When the first space-separated word of a magic comment's +~ body is a valid integer literal, this is taken as a parameter; different +~ types of magic comment use that parameter in different ways. The parameter +~ can be positive or negative, and can be written in any base Evocation +~ normally supports. +~ +~ When a parameter is given (even if it's zero), and the comment doesn't +~ match some other more-specific syntax, it's treated as a "suffix comment". +~ Much like a line comment, when a suffix comment is executed, it creates a +~ metadata entry at the end, of -entry-type-suffix-comment, attached to the +~ latest output point. The command uses the parameter as the initial value of +~ the entry's length field, but it's worth being aware that other commands may +~ change it later. +~ +~ It's called a suffix comment because, when all the passes described below +~ have run and the formatting happens, it will appear at the end of a line of +~ text, after the byte values it's attached to. Things like assembly-language +~ instructions should almost always be suffix comments, because this makes it +~ easy for a human reader to see which bytes of machine code go with them. +~ +~ There's also an implicit way to create a suffix comment, which is to have +~ non-blank words of any kind on the line prior to the magic comment. In this +~ case, the default is to be a suffix comment rather than a line comment. +~ Suffix comments created this way will have zero length. +~ +~ Before we move on from magic comments that produce descriptive text, also +~ look at the third example above. This is another suffix comment, but it's +~ got a negative number as its parameter. It's got a bit of code on the line +~ before it, to illustrate the typical way these negative parameters are used. +~ Just as a suffix comment with a positive length describes a span of bytes +~ with its start anchored at the address the comment is attached to, so when +~ the length is negative, the span of bytes has its END anchored at the +~ address. +~ +~ You can see examples of how this syntax is useful in compilation/elf.e, +~ where it describes fields of various header structures. When it fits the +~ situation, this notation is concise and easy to visually skim both in the +~ actual source code, and in the generated hex dump. +~ +~ Now let's look at some magic comments that do more interesting things. +~ The two commands "indent" and "deindent" both create an entry of +~ -entry-type-indent. Neither takes a parameter as part of the comment syntax, +~ but the entry structure has an internal value which is set to 2 for "indent" +~ and -2 for "deindent". When output is eventually formatted, these entries +~ will be used to adjust the amount of space at the beginning of each line, +~ which can be very useful to visually group code together. The formatting +~ process has some mutable state of its own, which doesn't exist until this +~ very late stage (long after magic comments execute); among other things, +~ there's a notion of current indentation level, which is preserved until +~ these entries adjust it. +~ +~ The commands "fresh-line" and "blank-line" create entries of +~ -entry-type-fresh-line and -entry-type-blank-line, respectively. Just as the +~ output formatting knows how deeply indented it is, it also knows whether +~ it's at the start of a line. A fresh-line entry will insert a line break, +~ but only if we're somewhere deep in a line; if we're already at the start, +~ it does nothing. A blank-line entry will do the same thing as fresh-line, +~ then it will add another line break, so that the result is to output a fully +~ blank line in the hex dump. This can be useful to break things up visually. +~ +~ Thus far, though we've taken pains to allude to the fact that there are +~ many stages of processing of metadata entries after they're created, we +~ haven't introduced any interesting features that rely on that. That changes +~ with the "adjust-line" command. When this command executes, it doesn't +~ create a new metadata entry. Rather, it scans backwards from the end of the +~ entry array, to find the most recently-created entry which is a line or +~ suffix comment. It skips over entries of other types. Then it adjusts the +~ length of that entry, in-place! The parameter of adjust-line may be positive +~ or negative, and either way it's added to the pre-existing length of the +~ entry it modifies. +~ +~ If that doesn't sound confusing, you may not have fully appreciated that +~ adjust-line commands don't have to come in the same word as the command that +~ created the entry they're modifying. For example, in +~ language/static/amd64.e, rex-w is a common helper word called as part of +~ many assembly instructions. It always outputs a single byte. Commentary in +~ that file describes an accounting convention whereby keeping track of this +~ byte is rex-w's job, so rex-w also has a magic comment with an adjust-length +~ command. Whatever word calls rex-w will have already created a suffix +~ comment giving the name of the instruction it's implementing, by the time +~ this is called, and rex-w's call to adjust-length will find that comment's +~ entry and modify it. +~ +~ The use of this may not seem like much for a word like rex-w, which always +~ outputs the same number of bytes, but now look at the definition of +~ addressing-indirect-reg64. This is a common addressing mode shared by many +~ instructions, and called as a helper from various instruction words. Notice +~ how it has to generate a different encoding when it's given the register +~ :rsp, which has an extra SIB byte that's not needed for any other register. +~ There's no magic comments in addressing-indirect-reg64 itself, but the word +~ "sib" is another helper which does an appropriate adjust-length, if and only +~ if sib is actually called. +~ +~ You can see how it might become quite confusing trying to decide which +~ word should take responsibility for a particular byte, when there are +~ multiple layers of abstraction like this. It's especially confusing because +~ this is an extra form of side-effect which acts on global mutable state, and +~ is out-of-band compared to the normal Forth value stack. When you're +~ implementing an instruction set architecture, you should strongly consider +~ defining an accounting convention which makes that clear and unambiguous. +~ Feel free to consult the convention described at the top of +~ language/static/amd64.e for inspiration. +~ +~ All the adjust-* commands change something immediately, as they execute. +~ What they adjust differs. In the cases of adjust-length and adjust-start, +~ it's a field of a recent entry. Notionally, adjust-start does the same thing +~ as adjust-length but for the start address the entry is attached to, rather +~ than the length of the attachment. Also, while adjust-length only affects +~ comment entries, adjust-start affects any entry that produces output, such +~ as fresh-line. +~ +~ The various -output-point commands are used to directly interact with the +~ notion of the latest output point, which can be useful if a location has +~ been skipped over and there are commands which should be run before +~ outputting anything at the new location. The simplest variant is +~ set-output-point, which sets it to the parameter value. The data-set- +~ variant obtains a value in the manner of the other data-* commands, adds it +~ to the parameter value, and sets the latest output point to the result. The +~ adjust- and data-adjust- variants, like other adjust commands, add a value +~ to the existing one. +~ +~ Notice that the general rule is that data-set-* commands use their +~ parameter as an addend, while data-adjust-* commands use it as a multiplier. +~ +~ The data-adjust-* variants deserve further commentary, though you may find +~ it more productive to read and understand the entire explanation about +~ provide-* commands before worrying about these. These variants adjust the +~ same things as the non-data ones, but instead of adjusting by a fixed +~ amount, they look for the most recent -entry-type-saved-data metadata entry. +~ They delete that entry from the entry array entirely, and use the value +~ saved in it as the amount to adjust by. Such entries are created by the +~ provide-data command, on which more below. Additionally, the data-adjust-* +~ commands use their parameters as scaling factors to the adjustment, and +~ these factors may be negative. You'll usually specify a parameter of 1 or +~ -1. +~ +~ We will revisit the topic of adjust-length later, when we describe the +~ comment template feature, but for now we'll leave it there. +~ +~ Direct your attention to the "suppress" command. Sometimes, you intend to +~ call a helper word which is going to create its own entries, but you know +~ that those entries don't suit your purpose, and you want to ignore them. +~ For example, you may be calling a word which outputs a binary string and +~ creates entries describing the string, but you're doing something more +~ specific and would prefer to create entries that describe your special case, +~ instead. +~ +~ When you need this, you can call "suppress" with an integer parameter. If +~ the parameter is 1, the next time a metadata entry would have been created, +~ it won't; then things go back to normal. If you want to suppress more than +~ one entry in a row, you can pass it a larger number. If you change your +~ mind, you can pass it a negative number. At the time magic comments are +~ executing, there's a running count of how many entries should be suppressed, +~ which is decremented any time it's greater than zero and an entry would be +~ created, and adjusted every time a call to "suppress" is executed. +~ +~ Sometimes, an entry has already been created, and you now need to get rid +~ of it. While there are reasons to be cautious of more-general mechanisms, +~ the commands delete-*-comment-at-output-point will search for a metadata +~ entry which is one of the comment types and which has its start position set +~ to the same address as the current output point. The -first- variant will +~ search forward from the start of the array, while -last- will search +~ backward from the end; both will stop as soon as they encounter one. This is +~ usually desired when overwriting a location that has already been written +~ to. +~ +~ Take a quick look at the remaining examples. These are part of the comment +~ template feature. We'll need to describe the other phases of processing +~ before we can talk about how they work, but notice that one of them has the +~ space-separated word # inside it, while the others are commands whose names +~ start with provide-. Any line or suffix comment can have # in it, which will +~ consume a subitem from somewhere else, replacing the # like it's a template. +~ The various provide- commands are the origins of those subitems. Which # +~ goes with which provide- command? Thereby hangs a tale. +~ +~ So. Let's pretend for the moment that we've finished talking about things +~ that happen when magic comments execute. We haven't, quite, but to finish it +~ out you'll need the full picture in your head. +~ +~ All this time, the hex transform has been executing the compiler, which +~ is running its guts inside a utility called the label loop (see +~ compilation/labels.e). The compiler guts have been calling their own various +~ internal words and accumulating their binary output in the label-loop +~ buffer. While this is happening, the hex transform's various traps have been +~ tracking the latest output point, executing magic comments, and doing other +~ vital but simpler tasks. +~ +~ Eventually the label-loop will reach the end of its iteration, and then it +~ will either conclude that all the labels in the program have been resolved +~ successfully, or it will notice that some of them were given addresses that +~ don't work. If the labels aren't all resolved, the label loop will start +~ over from the beginning, running all the same compiler guts again. The hex +~ transform will notice that this happened, clear out all the metadata +~ entries, and restart its own activities as well. +~ +~ When the label loop finally declares success, and exits, the compiler will +~ want to actually produce its binary output. The hex transform has trapped +~ this, too! Now there's a postprocessing phase which cleans up all the +~ metadata entries, and then finally there's a formatting phase where the +~ entries are "run", as if they're a program, and the final hex dump with all +~ its comments is output. +~ +~ As a side note, the hex transform doesn't stop the compiler short; it lets +~ the compiler decide which bytes will be output and in what order. In +~ practice, Evocation and the various supplemental programs around it only +~ ever write their output in a single giant chunk, but the formatting phase +~ actually runs every single time the compiler calls sys-write, and it only +~ handles the portion of the label-loop buffer which the compiler passed to +~ sys-write. +~ +~ The postprocessing phase is idempotent - doing it twice is the same as +~ doing it once - so it's simply called every time sys-write is, before the +~ formatting phase starts. Postprocessing considers every metadata entry, +~ every time it runs, even when sys-write is called on something less than +~ the whole buffer. +~ +~ If you add postprocessing features, make sure to keep them idempotent. +~ While it's possible to imagine reasons one might want to have back-and-forth +~ interplay between successive formatting and postprocessing phases, Evocation +~ goes to some trouble to make sure there is none. In Irenes' opinion, this +~ is already quite enough complexity and is nearing the limit of what humans +~ will find worthwhile. If you break idempotence, you are hereby warned that +~ you are at risk of falling into the abyss[1]. +~ +~ Postprocessing re-organizes entries, without changing their meaning, but +~ in a way that will make them easier to format. There are two things it +~ does, and they happen simultaneously: The whole array is bubble-sorted; +~ and some suffix comment entries are split apart. +~ +~ The bubble sort resolves ordering first by the address each entry is +~ attached to, low-to-high. As a tiebreaker when two entries have the same +~ address, a comment entry is sorted after a stack-manipulation entry. The +~ tiebreaker rule means that stack-manipulation commands can come after the +~ last byte that's within the comment's span; otherwise they would always have +~ to be before it. If neither criterion applies, the entries will stay in +~ whatever order they were in to begin with; that is, it's a stable sort. +~ +~ The comment splitting applies only to suffix comments, and only when their +~ length is not zero. They become a fresh-line entry at the start of the span +~ the comment originally attached to, and a zero-length suffix comment entry +~ at the end of the span. +~ +~ Please notice that adjust-length commands happen when magic comments are +~ executed, which was long, long ago by now. By the time the postprocessing +~ happens, every entry's length has been as adjusted as it's ever going to be. +~ This is important, because the order of entries is going to matter during +~ the formatting phase. +~ +~ Notice also that the splitting of suffix comment entries changes what +~ address the comment is attached to (unless it's a negative length, which is +~ already at the end of its span). Any entries that were attached to output +~ points somewhere inside the span will now definitely be before the suffix +~ comment's entry in the entry array, whereas up till now they were likely +~ somewhere after it, depending on the exact order everything was created in. +~ +~ The postprocessing phase is conceptually simple - unlike the magic-comment +~ execution phase before it, and unlike the formatting phase after it, there +~ isn't any mutable state inside the phase that would make it matter what +~ order things happen in. Enjoy this, because ordering is about to become +~ super confusing. +~ +~ Now let's consider the formatting phase. We've already mentioned that +~ there is some global state which is tracked throughout the phase, such as +~ the current indentation level and whether the output is at the start of a +~ line. The formatting phase is given an overall block of bytes in the +~ label-loop buffer, which it iterates through in order from beginning to end. +~ Any time the formatting phase reaches an address which has metadata entries +~ attached to it, all the entries at that address are executed, in the order +~ they exist in the entry array. +~ +~ Yes, that's right, executing an entry is different from executing the +~ magic comment that created the entry. That's important. +~ +~ The line-comment and suffix-comment entries do the obvious things: They +~ print out the appropriate comment text. Suffix comments are also all +~ horizontally aligned with each other, to make them easier to visually skim. +~ The two kinds of comment entries also do a less-obvious thing, which is fill +~ in any # template items that are part of the comment text, in order from +~ left to right. +~ +~ Where do those items come from? Well, there's a separate stack, the +~ subitem entry stack, which is kept in the hex output metadata header. The +~ subitem entry stack has an extremely small, fixed maximum depth. The maximum +~ depth is deliberately small, so that any leaks onto it will be detected +~ somewhere close to their root causes, for ease of diagnosis. Unusually for +~ Evocation, if you underflow or overflow it, you'll get a friendly error +~ message rather than a segfault - how nice! Of course, the error message will +~ appear in the middle of your regular output and corrupt it, but you can't +~ have everything... +~ +~ Each time a comment template is being filled and encounters a # +~ placeholder, an item is popped off the subitem entry stack and formatted +~ according to what type of entry it is. It's called an "entry stack" for a +~ reason: it's a stack of pointers to metadata entries, whose backing store +~ is in this very same entry array that we're executing. So, all the various +~ internal fields of an entry are available to provide rich behavior which +~ allows different kinds of values to be formatted in different ways. +~ +~ For those used to languages with automatic memory management, keep in mind +~ that there's no language-provided guarantee that the thing a pointer points +~ to still exists, and hasn't been replaced with some other +~ misleadingly-similar thing. The separation of all this stuff into phases +~ provides that guarantee, because there's nothing in the formatting phase +~ that destroys or moves metadata entries in the array, but if that separation +~ were to break down, astonishingly complex wrong behavior would be entirely +~ likely. If you're reading all this, it's probably to touch that code, so +~ please keep those consequences in mind. +~ +~ How do entry pointers get pushed onto the subitem entry stack, and where +~ do the values inside them come from? Okay, now this is the payoff, the part +~ where you need to understand all these phases at once. Cast your mind all +~ the way back to the magic-comment execution, and take another look at the +~ provide-* examples above. Recall that magic comment commands execute when +~ the word they're contained in is running. For the other comment commands, +~ this is mostly a curiosity; we talked about adjust-length, above, but that's +~ still just recording the fact that the word did or didn't run. +~ +~ The provide-* commands are deeper than that. They peek at the top item on +~ the stack OF THE COMPILER, the code being transformed. The -decimal and +~ -hex* variants create a corresponding -entry-type-push-subitem- metadata +~ entry of the matching type, and copy the numeric value into it. +~ +~ The -string variant expects to find a pointer to a null-terminated string; +~ -string-copy does too, but copies the contents of it to a safe place +~ immediately, on the assumption they'll be overwritten. The -keyword variant +~ expects to find a keyword (a word which, when executed, pushes its own +~ execution token onto the stack); the command extracts the name of the +~ keyword from the dictionary entry header. All three of these save pointer to +~ their respective strings in a push-subitem-string metadata entry. +~ +~ Admit it, you thought writing out "metadata entry" so many times instead +~ of just "entry" was silly because there was no other kind of entry in +~ consideration, right up until you got to "dictionary entry". If so, hey +~ look, neat, right? ;) +~ +~ The -data variant treats the value as opaque, and creates an +~ -entry-type-saved-data metadata entry. Unlike the others, these entries do +~ not affect the subitem entry stack; they're consumed and removed from the +~ entry array before entry execution even starts, by the matching data-* +~ command. +~ +~ Anyway, notice that when the various provide-* commands execute, all +~ they're doing is capturing values and creating metadata entries. Like all +~ metadata entries, these are attached to the latest output point as of the +~ time the magic comment executes. In particular, the subitem entry stack +~ isn't touched until all the way at the end, in the formatting phase. It's +~ only when the formatting reaches the address the entry is attached to that +~ the entry is executed, which has the effect of pushing a pointer to the +~ entry onto the subitem entry stack. +~ +~ It's been said already, but just to bring it all full circle, the life +~ cycle of a push-* entry ends later in the formatting phase, when a comment +~ entry pops it off the subitem entry stack and pretty-prints the value it +~ holds. +~ +~ With all this context, it likely doesn't need much explanation, but the +~ drop-subitem command will create an -entry-type-drop-subitem entry. When +~ this entry is executed, it will discard an item from the subitem entry +~ stack. Similarly, swap-subitems creates an -entry-type-swap-subitems, which +~ swaps the top two items, and roll-subitems creates an +~ -entry-type-roll-subitems, which rolls them, or unrolls them with a negative +~ parameter. + +~ In general, the order in which provide-* commands execute will most likely +~ match the order in which the corresponding bytes are output. This may not +~ always match the order in which the comment template wants to pretty-print +~ them. Therefore, you will want to use these commands to get the subitem +~ stack in the proper order. +~ +~ In using these commands that manipulate the subitem stack, it's important +~ to remember that, when formatting happens, the left-to-right output of +~ subitems in the final output comment is top-to-bottom on the subitem stack, +~ because they're popped in order. This is the opposite of the usual visual +~ representation of stacks, and can easily be confusing. +~ +~ It's probably clear from the extreme emphasis this documentation has taken +~ on describing the order things happen in, but it is quite common to define +~ a suffix comment at the start of an assembly-instruction word; adjust its +~ length as other words are called; accumulate push-* entries; then rely +~ on the address sorting to make sure the push-* entries are executed +~ before the suffix comment entry, even though the suffix-comment command +~ executed before the others, and even though the adjust-length commands may +~ have come before or after the provide-* commands that go with the same +~ suffix comment. +~ +~ Now that you have this full picture in your head, you finally also have +~ the background to understand the data-adjust-* commands. You may find it +~ helpful to go re-read the explanation of them, above, to let it sink in. +~ +~ This is the kind of forwards-and-backwards thinking which humans are +~ surprisingly good at when reading and writing software, but which it's +~ quite difficult to formalize in a way that lets a computer share that +~ understanding. If you've read this far, Irenes would absolutely adore +~ hearing about how you found the explanation and whether you did anything +~ with it; don't be shy! +~ +~ [1] https://katalepsis.net/ + +~ We have a bunch of accessors and traversers for the output metadata, which +~ are all functions from pointers to pointers. Don't confuse these with the +~ accessors for the transformation state, defined far, far above. +: hex-output-metadata-label-loop-buffer-start ; +: hex-output-metadata-label-loop-buffer-length 8 + ; +: hex-output-metadata-latest-output-point 2 8 * + ; +: hex-output-metadata-has-non-space-this-input-line 3 8 * + ; +: hex-output-metadata-suppression-count 4 8 * + ; +: hex-output-metadata-is-fresh-line 5 8 * + ; +: hex-output-metadata-indentation-depth 6 8 * + ; +: hex-output-metadata-current-column 7 8 * + ; +: hex-output-metadata-n-bytes-this-line 8 8 * + ; +: hex-output-metadata-subitem-entry-stack-depth 9 8 * + ; +: hex-output-metadata-subitem-entry-stack-zero 10 8 * + ; +: hex-output-metadata-subitem-entry-stack-capacity 8 ; +: hex-output-metadata-first-entry + 10 hex-output-metadata-subitem-entry-stack-capacity + 8 * + ; +: hex-output-metadata-entry-data-start ; +: hex-output-metadata-entry-data-length 8 + ; +: hex-output-metadata-entry-type 2 8 * + ; +: hex-output-metadata-entry-content 3 8 * + ; +: hex-output-metadata-next-entry 4 8 * + ; +: hex-output-metadata-previous-entry 4 8 * - ; +: hex-output-metadata-entry-type-line-comment 0 ; +: hex-output-metadata-entry-type-suffix-comment 1 ; +: hex-output-metadata-entry-type-fresh-line 2 ; +: hex-output-metadata-entry-type-blank-line 3 ; +: hex-output-metadata-entry-type-indent 4 ; +: hex-output-metadata-entry-type-string-literal 5 ; +: hex-output-metadata-entry-type-raw-string-literal 6 ; +: hex-output-metadata-entry-type-push-subitem-decimal 7 ; +: hex-output-metadata-entry-type-push-subitem-hex 8 ; +: hex-output-metadata-entry-type-push-subitem-hex8 9 ; +: hex-output-metadata-entry-type-push-subitem-hex16 10 ; +: hex-output-metadata-entry-type-push-subitem-hex32 11 ; +: hex-output-metadata-entry-type-push-subitem-hex64 12 ; +: hex-output-metadata-entry-type-push-subitem-string 13 ; +: hex-output-metadata-entry-type-saved-data 14 ; +: hex-output-metadata-entry-type-drop-subitem 15 ; +: hex-output-metadata-entry-type-swap-subitems 16 ; +: hex-output-metadata-entry-type-roll-subitems 17 ; + +~ Initialize the contents of the output metadata to all zeroes. This is +~ called from hex-transform at its top level, at the very start, to make sure +~ everything is structurally valid. We can't store real values at that point, +~ because they don't exist yet. The actual label-loop buffer, per its name, +~ won't be allocated until label-loop is called, so we can't possibly know its +~ address till then. +: zero-hex-output-metadata + transformation-state transformation-state-output-metadata @ + dup hex-output-metadata-label-loop-buffer-start 0 swap ! + dup hex-output-metadata-label-loop-buffer-length 0 swap ! + dup hex-output-metadata-latest-output-point 0 swap ! + dup hex-output-metadata-has-non-space-this-input-line 0 swap ! + dup hex-output-metadata-suppression-count 0 swap ! + dup hex-output-metadata-is-fresh-line 1 swap ! + dup hex-output-metadata-indentation-depth 0 swap ! + dup hex-output-metadata-current-column 0 swap ! + dup hex-output-metadata-n-bytes-this-line 0 swap ! + dup hex-output-metadata-subitem-entry-stack-depth 0 swap ! + dup hex-output-metadata-first-entry 0 swap ! ; + +~ (pointer -- boolean) +: is-in-label-loop-buffer + dup + transformation-state transformation-state-output-metadata @ + hex-output-metadata-label-loop-buffer-start @ + <= + swap + transformation-state transformation-state-output-metadata @ + dup hex-output-metadata-label-loop-buffer-start @ + swap hex-output-metadata-label-loop-buffer-length @ + + > && ; + +: has-non-space-this-input-line@ + transformation-state transformation-state-output-metadata @ + hex-output-metadata-has-non-space-this-input-line @ ; + +: has-non-space-this-input-line! + transformation-state transformation-state-output-metadata @ + hex-output-metadata-has-non-space-this-input-line ! ; + +: is-comment-entry + hex-output-metadata-entry-type @ + dup hex-output-metadata-entry-type-line-comment = swap + hex-output-metadata-entry-type-suffix-comment = || ; + +: is-output-producing-entry + dup is-comment-entry swap + hex-output-metadata-entry-type @ + dup hex-output-metadata-entry-type-fresh-line = 3roll || swap + hex-output-metadata-entry-type-blank-line = || ; + +: is-saved-data-entry + hex-output-metadata-entry-type @ + hex-output-metadata-entry-type-saved-data = ; + +: is-push-subitem-entry + ~ Notably, the subitem swap, drop, and roll entry types are not + ~ push-subitem entries. + hex-output-metadata-entry-type @ + dup hex-output-metadata-entry-type-push-subitem-decimal = swap + dup hex-output-metadata-entry-type-push-subitem-hex = 3roll || swap + dup hex-output-metadata-entry-type-push-subitem-hex8 = 3roll || swap + dup hex-output-metadata-entry-type-push-subitem-hex16 = 3roll || swap + dup hex-output-metadata-entry-type-push-subitem-hex32 = 3roll || swap + dup hex-output-metadata-entry-type-push-subitem-hex64 = 3roll || swap + hex-output-metadata-entry-type-push-subitem-string = || ; + +: is-subitem-stack-manipulation-entry + hex-output-metadata-entry-type @ + dup hex-output-metadata-entry-type-drop-subitem = swap + dup hex-output-metadata-entry-type-swap-subitems = 3roll || swap + hex-output-metadata-entry-type-roll-subitems = || ; + +: is-fresh-line@ + transformation-state transformation-state-output-metadata @ + hex-output-metadata-is-fresh-line @ ; + +: is-fresh-line! + transformation-state transformation-state-output-metadata @ + hex-output-metadata-is-fresh-line ! ; + +: fresh-line is-fresh-line@ not { newline } if ; + +: indentation-depth@ + transformation-state transformation-state-output-metadata @ + hex-output-metadata-indentation-depth @ ; + +: indentation-depth! + transformation-state transformation-state-output-metadata @ + hex-output-metadata-indentation-depth ! ; + +: current-column@ + transformation-state transformation-state-output-metadata @ + hex-output-metadata-current-column @ ; + +: current-column! + transformation-state transformation-state-output-metadata @ + hex-output-metadata-current-column ! ; + +: advance-current-column current-column@ + current-column! ; + +: n-bytes-this-line@ + transformation-state transformation-state-output-metadata @ + hex-output-metadata-n-bytes-this-line @ ; + +: n-bytes-this-line! + transformation-state transformation-state-output-metadata @ + hex-output-metadata-n-bytes-this-line ! ; + +: increment-bytes-this-line n-bytes-this-line@ 1+ n-bytes-this-line! ; + +: describe-subitem-entry-stack + transformation-state transformation-state-output-metadata @ + ." depth " + dup hex-output-metadata-subitem-entry-stack-depth @ .hex64 newline + ." capacity " + hex-output-metadata-subitem-entry-stack-capacity .hex64 newline + hex-output-metadata-subitem-entry-stack-zero + hex-output-metadata-subitem-entry-stack-capacity 8 * hexdump-from ; + +: subitem-entry-stack-depth@ + transformation-state transformation-state-output-metadata @ + hex-output-metadata-subitem-entry-stack-depth @ ; + +: subitem-entry-stack-depth! + transformation-state transformation-state-output-metadata @ + hex-output-metadata-subitem-entry-stack-depth ! ; + +: is-subitem-entry-stack-empty + subitem-entry-stack-depth@ 0 = ; + +: is-subitem-entry-stack-full + subitem-entry-stack-depth@ + hex-output-metadata-subitem-entry-stack-capacity = ; + +: push-subitem-entry-stack + is-subitem-entry-stack-full not { + transformation-state transformation-state-output-metadata @ + hex-output-metadata-subitem-entry-stack-zero + subitem-entry-stack-depth@ 8 * + ! + + subitem-entry-stack-depth@ 1+ subitem-entry-stack-depth! + } { + ~ We still consume the value, so that our caller doesn't crash. + drop + + ." Subitem entry stack overflow." newline + } if-else ; + +: pop-subitem-entry-stack + is-subitem-entry-stack-empty not { + subitem-entry-stack-depth@ 1- subitem-entry-stack-depth! + + transformation-state transformation-state-output-metadata @ + hex-output-metadata-subitem-entry-stack-zero + subitem-entry-stack-depth@ 8 * + @ + } { + ." Subitem entry stack underflow." newline + + ~ We still return a value, so that our caller doesn't crash. + 0 + } if-else ; + +~ (data start, data length, entry type, string pointer, entry pointer --) +: hex-output-metadata-entry! + dup hex-output-metadata-entry-content 3roll swap ! + dup hex-output-metadata-entry-type 3roll swap ! + dup hex-output-metadata-entry-data-length 3roll swap ! + dup hex-output-metadata-entry-data-start 3roll swap ! + drop ; + +~ (entry pointer, data start, data length, entry type, string pointer +~ -- next entry pointer) +: pack-hex-output-metadata-entry + 4 pick hex-output-metadata-entry! + hex-output-metadata-next-entry ; + +~ (data start, data length, entry type, string pointer --) +: add-hex-output-metadata-entry + ~ If the suppression count is greater than zero, decrement it. + transformation-state transformation-state-output-metadata @ + hex-output-metadata-suppression-count @ + dup 0 < { + 1- + transformation-state transformation-state-output-metadata @ + hex-output-metadata-suppression-count ! + 4 ndrop exit + } { drop } if-else + + 3 pick is-in-label-loop-buffer { + transformation-state transformation-state-output-metadata @ + hex-output-metadata-first-entry + { dup @ dup { dup 3 pick >= drop } if } { + hex-output-metadata-next-entry + } while + + 5 unroll pack-hex-output-metadata-entry + 0 pack64 drop + } { drop drop drop drop } if-else ; + +~ (entry pointer --) +: delete-metadata-entry + transformation-state transformation-state-output-metadata @ + hex-output-metadata-first-entry + { dup @ } { hex-output-metadata-next-entry } while + ~ (entry pointer, array-end marker pointer) + over hex-output-metadata-next-entry - 8 + + ~ (entry pointer, length of block to move) + swap dup hex-output-metadata-next-entry swap 3roll memmove ; + + +~ (-- entry pointer or 0) +: find-latest-metadata-entry + ~ Loop forward from the beginning of the entry array to find the last + ~ actual entry. + transformation-state transformation-state-output-metadata @ + hex-output-metadata-first-entry + dup @ { + { dup hex-output-metadata-next-entry @ } + { hex-output-metadata-next-entry } while + } if ; + +~ (-- entry pointer or 0) +: find-latest-comment-metadata-entry + find-latest-metadata-entry + ~ (entry pointer or 0) + + ~ If and only if we actually found one, now loop backwards to find the + ~ most recent one that's a comment entry. All length adjustments apply to + ~ comment entries, anything else needs to be skipped for this purpose. + { dup { dup is-comment-entry not } { 0 } if-else } + { transformation-state transformation-state-output-metadata @ + hex-output-metadata-first-entry + over > { + hex-output-metadata-previous-entry } { drop 0 } if-else } while + ~ (entry pointer or 0) + ; + +~ (-- entry pointer or 0) +: find-latest-output-producing-metadata-entry + find-latest-metadata-entry + ~ (entry pointer or 0) + + ~ If and only if we actually found one, now loop backwards to find the + ~ most recent one that's an output-producing entry. + { dup { dup is-output-producing-entry not } { 0 } if-else } + { transformation-state transformation-state-output-metadata @ + hex-output-metadata-first-entry + over > { + hex-output-metadata-previous-entry } { drop 0 } if-else } while + ~ (entry pointer or 0) + ; + +~ (-- entry pointer or 0) +: find-latest-saved-data-metadata-entry + find-latest-metadata-entry + ~ (entry pointer or 0) + + ~ If and only if we actually found one, now loop backwards to find the + ~ most recent one that's a comment entry. All length adjustments apply to + ~ comment entries, anything else needs to be skipped for this purpose. + { dup { dup is-saved-data-entry not } { 0 } if-else } + { transformation-state transformation-state-output-metadata @ + hex-output-metadata-first-entry + over > { + hex-output-metadata-previous-entry } { drop 0 } if-else } while + ~ (entry pointer or 0) + ; + +~ (target start location -- entry pointer or 0) +: find-first-comment-metadata-entry-with-start + ~ Find the first entry, but if it doesn't exist, we want 0 instead. + transformation-state transformation-state-output-metadata @ + hex-output-metadata-first-entry + ~ (target start location, entry pointer) + + ~ Now loop forwards to find the first that's a comment entry and has the + ~ desired start location. + { dup @ } { + dup is-comment-entry + over hex-output-metadata-entry-data-start @ 3 pick = + && { swap drop exit } if + hex-output-metadata-next-entry + } while drop drop 0 ; + + +~ (target start location -- entry pointer or 0) +: find-latest-comment-metadata-entry-with-start + find-latest-metadata-entry + ~ (target start location, entry pointer or 0) + + dup { + { dup @ } { + dup is-comment-entry + over hex-output-metadata-entry-data-start @ 3 pick = + && { swap drop exit } if + + dup + transformation-state transformation-state-output-metadata @ + hex-output-metadata-first-entry + = { drop drop 0 exit } if + + hex-output-metadata-previous-entry + } while + } if swap drop ; + + +~ (length adjustment --) +: adjust-latest-comment-metadata-entry-length + find-latest-comment-metadata-entry + + ~ We've found the entry we want to act on, if it exists, so do it. + dup { + hex-output-metadata-entry-data-length dup @ 3roll + swap ! + } { + drop + + ~ If we get here, that's a problem. Emit an error message to make sure + ~ it's easy to diagnose. + ." No pre-existing metadata entry to adjust." newline + crash + } if-else ; + +~ (length adjustment --) +: adjust-latest-output-producing-metadata-entry-start + find-latest-output-producing-metadata-entry + + ~ We've found the entry we want to act on, if it exists, so do it. + dup { + hex-output-metadata-entry-data-start dup @ 3roll + swap ! + } { + drop + + ~ If we get here, that's a problem. Emit an error message to make sure + ~ it's easy to diagnose. + ." No pre-existing metadata entry to adjust." newline + crash + } if-else ; + + +~ (target start location --) +: delete-first-comment-metadata-entry-with-start + find-first-comment-metadata-entry-with-start + dup { delete-metadata-entry } { drop } if-else ; + +~ (target start location --) +: delete-latest-comment-metadata-entry-with-start + find-latest-comment-metadata-entry-with-start + dup { delete-metadata-entry } { drop } if-else ; + +: delete-first-comment-metadata-entry-at-output-point + transformation-state transformation-state-output-metadata @ + hex-output-metadata-latest-output-point @ + delete-first-comment-metadata-entry-with-start ; + +: delete-latest-comment-metadata-entry-at-output-point + transformation-state transformation-state-output-metadata @ + hex-output-metadata-latest-output-point @ + delete-latest-comment-metadata-entry-with-start ; + + +~ (value --) +: set-latest-output-point + transformation-state transformation-state-output-metadata @ + hex-output-metadata-latest-output-point ! ; + +~ (adjustment --) +: adjust-latest-output-point + transformation-state transformation-state-output-metadata @ + hex-output-metadata-latest-output-point + dup 3unroll @ + swap ! ; + +~ (-- data value) +: data-consume + find-latest-saved-data-metadata-entry + ~ (entry pointer or 0) + + dup { + dup hex-output-metadata-entry-content @ swap + ~ (data value, entry pointer) + + delete-metadata-entry + } { + drop + + ~ If we get here, that's a problem. Emit an error message to make sure + ~ it's easy to diagnose. + ." No saved-data metadata entry to take a value from." newline + + ~ We still return a value, so that our caller doesn't crash. + 0 + } if-else ; + +~ (adjustment scale factor --) +: data-adjust-latest-comment-metadata-entry-length + ~ All the entry array manipulation happens in data-consume. Once we're + ~ done doing that, the array is now in a consistent state, so we can hand + ~ off to the non-data implementation to do the rest of the work. + data-consume * adjust-latest-comment-metadata-entry-length ; +~ (adjustment scale factor --) +: data-adjust-latest-output-producing-metadata-entry-start + ~ Same as for adjusting the entry length. + data-consume * adjust-latest-output-producing-metadata-entry-start ; + +~ (value offset --) +: data-set-latest-output-point + data-consume + set-latest-output-point ; +~ (adjustment scale factor --) +: data-adjust-latest-output-point + data-consume * adjust-latest-output-point ; + + +~ (count adjustment --) +: adjust-hex-output-suppression-count + transformation-state transformation-state-output-metadata @ + hex-output-metadata-suppression-count dup @ 3roll + swap ! ; + +~ This is the followup to zero-hex-output-metadata, above. Once we do have +~ the label-loop buffer, we need to create some default entries. This is +~ called in the first label-loop iteration by hex-allocate-replacement, and +~ it's called before subsequent iterations by the hook on reset-labels defined +~ in hex-colon-alternate. +~ +~ This overwrites whatever entries currently exist with the new ones, and +~ adds a zero delimiter at the end. +~ +~ In addition to setting up the entries, we also set latest-output-point to +~ the start of the label-loop buffer here. This needs to happen at the start +~ of each pass, so although it doesn't quite fit the name, we do it here. +~ +~ It requires the header to have already been initialized; in particular it +~ requires the label-loop buffer start pointer to be valid. +: create-hex-output-metadata-top-of-file-entries + transformation-state transformation-state-output-metadata @ + dup hex-output-metadata-label-loop-buffer-start @ + swap hex-output-metadata-first-entry + ~ (label buffer start, metadata output point) + + ~ This likely won't affect anything, because every call to pack* + ~ overwrites it, but we initialize the latest output point to the start of + ~ the label loop buffer here. That way, in the event the very first thing in + ~ the output is something that's supposed to have a magic comment on it, + ~ hex-tilde-replacement will know where to attach the metadata entry. + ~ + ~ If that was word soup, go read about hex-tilde-replacement and + ~ hex-tilde-alternate and come back. :) + over + transformation-state transformation-state-output-metadata @ + hex-output-metadata-latest-output-point ! + 0 has-non-space-this-input-line! + + over 0 hex-output-metadata-entry-type-line-comment + s" This file is a commented hexadecimal representation of a compiled" + pack-hex-output-metadata-entry + + over 0 hex-output-metadata-entry-type-line-comment + s" program, the output of Evocation's hex transform. The comments are" + pack-hex-output-metadata-entry + + over 0 hex-output-metadata-entry-type-line-comment + s" intended to allow a human reader to audit it. You can convert it to a" + pack-hex-output-metadata-entry + + over 0 hex-output-metadata-entry-type-line-comment + s" runnable executable by piping it through the program named 'hex' and" + pack-hex-output-metadata-entry + + over 0 hex-output-metadata-entry-type-line-comment + s" chmodding the output 755. The difference between a source and a binary" + pack-hex-output-metadata-entry + + over 0 hex-output-metadata-entry-type-line-comment + s" is comments! Enjoy. :)" + pack-hex-output-metadata-entry + + over 0 hex-output-metadata-entry-type-blank-line 0 + pack-hex-output-metadata-entry + + 0 swap ! ~ End-of-entries delimiter + drop ; + +: hex-self-codeword-alternate self-codeword ; + +: hex-string-alternate + ~ The technique we use to wrap s" is analogous to how we do it in the + ~ label transform, but with one critical decision different. + ~ + ~ In the label transform, the output buffer is going to be written + ~ directly into the final executable at some point, and we'd really rather + ~ not scribble temporary values into it, even if we expect to overwrite them + ~ later. Additionally, everything the label transform does runs inside the + ~ label loop, so the code it's transforming isn't expected to do any sort of + ~ miscellaneous setup things, just define a bunch of words. In that + ~ situation, making sure to use temporary space from the outer, "real" log + ~ is helpful. + ~ + ~ In the hex transform, most of that is different. The output buffer will + ~ include all sorts of things that don't become part of the final + ~ executable, though they're used in producing it. So, it's fine to use it + ~ for scratch space. Additionally, using the outer log for string literals + ~ gets us into trouble, since some of these temporary strings need to be + ~ made more durable by calling allocate-string on them. If we do that with + ~ a string pointer backed by the outer log, the inner "here" will wind up + ~ pointing to the end of the outer log, and all sorts of things will start + ~ scribbling atop each other. + ~ + ~ We do still benefit from using the interpreted version of s" rather than + ~ the compiled one; as with the label transform, it's convenient to handle + ~ the litstring dance ourselves. + ~ + ~ If we're in compile mode, we have reinvented a problem which the + ~ regular, non-transformed s" already deals with: The temporary space we're + ~ writing to is in the same place as the code we're compiling. We solve it + ~ the same way: Check beforehand and write out the call to litstring if we + ~ need it, so that we won't have to slide things around later. + interpreter-flags @ 0x01 & { + ~ We look up the inner version of litstring to reference here. This is + ~ similar to what the label transform does, except we don't use a label + ~ for it. + s" litstring" find entry-to-execution-token , + } if + + ~ Now, regardless of what mode we're actually in, call the interpreted + ~ version of s" to read the actual string and its null terminator. + interpreter-flags @ + ' s" entry-to-execution-token + [ ' [ entry-to-execution-token , ] + execute + swap interpreter-flags ! + + ~ Now we have a string pointer on the stack at transform time. If we're in + ~ immediate mode, that's sufficient. If we're in compile mode, we also + ~ already output a litstring invocation, and the string is in the right + ~ place, but "here" is pointing before it and needs to point after it, then + ~ we also still need to add alignment padding. Plus, we need to be sure to + ~ not leave the string pointer itself on the stack. + interpreter-flags @ 0x01 & { + dup stringlen 1+ + 8 packalign here ! + } if + ; make-immediate + +: hex-dot-string-alternate + ' hex-string-alternate entry-to-execution-token execute + + interpreter-flags @ 0x01 & { + ~ We look up the inner version of emitstring, too. + s" emitstring" find entry-to-execution-token , + } { emitstring } if-else + ; make-immediate + +: hex-create-alternate create ; + +~ This is a helper called from the patched version of "variable", described +~ in more detail in hex-colon-alternate, below. It expects to be called after +~ outputting the entry header for "variable", during the body of the +~ definition, so that it can output compiled code which will run as part of +~ the transformed "variable". +~ +~ The helper accepts a string pointer giving a variable name. The code it +~ produces checks the name of the variable being defined and, if the two names +~ match, alters the resulting inner variable to point to the same backing +~ store as the outer variable of the same name. +~ +~ In many ways the hex transform is the trippiest one. To wit, there's two +~ layers of compilation happening here... so don't get confused. When the +~ helper is called, we're compiling the inner "variable", but "variable" +~ itself is a word-defining word which also has the task of compilation... +~ which we're modifying. +~ +~ (name pointer --) +: hex-variable-use-outer + ~ The actual payload here is that we check whether we're defining the + ~ word described by the name pointer we were given, and, if so, we make it + ~ reference the outer backing store instead of the inner one. + ~ + ~ We want to do this all as references to inner words, which fortunately + ~ have been defined by now, but we have to do that a bit indirectly... + s" dup" find entry-to-execution-token , + s" litstring" find entry-to-execution-token , + here @ over packstring 8 packalign here ! + s" stringcmp" find entry-to-execution-token , + s" lit" find entry-to-execution-token , 0 , + s" =" find entry-to-execution-token , + + ~ Also, we don't have high-level flow control yet, and even if we did, + ~ it would be awkward to use it here. So we count the branch by hand. Ah + ~ well. As always, remember that with forward branches, the offset to + ~ branch by is the first word to be skipped, and is included in the count. + s" 0branch" find entry-to-execution-token , 6 8 * , + + ~ If control reaches here in the generated code, the string matched. + s" swap" find entry-to-execution-token , + s" drop" find entry-to-execution-token , + + ~ To get the value of the outer variable, we just call it. Of course, + ~ looking up an outer entry is a pain, but at least it's a pain in a way + ~ that should be familiar by now. + s" lit" find entry-to-execution-token , + swap-transform-variables + find + swap-transform-variables + entry-to-execution-token execute , + + s" swap" find entry-to-execution-token , ; + +~ This is a helper used by postprocess-metadata-entries, below. It +~ duplicates the entry it's given, creating a second identical copy +~ immediately after it in the entry array, sliding everything else forward to +~ make room. +~ +~ (entry pointer --) +: dup-metadata-entry + ~ Find all the remaining entries, and the final terminator, and slide + ~ them forward by the length of one entry. + dup dup + ~ (entry pointer, entry pointer, entry pointer) + + { dup @ } { hex-output-metadata-next-entry } while 8 + + ~ (entry pointer, entry pointer, entry array end pointer) + swap - + ~ (entry pointer, entry array tail length) + swap dup hex-output-metadata-next-entry 3roll + ~ (entry pointer, adjusted entry pointer, entry array tail length) + memmove ; + +~ This is a helper used by postprocess-metadata-entries, below. It swaps +~ the contents of the entry it's given with the contents of the entry +~ immediately after it in the entry array. +~ +~ (entry pointer --) +: swap-metadata-entries + ~ Copy the current entry to a scratch area. + dup dup dup hex-output-metadata-next-entry swap - + ~ (entry pointer, entry pointer, entry length) + swap-transform-variables here @ swap-transform-variables swap + ~ (entry pointer, entry pointer, scratch pointer, entry length) + memcopy + ~ (entry pointer) + + ~ Copy the next entry over the current entry. + dup dup hex-output-metadata-next-entry dup + ~ (entry pointer, entry pointer, next entry pointer, next entry pointer) + 3roll - + ~ (entry pointer, next entry pointer, entry length) + 2 pick swap + ~ (entry pointer, next entry pointer, entry pointer, entry length) + memcopy + ~ (entry pointer) + + ~ Copy the scratch area over the next entry. + dup hex-output-metadata-next-entry dup + ~ (entry pointer, next entry pointer, next entry pointer) + 3roll - + ~ (next entry pointer, entry length) + swap-transform-variables here @ swap-transform-variables 3unroll + ~ (scratch pointer, next entry pointer, entry length) + memcopy ; + +~ Metadata entries are post-processed before they're output, in order to +~ make the structure of the output code simpler. The post-processing won't +~ happen until output actually occurs, which is important for operations that +~ modify entries in-place. +~ +~ Any time output is performed, all existing entries are post-processed. +~ Therefore, the post-processing is required to be idempotent, and is. +: postprocess-metadata-entries + 1 { } { + 0 + ~ (did anything this iteration) + + transformation-state transformation-state-output-metadata @ + hex-output-metadata-first-entry + { dup @ } { + ~ (did anything, entry pointer) + + ~ Is the entry a suffix comment with a non-zero data length? + dup hex-output-metadata-entry-type @ + hex-output-metadata-entry-type-suffix-comment = + over hex-output-metadata-entry-data-length @ 0 != && { + ~ Split it into two entries so as not to rely on data length for + ~ formatting. + dup dup-metadata-entry + ~ (did anything, entry pointer) + ~ Now we have the original entry at the original location, and a copy + ~ of it at the next location. + + ~ We have a feature which lets a suffix comment be created with a + ~ negative length, which means it covers bytes before the current output + ~ point rather than after it. This logic here is the implementation of + ~ that feature. + dup hex-output-metadata-entry-data-start @ dup + 2 pick hex-output-metadata-entry-data-length @ + + 2dup min 3unroll max + ~ (..., entry pointer, low end of data, high end of data) + + ~ Now we modify the original in-place to turn it into a fresh-line + ~ entry at the low end of the data... + 2 pick hex-output-metadata-entry-type + hex-output-metadata-entry-type-fresh-line swap ! + ~ (..., entry pointer, low end of data, high end of data) + + 2 pick hex-output-metadata-entry-data-start 3roll swap ! + ~ (..., entry pointer, high end of data) + over hex-output-metadata-entry-data-length 0 swap ! + ~ (..., entry pointer, high end of data) + + ~ ... then we modify the next entry in-place to set its location to + ~ the end of the data instead of the beginning, and its length to zero. + over hex-output-metadata-next-entry + ~ (..., entry pointer, high end of data, next entry pointer) + + dup hex-output-metadata-entry-data-start 3roll swap ! + ~ (..., entry pointer, next entry pointer) + hex-output-metadata-entry-data-length 0 swap ! + ~ (..., entry pointer) + + ~ This rule won't fire again on the fresh-line entry because it's of a + ~ a different type, and it won't fire on the modified suffix-comment + ~ entry because its data length is zero. + ~ + ~ It's possible that one or the other of the bubble-sort rules will + ~ fire on either of them. + + ~ (did anything, entry pointer) + swap drop 1 swap + } if + ~ (did anything, entry pointer) + + ~ Do this entry and the next entry both exist, and they're out of + ~ order based on their data starts? + dup hex-output-metadata-next-entry dup @ + { hex-output-metadata-entry-data-start @ + over hex-output-metadata-entry-data-start @ > } + { drop 0 } if-else + { + ~ Swap the order of the two entries. Since this whole thing is in a + ~ loop, this functions as a bubble sort. + dup swap-metadata-entries + swap drop 1 swap + + ~ This rule won't fire again on the same pair of entries because now + ~ they're in the correct order. It may fire again next iteration, + ~ but only until everything is sorted. + ~ + ~ This rule does nothing to alter whether the suffix-comment rule + ~ will fire on either of the entries it swapped, so there's no + ~ possibility of an infinite loop through that one. + } if + ~ (did anything, entry pointer) + + ~ Do this entry and the next entry both exist, and they have the same + ~ data start, and this one is a comment entry, and the next one is a + ~ stack-manipulation entry with zero length? + dup is-comment-entry over hex-output-metadata-next-entry @ && + { ~ (did anything, entry pointer) + dup hex-output-metadata-next-entry + over hex-output-metadata-entry-data-start @ + over hex-output-metadata-entry-data-start @ = + swap dup hex-output-metadata-entry-data-length @ 0 = 3roll && swap + is-subitem-stack-manipulation-entry && } + { 0 } if-else + { + ~ Swap the order of the two entries. Since this whole thing is in a + ~ loop, this functions as a bubble sort. + dup swap-metadata-entries + swap drop 1 swap + + ~ This rule can't fire in the same iteration as the previous swap, + ~ because that one doesn't fire when the starts are equal. It also + ~ won't fire again on the same pair of entries, again because now + ~ they're in the correct order. If it fires again in the next + ~ iteration that'll only be in the usual way it does for a bubble + ~ sort. + ~ + ~ As with the other swap rule, this rule also does nothing to alter + ~ whether the suffix-comment rule will fire on either entry it + ~ touches, so it won't loop through that one. + } if + ~ (did anything, entry pointer) + + hex-output-metadata-next-entry + } while drop + } while ; + +: hex-comment-first-tab-stop 36 ; +: hex-comment-second-tab-stop 58 ; + +~ This is a helper called from hex-emit-template-string. The word pointer +~ is in the outer log's scratch space, and must not be overwritten. +~ +~ (queued space count, word pointer -- should ignore space) +: hex-emit-template-string-word + dup s" #" stringcmp 0 = { + ~ (queued space count, word pointer) + drop + + pop-subitem-entry-stack + ~ (queued space count, subitem entry) + + ~ If we underflowed the stack, we got zero back, so make sure not to + ~ crash. A crash here would be a pain to debug because it would be + ~ unclear how much of the overall system is working, and it's a + ~ complex system. + dup { + ~ (queued space count, subitem entry) + + dup hex-output-metadata-entry-type @ + hex-output-metadata-entry-type-push-subitem-decimal = { + swap dup indent advance-current-column 0 swap + + dup hex-output-metadata-entry-content @ + dup 10 swap logfloor 1+ advance-current-column + . + } if + + dup hex-output-metadata-entry-type @ + hex-output-metadata-entry-type-push-subitem-hex = { + swap dup indent advance-current-column 0 swap + + ." 0x" + dup hex-output-metadata-entry-content @ + dup 16 swap logfloor 1+ 2 + advance-current-column + .hex + } if + + dup hex-output-metadata-entry-type @ + hex-output-metadata-entry-type-push-subitem-hex8 = { + swap dup indent advance-current-column 0 swap + + ." 0x" + dup hex-output-metadata-entry-content @ .hex8 + 4 advance-current-column + } if + + dup hex-output-metadata-entry-type @ + hex-output-metadata-entry-type-push-subitem-hex16 = { + swap dup indent advance-current-column 0 swap + + ." 0x" + dup hex-output-metadata-entry-content @ .hex16 + 6 advance-current-column + } if + + dup hex-output-metadata-entry-type @ + hex-output-metadata-entry-type-push-subitem-hex32 = { + swap dup indent advance-current-column 0 swap + + ." 0x" + dup hex-output-metadata-entry-content @ .hex32 + 10 advance-current-column + } if + + dup hex-output-metadata-entry-type @ + hex-output-metadata-entry-type-push-subitem-hex64 = { + swap dup indent advance-current-column 0 swap + + ." 0x" + dup hex-output-metadata-entry-content @ .hex64 + 18 advance-current-column + } if + + dup hex-output-metadata-entry-type @ + hex-output-metadata-entry-type-push-subitem-string = { + swap dup indent advance-current-column 0 swap + + dup hex-output-metadata-entry-content @ + dup stringlen advance-current-column + emitstring + } if + } if + + drop drop 0 exit + } if + + dup s" #--" stringcmp 0 = { + ~ (queued space count, word pointer) + + hex-comment-second-tab-stop current-column@ + 2dup > { + drop drop swap drop 0 swap + newline + hex-comment-first-tab-stop indent ." ~" + hex-comment-second-tab-stop hex-comment-first-tab-stop 1+ - indent + hex-comment-second-tab-stop current-column! + } { + - dup indent advance-current-column + swap drop 0 swap + } if-else + + drop drop 1 exit + } if + + ~ (queued space count, word pointer) + swap dup indent advance-current-column + dup stringlen advance-current-column + emitstring 0 ; + + +~ (string pointer --) +: hex-emit-template-string + 0 swap + ~ (queued space count, input point) + + { unpack8 dup } { + ~ (queued space count, input point, latest character) + dup is-space { + ~ Instead of outputting space characters as we reach them, we keep a + ~ count of how many are pending. This allows us to do word-wrapping + ~ without having to worry about trailing spaces. + 3roll 1+ 3unroll drop + } { + ~ (queued space count, input point, first word character) + ~ We use the outer log's scratch space to accumulate each word in. + swap-transform-variables here @ swap-transform-variables + swap + pack8 + swap + ~ (queued space count, scratch output point, input point) + + ~ Scan forward and find a space-separated word. + { unpack8 dup dup is-space not && } { + ~ (queued space count, scratch output point, adjusted input point, + ~ next character) + 3roll swap pack8 swap + ~ (queued space count, adjusted scratch output point, + ~ adjusted input point) + } while + ~ (queued space count, scratch output point, input point, + ~ first non-word character or 0) + 3roll 0 pack8 drop + ~ (queued space count, input point, first non-word character or 0) + + swap-transform-variables here @ swap-transform-variables + ~ (queued space count, input point, first non-word character or 0, + ~ word pointer) + + 4 roll swap 1 max hex-emit-template-string-word 0 4 unroll + ~ (queued space count, input point, first non-word character, + ~ should ignore space) + + over { + ~ If the character is non-zero, it's a space, so it can't be part of + ~ a word, so we can queue it for later output... unless we were asked + ~ to skip it, of course. + not { + ~ (queued space count, input point, first non-word character) + 3roll 1+ 3unroll + } if + drop + } { + ~ If this is the end of the string, we can discard the 0, but then + ~ we need to nudge the input point back so the loop will terminate. + drop drop 1- + } if-else + ~ (queued space count, input point) + } if-else + } while drop drop drop ; + + +~ (string pointer --) +: hex-emit-comment + hex-comment-second-tab-stop current-column@ > { + newline + hex-comment-first-tab-stop dup indent current-column! + 1 is-fresh-line! + 0 n-bytes-this-line! + } if + + ." ~" + 1 advance-current-column + + dup @ { + space + 1 advance-current-column + } if + + hex-emit-template-string ; + + +~ This "replacement" is a little different from an alternate: When the code +~ under transformation attempts to compile its own version of sys-write, it +~ gets a stub that calls this word instead. It's swapped out by +~ hex-colon-alternate, whereas the regular alternates are swapped out by +~ hex-transform-one. +~ +~ In the programs we're actually running the transform on, there's only ever +~ a single call to sys-write which is the whole binary in one go, but we don't +~ want to require that, so we keep all the formatting state in the output +~ metadata header. +~ +~ (file descriptor, base address, length to write -- length written) +: hex-sys-write-replacement + ~ In intended operation, the code under transformation is a compilation + ~ process, and it won't output anything that isn't executable binary. + ~ However, while developing, it's common to have crashes, and it's nice to + ~ get the crash dump as ASCII rather than hex. + over is-in-label-loop-buffer not { + sys-write exit + } if + 3roll drop dup 3roll + ~ The result should be the length written. We just proactively declare + ~ that we'll write the full length we were asked to, threading that through + ~ as the result we'll eventually return. + ~ (result, length to write, base address) + + postprocess-metadata-entries + + ~ Importantly, postprocessing leaves the entries sorted, which means we + ~ can go back and forth in a linear scan through the entry array while we + ~ also do a linear scan of the output data. It's fiddly, but we can't afford + ~ to be O(n^2) here, it adds up to quite a lot of runtime. + + transformation-state transformation-state-output-metadata @ + hex-output-metadata-first-entry + ~ (result, length to write, base address, initial metadata scan pointer) + { 2 pick 0 <= } { + ~ (result, length remaining, current output address, + ~ metadata scan pointer) + + ~ Keep inspecting entries as long as the output address is greater than + ~ or equal to the entry address. If we're looking at the entries that come + ~ before the output, this will scan through all of them first, then + ~ eventually catch up. Since the scan pointer is kept across iterations, + ~ it's important we not go beyond that point. + { dup @ dup + ~ (result, length remaining, current output address, + ~ metadata scan pointer, entry's first field, entry's first field) + { + ~ This is an nop, but for clarity's sake it's best to not depend on + ~ details of the structure layout that perhaps not everyone has + ~ memorized... + ~ + ~ To get into this block, we tested for the array end indicator, but + ~ now that we know we have an entry here, we're looking at its + ~ contents. + hex-output-metadata-entry-data-start + + dup 3 pick + ~ (result, length remaining, current output address, + ~ metadata scan pointer, entry's data start, entry's data start, + ~ current output address) + >= swap drop + } if + } { + dup @ + 2 pick = { + ~ We found a matching metadata entry. + ~ (result, length remaining, current output address, + ~ metadata scan pointer) + + dup hex-output-metadata-entry-type @ + hex-output-metadata-entry-type-line-comment = { + fresh-line + indentation-depth@ dup indent advance-current-column + + dup hex-output-metadata-entry-content @ hex-emit-comment + newline + + 1 is-fresh-line! + 0 n-bytes-this-line! + 0 current-column! + } if + + dup hex-output-metadata-entry-type @ + hex-output-metadata-entry-type-suffix-comment = { + hex-comment-first-tab-stop current-column@ - 1 max + dup indent advance-current-column + + dup hex-output-metadata-entry-content @ hex-emit-comment + newline + + 1 is-fresh-line! + 0 n-bytes-this-line! + 0 current-column! + } if + + dup hex-output-metadata-entry-type @ + hex-output-metadata-entry-type-fresh-line = { + fresh-line + 1 is-fresh-line! + 0 n-bytes-this-line! + 0 current-column! + } if + + dup hex-output-metadata-entry-type @ + hex-output-metadata-entry-type-blank-line = { + fresh-line + newline + 1 is-fresh-line! + 0 n-bytes-this-line! + 0 current-column! + } if + + dup hex-output-metadata-entry-type @ + hex-output-metadata-entry-type-indent = { + dup hex-output-metadata-entry-content @ + indentation-depth@ + indentation-depth! + } if + + dup hex-output-metadata-entry-type @ + hex-output-metadata-entry-type-string-literal = { + fresh-line + indentation-depth@ dup indent advance-current-column + + ." ~ " + 0x22 value@ emitstring drop + dup hex-output-metadata-entry-content @ emitstring + 0x22 value@ emitstring drop + ." (string literal with null)" + + newline + 1 is-fresh-line! + 0 n-bytes-this-line! + 0 current-column! + } if + + dup hex-output-metadata-entry-type @ + hex-output-metadata-entry-type-raw-string-literal = { + fresh-line + indentation-depth@ dup indent advance-current-column + + ." ~ " + 0x22 value@ emitstring drop + dup hex-output-metadata-entry-content @ emitstring + 0x22 value@ emitstring drop + ." (raw string literal)" + + newline + 1 is-fresh-line! + 0 n-bytes-this-line! + 0 current-column! + } if + + dup is-push-subitem-entry { + dup push-subitem-entry-stack + } if + + dup hex-output-metadata-entry-type @ + hex-output-metadata-entry-type-saved-data = { + ~ It's not valid to execute these, so print a diagnostic. They're + ~ supposed to always be consumed by data-* commands. + ." A saved-data metadata entry leaked." newline + } if + + dup hex-output-metadata-entry-type @ + hex-output-metadata-entry-type-drop-subitem = { + pop-subitem-entry-stack drop + } if + + dup hex-output-metadata-entry-type @ + hex-output-metadata-entry-type-swap-subitems = { + ~ If we underflow the stack, we'll get zeroes back here. In order + ~ to avoid causing more confusing problems later, we make sure to + ~ discard the zeroes instead of pushing them. Just like in + ~ hex-emit-template-string, the reasoning is that it's easier to + ~ diagnose what's wrong with this subsystem when it runs to + ~ completion. + pop-subitem-entry-stack + pop-subitem-entry-stack + swap + dup { push-subitem-entry-stack } { drop } if-else + dup { push-subitem-entry-stack } { drop } if-else + } if + + dup hex-output-metadata-entry-type @ + hex-output-metadata-entry-type-roll-subitems = { + ~ The parameter giving the amount to roll by is in the content + ~ field. + dup hex-output-metadata-entry-content @ + ~ (... amount to roll by) + + ~ We resist the temptation to do the memcopy math. We don't need + ~ to, since the regular Forth roll and unroll do it for us. + dup 0 < { + ~ In this scenario, we're rolling. The amount to roll by is also + ~ the number of items. + + 0 { 2dup < } { pop-subitem-entry-stack 3unroll 1+ } while + drop + ~ (... items, amount to roll by / number of items) + + ~ Please notice that the items we have popped from the subitem + ~ entry stack are now, on the value stack, in the reverse of the + ~ order they were on the entry stack. They will reverse again when + ~ we push them back. + ~ + ~ The amount to roll by is positive, so the desired operation is + ~ a regular roll. Since the entries are reversed right now, we + ~ unroll them, which will be what we want when we put them back. + ~ + ~ Try it out interactively with a toy example and convince + ~ yourself. It's hard to really model in your head, but it's true. + ~ The intuition is that, if you're looking at the value stack, + ~ the bottommost item there will be the topmost one after + ~ re-reversing, so we think of our operations as proceeding + ~ up-stack from there, instead of the usual operations that start + ~ at the topmost item and proceed downwards. + ~ + ~ Anyway, this little trick does a regular unroll, which acts as + ~ a roll, and keeps the value around. + dup 3unroll 1+ unroll + } if + ~ (... items if positive, amount to roll by) + + ~ We did the positive case first, which means the amount to roll + ~ by is still its original value, regardless of whether that case + ~ ran or not. So, we can check if it's negative to see if we need + ~ to do the negative case. + dup 0 > { + ~ In this scenario, we're unrolling. We negate the amount to + ~ roll by to get the amount to unroll by, which is also the number + ~ of items. + negate + + 0 { 2dup < } { pop-subitem-entry-stack 3unroll 1+ } while + drop + ~ (... items, number of items) + + ~ The parameter was negative, so we were asked to do an unroll. + ~ This trick does a roll, and keeps the value. Notice that "roll" + ~ always needs a positive amount. + dup 1+ roll swap + } if + ~ (... items, number of items) + + ~ Now, regardless of which case ran, we have the items and we have + ~ a non-negative count of them. Notice that if we were asked to roll + ~ by zero, neither case ran, but that still describes what we have + ~ accurately. + + ~ Now we push them back. + 0 { 2dup < } { + 3roll + + ~ Again, if we underflowed we got zeros, and we discard them. + dup { push-subitem-entry-stack } { drop } if-else + + 1+ + } while + drop drop + } if + } if + hex-output-metadata-next-entry + } while + ~ (result, length remaining, current output address, + ~ metadata scan pointer) + + 2 pick { + n-bytes-this-line@ 16 = { + newline + 1 is-fresh-line! + 0 n-bytes-this-line! + 0 current-column! + } if + + is-fresh-line@ { + indentation-depth@ dup indent advance-current-column + } { + space + 1 advance-current-column + } if-else + + over 8@ .hex8 + + 2 advance-current-column + increment-bytes-this-line + 0 is-fresh-line! + } if + ~ (result, length remaining, current output address, + ~ metadata scan pointer) + + 3unroll 1+ swap 1- swap 3roll + } while drop ; + +~ This has complex logic, so we do it as a trace rather than with magic +~ comments. +~ +~ (new label value, label entry pointer --) +: hex-set-label-trace + ~ If the value is zero, don't bother making an entry. There's a bunch of + ~ unused labels set to zero and even if they were used, showing them at the + ~ start of the file would not provide useful information. + over 0 = { drop drop exit } if + + ~ Similarly, if the label has its non-offset bit set, we don't need an + ~ entry for that, either + ~ + ~ It's worth noting that this bit is set by the code being compiled, + ~ potentially after it has already called label-set, and therefore after + ~ this trace. This introduces an ordering dependency, though it's unlikely + ~ to come up in practice because the bit is kept across label-loop passes + ~ and as long as there are any forward references there will always be at + ~ least two passes. + dup is-label-non-offset { drop drop exit } if + + ~ Recall that label values are offsets within the output file. This is not + ~ otherwise a requirement, compiled code can use their values for anything + ~ it wants, but the label transform treats them as offsets and so does the + ~ ELF code. + ~ + ~ We want to create metadata entries associated with that offset; recall + ~ also that entries' data starts are host addresses which must be within + ~ the label-loop buffer. So... + over + transformation-state transformation-state-output-metadata @ + hex-output-metadata-label-loop-buffer-start @ + + 0 + ~ (new label value, label entry pointer, entry data start, + ~ entry data length) + + 2dup hex-output-metadata-entry-type-push-subitem-hex64 7 roll + add-hex-output-metadata-entry + ~ (label entry pointer, entry data start, entry data length) + + 2dup hex-output-metadata-entry-type-push-subitem-string + 6 roll entry-to-name + + swap-transform-variables here @ swap-transform-variables dup 3unroll + ~ (copy, original, copy) + over stringlen 1+ memcopy + swap-transform-variables allocate-string swap-transform-variables + add-hex-output-metadata-entry + + hex-output-metadata-entry-type-line-comment + s" label # at offset # is here." + add-hex-output-metadata-entry ; + +~ (data start, data length --) +: hex-pack-trace + ~ We keep a running tally of the next address in the output, for the + ~ benefit of hex-tilde-replacement, which needs to know where to attach its + ~ metadata entries. + + + dup is-in-label-loop-buffer { + transformation-state transformation-state-output-metadata @ + hex-output-metadata-latest-output-point ! + } { drop } if-else ; + +~ (output point, string pointer --) +: hex-packstring-trace + dup stringlen 1+ + ~ (output point, string pointer, string length) + 2 pick over hex-pack-trace + swap + ~ (output point, string length, string pointer) + drop over + hex-output-metadata-entry-type-string-literal swap + add-hex-output-metadata-entry ; + +~ The purpose of this word is to make sure that the memory-sliding +~ techniques used by the high-level flow control words are appropriately +~ reflected in the output metadata. This is accomplished by finding all +~ entries whose data start pointer or string content pointer is within the +~ slid region, and adjusting the relevant pointer appropriately. +~ +~ Notice that this strategy means there's no metadata entry describing the +~ old location of the slid memory, though sliding is non-destructive. The +~ assumption is that that location is going to be overwritten with new code, +~ which will have its own metadata entries created as the new code is written. +~ Otherwise, since we don't do anything to erase metadata entries pertaining +~ to overwritten locations, old and new metadata entries would overlap and +~ cause confusion. The assumption is correct for all existing uses of +~ memory-sliding as of the time this was written, but it's possible it will +~ need to be revisited someday. +~ +~ This word is called for both memcopy and memmove. It's called by +~ hex-memory-slide-replacement, which is itself installed by a variety of +~ means for different situations; see its documentation for more detail. +~ +~ (source, destination, length --) +: hex-memory-slide-trace + 2 pick is-in-label-loop-buffer + 2 pick is-in-label-loop-buffer && + { + ~ At the time this runs, the metadata entry array isn't sorted, so we + ~ check the whole thing. + transformation-state transformation-state-output-metadata @ + hex-output-metadata-first-entry + ~ (source, destination, length, initial metadata scan pointer) + { dup @ } { + ~ (source, destination, length, metadata scan pointer) + + dup hex-output-metadata-entry-data-start @ 4 pick <= + ~ (source, destination, length, metadata scan pointer, lower bound okay) + over hex-output-metadata-entry-data-start @ 5 pick 4 pick + > + && { + ~ (source, destination, length, metadata scan pointer) + dup hex-output-metadata-entry-data-start @ 4 pick - 3 pick + + over hex-output-metadata-entry-data-start ! + } if + + dup hex-output-metadata-entry-type @ + dup hex-output-metadata-entry-type-string-literal = + swap hex-output-metadata-entry-type-raw-string-literal = || { + dup hex-output-metadata-entry-content @ + ~ (source, destination, length, scan pointer, content pointer) + dup 5 pick <= + ~ (source, destination, length, scan pointer, content pointer, + ~ lower bound okay) + over 6 pick 5 pick + > && { + ~ (source, destination, length, scan pointer, content pointer) + 4 pick - 3 pick + + over hex-output-metadata-entry-content ! + } { drop } if-else + } if + + hex-output-metadata-next-entry + } while 4 ndrop + } { drop drop drop } if-else ; + + +~ By overriding colon, we can special-case the definitions of particular +~ words. It's very metacircular. +~ +~ This lets us prepend trap code to the original definition. For those not +~ familiar, a "trap" in systems programming is code that runs instead of some +~ other, pre-existing code, by violating the usual abstractions in some way. +~ We implement our traps by creating the word header, including the docol +~ pointer, then compiling the trap code before we return from the colon +~ alternate, so that it's already been output before the original word's body +~ starts to compile. +~ +~ Prepending our trap code to the original code like this can also be used +~ to get the effect of replacing the original code entirely by calling "exit" +~ at the end of our trap. The original code will still be compiled, after the +~ trap code, but doing this will make sure it doesn't run. Most of the time, +~ though, we do want to run the original code, so we allow the trap to fall +~ through to it. We carefully note fall-through with comments, to make the +~ traps easier to maintain. +~ +~ The kinds of side-effects we care about can all be written this way, +~ though occasionally we wind up having to compute an intermediate value that +~ the code we're trapping will then compute independently, and it's important +~ to keep those implementations in sync. +~ +~ The most significant challenge is that, because our trap will be defined +~ at the same point in loading as the original word would be, it can only use +~ words that are available at that point. Occasionally this results in some +~ awkwardness. +~ +~ Code for these traps winds up looking a lot like code written for log-load +~ alternates, in that we have to do the dictionary lookups and other +~ compilation ourselves. +~ +~ At times, we also need to trap words that are implemented in assembler. +~ The most notable of these is "sys-write", which is fundamental to what the +~ hex transform does. The details of how we make our traps work with assembler +~ words are explained below, as they arise. +: hex-colon-alternate + word value@ + + ~ The word "variable" is itself a word-defining word, and we will + ~ special-case its definition to special-case the definitions of particular + ~ variables. It's very very metacircular. + dup s" variable" stringcmp 0 = { + ~ Don't lose track of the layering happening here. The word "variable" + ~ is a regular docol word; it's defining a word that's implemented in + ~ assembly, but it can use whatever Forthy logic it wants to do so. In + ~ this case we're going to have it run a little extra logic, then continue + ~ with the rest of its usual definition. + + ~ Before we get to the extra logic, we do want an entry header for + ~ "variable" itself, so we do that... This takes care of all of colon's + ~ responsibilities except switching to compile mode; we'll do that part + ~ after we've output our payload. + create dropstring + s" docol" find entry-to-execution-token execute , + make-hidden + + ~ There's two variables that we want to point to the outer backing + ~ stores, rather than the inner ones. The code for that is a bit + ~ repetitive, so we have a helper for it; see above. + ~ + ~ It is worth stopping to contemplate the meaning of sharing these two + ~ variables in particular between the inner and outer contexts. + ~ Essentially this says that they're both reading the same input stream, + ~ and the two copies of the interpreter both share the same state. Thus, + ~ trading off responsibility for lexing between inner and outer contexts + ~ works just like trading off responsibility between two interpreters + ~ when there's no transformation involved, or between a regular + ~ interpreter and a transform. + ~ + ~ If we didn't do this, we'd still have to invent some way to control + ~ what input the inner context sees, and the concept of "the next + ~ character" would become more complex during the transform and require + ~ care and attention. Sharing this stuff keeps it simple. + s" interpreter-flags" hex-variable-use-outer + s" main-input-buffer" hex-variable-use-outer + ~ After this, we can return control to the regularly-scheduled + ~ "variable", which will do the "create" and all that. That stuff isn't + ~ colon's responsibility, so it isn't our responsibility, it'll happen + ~ regardless. + + ~ Notably, we do NOT share transformation-state. There are at least a + ~ hundred places where it would need to be carefully swapped, and we + ~ already have quite enough of that kind of brittleness in the log-load + ~ transform. Plus, there's no obvious way to do a real push/pop notion for + ~ it, so we wouldn't get arbitrary nesting of transforms anyway. By + ~ letting the code inside the hex transform have its own, separate copy of + ~ transformation-state, we avoid that problem entirely, and it's still not + ~ hard to access the "other" one when circumstances require. + + ~ Now we close out colon's responsibilities by switching to compile + ~ mode. We return from colon after this. The hex transform will continue + ~ by processing the source words that form the regular body of "variable", + ~ eventually hitting the matching semicolon. Our friendly tampering is + ~ now complete! + ' ] entry-to-execution-token execute + exit + } if + + ~ Now we want to override s". As usual, that's the single most annoying + ~ string to quote, so we cheat. + dup ' s" entry-to-name stringcmp 0 = { + ~ Create the word header. It's a normal docol word, so that much is + ~ simple. + create dropstring + s" docol" find entry-to-execution-token execute , + make-hidden + + ~ This time around we would really rather just always use the alternate, + ~ which already untangles the layered nonsense. So we have the payload + ~ call the alternate directly, then exit. We could come up with a way to + ~ then skip forward in the code under transformation, but that would be + ~ complex, and it's unnecessary: We let it keep running, outputting the + ~ usual body of s", which we know will never be reached. + ' hex-string-alternate entry-to-execution-token , + s" exit" find entry-to-execution-token , + + ~ As before, finish up colon's responsibilities, then return control to + ~ the code under transformation. + ' ] entry-to-execution-token execute + exit + } if + + ~ Same deal for .". Hey, we're getting good at this! + dup ' ." entry-to-name stringcmp 0 = { + create dropstring + s" docol" find entry-to-execution-token execute , + make-hidden + + ' hex-dot-string-alternate entry-to-execution-token , + s" exit" find entry-to-execution-token , + + ' ] entry-to-execution-token execute + exit + } if + + ~ We want to suppress the behavior of relink-main-input-buffer-to-stdin + ~ entirely. Happily, that's easy. We need to do this because otherwise the + ~ transformed code will mess with our outer interpreter! + dup s" relink-main-input-buffer-to-stdin" stringcmp 0 = { + create dropstring + s" docol" find entry-to-execution-token execute , + make-hidden + + ~ Return before doing anything. + s" exit" find entry-to-execution-token , + + ' ] entry-to-execution-token execute + exit + } if + + ~ We are entirely replacing sys-write with our own version. It's an + ~ assembly word, and we're replacing it with a Forth word, so there's some + ~ matching code in the ;asm alternate that makes sure to not mess that up. + dup s" sys-write" stringcmp 0 = { + create dropstring + s" docol" find entry-to-execution-token execute , + make-hidden + + ' hex-sys-write-replacement entry-to-execution-token , + s" exit" find entry-to-execution-token , + + ' ] entry-to-execution-token execute + exit + } if + + ~ Now we have a bunch of patches that keep track of output metadata. These + ~ are all Forth words, nothing too complicated about the actual patching. + dup s" reset-labels" stringcmp 0 = { + create dropstring + s" docol" find entry-to-execution-token execute , + make-hidden + + ~ This is only ever called in one place, the end of a failed iteration + ~ of label-loop. We need to reinitialize all the metadata entries whenever + ~ that happens, and reset-labels is a good semantic match for that + ~ behavior, so it's a good place for the hook. + ' create-hex-output-metadata-top-of-file-entries + entry-to-execution-token , + ~ Fall through to the inner implementation. + + ' ] entry-to-execution-token execute + exit + } if + + dup s" set-label" stringcmp 0 = { + create dropstring + s" docol" find entry-to-execution-token execute , + make-hidden + + ~ (new label value, label entry pointer --) + s" 2dup" find entry-to-execution-token , + ' hex-set-label-trace entry-to-execution-token , + ~ Fall through to the inner implementation. + + ' ] entry-to-execution-token execute + exit + } if + + + dup s" packstring" stringcmp 0 = { + create dropstring + s" docol" find entry-to-execution-token execute , + make-hidden + + ~ (output point, string pointer) + s" 2dup" find entry-to-execution-token , + ~ This one's a lot of work, so we defer to a single-purpose word. + ' hex-packstring-trace entry-to-execution-token , + ~ Fall through to the inner implementation. + + ' ] entry-to-execution-token execute + exit + } if + + ~ This is highly similar to packstring. + dup s" pack-raw-string" stringcmp 0 = { + create dropstring + s" docol" find entry-to-execution-token execute , + make-hidden + + ~ (output point, string pointer) + s" 2dup" find entry-to-execution-token , + s" dup" find entry-to-execution-token , + s" stringlen" find entry-to-execution-token , + ~ (output point, string pointer, string length NOT including terminator) + + ~ Before we add the metadata entry, we also want to call hex-pack-trace + ~ to make sure latest-output-point gets adjusted for the string. + ~ Otherwise, we'd see magic comments in the wrong place. + s" 3roll" find entry-to-execution-token , + s" dup" find entry-to-execution-token , + s" lit" find entry-to-execution-token , + 4 , + s" unroll" find entry-to-execution-token , + s" swap" find entry-to-execution-token , + s" dup" find entry-to-execution-token , + s" 3unroll" find entry-to-execution-token , + ~ (output point, string pointer, string length NOT including terminator, + ~ output point, string length NOT including terminator) + ' hex-pack-trace entry-to-execution-token , + + ~ (output point, string pointer, string length including terminator) + s" swap" find entry-to-execution-token , + ' hex-output-metadata-entry-type-raw-string-literal + entry-to-execution-token , + s" swap" find entry-to-execution-token , + ' add-hex-output-metadata-entry entry-to-execution-token , + ~ (output point, string pointer) + ~ Fall through to the inner implementation. + + ' ] entry-to-execution-token execute + exit + } if + + + dup s" pack8" stringcmp 0 = { + create dropstring + s" docol" find entry-to-execution-token execute , + make-hidden + + ~ We can't use over because this patch happens at the time pack8 is + ~ being defined, and it would be a forward reference. We do the equivalent + ~ by hand. + ~ (output point, value) + s" swap" find entry-to-execution-token , + s" dup" find entry-to-execution-token , + s" 3unroll" find entry-to-execution-token , + ~ (output point, value, output point) + s" lit" find entry-to-execution-token , + 1 , + ' hex-pack-trace entry-to-execution-token , + ~ Fall through to the inner implementation. + + ' ] entry-to-execution-token execute + exit + } if + + ~ Extremely similar to the pack8 patch. + dup s" pack16" stringcmp 0 = { + create dropstring + s" docol" find entry-to-execution-token execute , + make-hidden + + ~ We can't use over because this patch happens at the time pack8 is + ~ being defined, and it would be a forward reference. We do the equivalent + ~ by hand. + ~ (output point, value) + s" swap" find entry-to-execution-token , + s" dup" find entry-to-execution-token , + s" 3unroll" find entry-to-execution-token , + ~ (output point, value, output point) + s" lit" find entry-to-execution-token , + 2 , + ' hex-pack-trace entry-to-execution-token , + ~ Fall through to the inner implementation. + + ' ] entry-to-execution-token execute + exit + } if + + ~ Extremely similar to the pack8 patch. + dup s" pack32" stringcmp 0 = { + create dropstring + s" docol" find entry-to-execution-token execute , + make-hidden + + ~ We can't use over because this patch happens at the time pack8 is + ~ being defined, and it would be a forward reference. We do the equivalent + ~ by hand. + ~ (output point, value) + s" swap" find entry-to-execution-token , + s" dup" find entry-to-execution-token , + s" 3unroll" find entry-to-execution-token , + ~ (output point, value, output point) + s" lit" find entry-to-execution-token , + 4 , + ' hex-pack-trace entry-to-execution-token , + ~ Fall through to the inner implementation. + + ' ] entry-to-execution-token execute + exit + } if + + ~ Extremely similar to the pack8 patch. + dup s" pack64" stringcmp 0 = { + create dropstring + s" docol" find entry-to-execution-token execute , + make-hidden + + ~ We can't use over because this patch happens at the time pack8 is + ~ being defined, and it would be a forward reference. We do the equivalent + ~ by hand. + ~ (output point, value) + s" swap" find entry-to-execution-token , + s" dup" find entry-to-execution-token , + s" 3unroll" find entry-to-execution-token , + ~ (output point, value, output point) + s" lit" find entry-to-execution-token , + 8 , + ' hex-pack-trace entry-to-execution-token , + ~ Fall through to the inner implementation. + + ' ] entry-to-execution-token execute + exit + } if + + ~ If no special case matches, we fall back to just being a regular colon. + ~ We already read the word name above, so we have to do the rest of the + ~ steps ourselves as well. + create dropstring + s" docol" find entry-to-execution-token execute , + make-hidden + ' ] entry-to-execution-token execute + ; + + +~ This word gets installed as a trap at the beginning of every Forth word, +~ so that it can examine the control stack and produce kinds of metadata that +~ require information that only exists at runtime. See +~ hex-semicolon-alterante, below, for details of how it's installed. +~ +~ It is worth noting that hex-trap can coexist with the more-targeted traps +~ installed by hex-colon-alternate, above. It runs before them, because +~ hex-colon-alternate prepends its stuff first and then +~ hex-semicolon-alternate slides that all forward to prepend the call to +~ hex-trace. +~ +~ TODO this probably doesn't serve a purpose in the end, and should go away, +~ but the actual stack-walking code is useful and should be kept until it's +~ been refactored into something worth keeping. +: hex-trace + exit + ~ value@ here @ swap 0 hex-output-metadata-entry-type-line-comment + ~ add-hex-output-metadata-entry + newline + ~ This stack-walking code is modeled off the code in list-callers, in + ~ language/dynamic/dynamic.e. + ~ + + ~ This is the bit of the hex transform where all the memory-management + ~ assumptions are in play at once, so, in understanding it, it's probably + ~ a good idea to skim back over the background described under "hex + ~ transform implementation", above, and make sure it's fresh in your mind. + ~ + ~ It's important to notice that, as we walk it, the values of "here" and + ~ "latest" reflect the log and dictionary of the code being transformed, ie. + ~ they are the ones in the output buffer, not the log and dictionary of the + ~ outer program. Therefore, we can use containing-entry to find the entry + ~ headers for words inside the transform, and if we want to look at words + ~ outside the transform we can wrap containing-entry in + ~ swap-transform-variables. We can't use is-in-log, though, because the + ~ label-loop buffer is inside the output buffer, which is inside the outer + ~ log, so it doesn't make the distinctions we need it to. + ~ + ~ Every word inside the transform will have hex-transform-one exactly once + ~ in its call chain, so we walk the control stack only up to that point. + control@ + { ~ First, check to make sure we haven't run off the bottom of the control + ~ stack. Remember, the control stack is an array of pointers, so this is a + ~ simple bounds check. + dup r0 @ > + + ~ If it is, we also want to look at what it points to. + dup { + ~ Dereference the pointer we've been iterating (... we're actually + ~ looking at a pointer to a pointer ...), and check that it's not inside + ~ hex-transform-one's body. + drop dup @ + + ~ The inner dictionary is inside the outer log, so we'll get an entry + ~ header regardless of which dictionary we consider, but it may not be + ~ the one we care about. If we want to get the correct entry pointer, we + ~ have to make sure we're looking at the outer dictionary. + swap-transform-variables + + ~ Check what entry the control pointer is in. + containing-entry + + ~ Get the entry for hex-transform-one to compare against. This would + ~ be a forward reference, so we need to look it up at runtime. + s" hex-transform-one" find + + ~ Now switch back. + swap-transform-variables + + ~ If they're equal, we want to stop the loop. + != + } if } + { dup @ + + ~ TODO this isn't in quine.asm, so if we keep this code at all, that needs + ~ to be dealt with + ~ symbolize-pointer space + + ~ Iterate to the next pointer in the control stack. It's an array of + ~ pointers, so this is just arithmetic. + 8 + } while + drop ; + + +~ We want to trace every call to a Forth word, for use in generating +~ metadata. To do this, we want to add a prefix trap to the calls, but we +~ can't put the logic in hex-colon-alternate because we don't know which +~ words are Forth words and which are assembly words until we see the +~ semicolon. +~ +~ So, we do it here in hex-semicolon-alternate. The word body has already +~ been compiled at this point, so we slide the whole body forward to make +~ room. This is explicitly allowed by our compilation model; you may be +~ aware that it's also how high-level flow control is implemented. +~ +~ You might find it useful to compare this code to the implementations of +~ high-level flow-control utilities in language/static/log-load.e; it operates +~ under a very similar set of constraints, except that fortunately we don't +~ have the nightmare multi-dictionary situation that the log-load transform +~ does. +: hex-semicolon-alternate + latest @ entry-to-execution-token 8 + + dup dup dup + ~ (start, start, start, start) + + ~ The actual body of the trap is just a single word long (see below). If + ~ that ever changes, make sure to keep the amount we slide by in sync with + ~ it. + 8 + swap ~ TODO change number of words + ~ (start, start, adjusted start, start) + + here @ swap - + ~ (start, start, adjusted start, length) + + memmove + ~ (start) + + ~ We're going to use comma to append compiled code, so let's swap out the + ~ value of "here" to point to the space we just opened up. + here @ swap here ! + ~ (old "here") + + ~ This applies to every single word terminated with semicolon, which is + ~ all the Forth words that are created directly rather than with "variable" + ~ or some similar high-level facility. Because we also can't have forward + ~ references, that means we can't assume we'll have any available words in + ~ the transformed code to call, at all. So, the only thing we can usefully + ~ have the trap do is call a word that's defined as part of the transform. + ~ + ~ Fortunately, we do get to assume there will be a normal entry header in + ~ place for the word we're trapping. So we don't need to scrounge up any + ~ runtime data for the trap's benefit; the trap will be able to get what it + ~ needs by inspecting the control stack and the log. + ' hex-trace entry-to-execution-token , + + ~ We're done compiling the trap's contents, so swap "here" back. Again, + ~ keep this in sync with the length of the trap. + 8 + here ! + + ~ Now we do the regular semicolon stuff. + [ ' ; entry-to-execution-token , ] + ; make-immediate + + +: hex-semicolon-assembly-alternate + latest @ entry-to-name + dup s" sys-write" stringcmp 0 = { + ~ As detailed in hex-colon-alternate, above, sys-write is implemented in + ~ assembly but we replace it with Forth. This logic here collaborates with + ~ the logic there to make that work. + ~ + ~ The transformed code will still have compiled some assembly, though it + ~ won't be reached and would crash, so we need to fix alignment before we + ~ call semicolon. + drop + here @ 8 packalign here ! + [ ' ; entry-to-execution-token , ] + exit + } if + drop + + ~ If no special case matches, we fall back to the regular behavior. + [ ' ;asm entry-to-execution-token , ] + ; make-immediate + + +~ Because docol requires it, we provide a special mini-version of the label +~ system. We only do L@' and L!', because that's all we need. Unlike with the +~ label transform, these are NOT real labels; they're restricted similarly to +~ how they are for the log-load transform. +~ +~ Per the precedence rules in hex-transform-one, this version will apply to +~ all invocations, regardless of compile mode vs. immediate execution. +~ However, its own logic defers to the inner label system when that exists. +: hex-L@'-alternate + ~ If there's an inner copy of L@' defined, defer to it. + s" L@'" find dup { + entry-to-execution-token execute + exit + } { drop } if-else + + word dropstring + transformation-state transformation-state-label-scratch @ + ; make-immediate + +: hex-L!'-alternate + ~ If there's an inner copy of L!' defined, defer to it. + s" L!'" find dup { + entry-to-execution-token execute + exit + } { drop } if-else + + word dropstring + transformation-state transformation-state-label-scratch ! + ; make-immediate + + +~ The code we're compiling may try to exit when it's done. We want to return +~ to our caller instead; we turn that into an nop. This isn't quite correct, +~ but it will work as long as the inner code doesn't try to do anything else +~ after. +: hex-bye-alternate ; + + +~ Okay, now here's an interesting one. We need to provide additional +~ behavior for "allocate", but only when it is called directly from the body +~ of label-loop. That condition of being in the body is only testable +~ immediately, though "allocate" is not an immediate word and the effect of it +~ needs to happen later. This is crucial functionality that we can't work +~ without. So, we use a replacement and an alternate that work in tandem. +~ +~ First, we define the replacement... +~ +~ (size -- pointer) +: hex-allocate-replacement + dup allocate swap + ~ (result pointer, size) + + transformation-state transformation-state-output-metadata @ + hex-output-metadata-label-loop-buffer-length ! + ~ (result pointer) + + dup + transformation-state transformation-state-output-metadata @ + hex-output-metadata-label-loop-buffer-start ! + ~ (result pointer) + + create-hex-output-metadata-top-of-file-entries ; + +~ ... then, we define the alternate. +~ +~ The alternate is flagged as immediate, which means hex-transform-one will +~ treat it as always having the highest priority. It checks what mode we're +~ in; if we're immediate, it runs the inner "allocate" if it exists or the +~ outer one if not. If we're compiling, it checks if we're in the body of +~ label-loop and compiles an execution token for the replacement if so, or the +~ inner "allocate" if not. +~ +~ If that was confusing: The point is, by having the immediate effect be +~ compilation, we get to have it both ways. +: hex-allocate-alternate + interpreter-flags @ 0x01 & { + ~ Compile mode. + latest @ entry-to-name dup s" label-loop" stringcmp 0 = { + ~ In label-loop. + drop + ' hex-allocate-replacement entry-to-execution-token , + } { + ~ Not in label-loop. + drop + s" allocate" find entry-to-execution-token , + } if-else + } { + ~ Immediate mode. + s" allocate" find + dup { entry-to-execution-token execute } { drop allocate } if-else + } if-else ; make-immediate + + +~ In order to handle code that slides compiled code around, most notably the +~ high-level flow-control words, we trap memcopy and memmove. These are +~ assembly words, so trapping them with hex-colon-alternate is inconvenient; +~ instead we patch their callers. This is like the "allocate" trap, but +~ simpler. There's no special logic needed here, we just unconditionally +~ install the trap. +~ +~ This word is responsible for processing the hook. For code running +~ directly under the hex transform, it's installed by hex-memcopy-alternate +~ and hex-memmove-alternate, immediately below. For code running in a nested +~ label transform, see hex-label-word-replacement, further down. Note +~ carefully that when it's in a nested transform, the hook only runs in +~ immediate mode; running it in compile mode would affect the program being +~ compiled. +~ +~ It is worth noting that, as of the time this code was written, there is +~ never actually any memory sliding of compiled code that occurs outside of a +~ nested transform. This aligns with intuition, since sliding is really only +~ done in the high-level flow-control words. However, should it happen in the +~ future, we still want it to work, so it's implemented. +~ +~ The actual metadata handling is done by hex-memory-slide-trace, which is +~ above, near the other words that trace memory operations. We let the +~ replacement handle preserving the parameters so that the trace doesn't need +~ to, since that would be a difference between it and the other words similar +~ to it. +~ +~ (source, destination, length -- source, destination, length) +: hex-memory-slide-replacement + 2 pick 2 pick 2 pick hex-memory-slide-trace ; + +~ These two alternates are responsible for installing +~ hex-memory-slide-replacement, immediately above, in invocations of memcopy +~ and memmove that occur directly under the hex transform. For invocations +~ that occur in nested transforms, see hex-label-word-replacement, further +~ down. +: hex-memcopy-alternate + interpreter-flags @ 0x01 & { + ' hex-memory-slide-replacement entry-to-execution-token , + s" memcopy" find entry-to-execution-token , + } { + hex-memory-slide-replacement + s" memcopy" find entry-to-execution-token execute + } if-else ; make-immediate +: hex-memmove-alternate + interpreter-flags @ 0x01 & { + ' hex-memory-slide-replacement entry-to-execution-token , + s" memmove" find entry-to-execution-token , + } { + hex-memory-slide-replacement + s" memmove" find entry-to-execution-token execute + } if-else ; make-immediate + + +~ Similar to how we handle "allocate" as a replacement which is installed by +~ an alternate, we do the same for tilde, but for a different reason. The +~ semantics of ordinary, non-transformed tilde skip over comment text while +~ compiling, and discard it. We need to do processing of that text later, when +~ the comment's position would be reached in running the word it's part of, so +~ the alternate compiles it appropriately. +~ +~ Importantly, the replacement is only used for comments that have a special +~ comment syntax unique to the hex transform, as described in more detail in +~ the alternate, below. If it were used for all comments, it would interfere +~ with the distance of branches, which could get quite complex to work around. +~ It is the responsibility of words that use the hex-transform comment syntax +~ to avoid placing such comments within the scope of a branch. This applies +~ only to explicit, precomputed branches using the words "branch" and +~ "branch0". Fortunately, implicit branches that are part of high-level +~ flow-control words such as "if" and "while" will work fine, since the +~ substitution occurs before the flow control words start sliding code around, +~ and it updates "here" appropriately. +~ +~ Because hex-tilde-replacement is called in various situations during +~ various phases of compilation and interpreting, it can't take responsibility +~ for knowing how the string pointer it's given is allocated. It assumes the +~ string pointer will be valid indefinitely; it's the caller's responsibility +~ to make sure of that. +~ +~ (has non-space this line, string pointer --) +: hex-tilde-replacement + transformation-state transformation-state-output-metadata @ + hex-output-metadata-latest-output-point @ + swap + ~ (has non-space this line, data start, string pointer) + + ~ If the output wouldn't go in the label-loop buffer, don't do any of + ~ this. + ~ + ~ That may seem spurious, but remember, the same assembly instructions + ~ are used for compiling the final binary and defining words within the + ~ compiler. In particular, "variable" will hit this, and there are surely + ~ others. + over is-in-label-loop-buffer not { 3 ndrop exit } if + + ~ Copy the first space-separated word of the comment body into scratch + ~ space to see if it's a special instruction. To minimize interference with + ~ the program under transformation, we use the outer log's scratch space + ~ rather than the inner one's. + swap-transform-variables + dup + here @ + ~ (... source pointer, destination pointer) + { over 8@ dup 0x20 != && } { + swap unpack8 swap 3unroll pack8 + } while + 0 pack8 drop drop + + here @ + swap-transform-variables + + ~ (has non-space this line, data start, string pointer, + ~ first word string pointer) + dup read-integer 0 = { + ~ (has non-space, data start, string pointer, first word string pointer, + ~ data length) + 3unroll stringlen 1+ + + ~ (has non-space, data start, data length, adjusted string pointer) + 4 roll drop + 1 swap + ~ (data start, data length, is suffix, adjusted string pointer) + } { + drop 0 swap + ~ (has non-space, data start, data length, string pointer) + 4 roll swap + ~ (data start, data length, is suffix, string pointer) + } if-else + ~ (data start, data length, is suffix, adjusted string pointer) + + ~ Now look for instruction keywords. These keywords are expected to be the + ~ entire line, except that a length field can come before them. + dup s" adjust-length" stringcmp 0 = { + ~ Modify the most recent comment entry by adjusting its length. + drop drop swap drop adjust-latest-comment-metadata-entry-length + exit + } if + dup s" adjust-start" stringcmp 0 = { + ~ Modify the most recent output-producing entry by adjusting its start. + drop drop swap drop adjust-latest-output-producing-metadata-entry-start + exit + } if + dup s" set-output-point" stringcmp 0 = { + ~ Override the latest output point. + drop drop swap drop set-latest-output-point + exit + } if + dup s" adjust-output-point" stringcmp 0 = { + ~ Modify the latest output point. + drop drop swap drop adjust-latest-output-point + exit + } if + + dup s" data-adjust-length" stringcmp 0 = { + ~ Modify the most recent comment entry by adjusting its length, also + ~ using data from the most recent data entry. + drop drop swap drop data-adjust-latest-comment-metadata-entry-length + exit + } if + dup s" data-adjust-start" stringcmp 0 = { + ~ Modify the most recent output-producing entry by adjusting its start, + ~ also using data from the most recent data entry. + drop drop swap drop + data-adjust-latest-output-producing-metadata-entry-start + exit + } if + dup s" data-set-output-point" stringcmp 0 = { + ~ Override the latest output point, also using data from the most recent + ~ data entry. + drop drop swap drop data-set-latest-output-point + exit + } if + dup s" data-adjust-output-point" stringcmp 0 = { + ~ Modify the latest output point, also using data from the most recent + ~ data entry. + drop drop swap drop data-adjust-latest-output-point + exit + } if + + dup s" suppress" stringcmp 0 = { + ~ Suppress the next N entries, or adjust the remaining count of entries + ~ to suppress. + drop drop swap drop adjust-hex-output-suppression-count + exit + } if + + dup s" delete-first-comment-at-output-point" stringcmp 0 = { + 4 ndrop delete-first-comment-metadata-entry-at-output-point + exit + } if + + dup s" delete-last-comment-at-output-point" stringcmp 0 = { + 4 ndrop delete-latest-comment-metadata-entry-at-output-point + exit + } if + + dup s" fresh-line" stringcmp 0 = { + ~ Create a new fresh-line entry. + drop drop hex-output-metadata-entry-type-fresh-line 0 + add-hex-output-metadata-entry + exit + } if + + dup s" blank-line" stringcmp 0 = { + ~ Create a new blank-line entry. + drop drop hex-output-metadata-entry-type-blank-line 0 + add-hex-output-metadata-entry + exit + } if + + dup s" indent" stringcmp 0 = { + ~ Create a new indent entry with a positive indentation delta. + drop drop hex-output-metadata-entry-type-indent 2 + add-hex-output-metadata-entry + exit + } if + + dup s" deindent" stringcmp 0 = { + ~ Create a new indent entry with a negative indentation delta. + drop drop hex-output-metadata-entry-type-indent -2 + add-hex-output-metadata-entry + exit + } if + + ~ Now we have some commands for the various -provide-subitem- variants. + ~ Notice with all of these that we intentionally read one stack level deeper + ~ than anything of ours, which will be a value provided by the code the + ~ magic comment is embedded in. + dup s" provide-decimal" stringcmp 0 = { + ~ Create a new "push subitem decimal" entry. + drop drop + 2 pick hex-output-metadata-entry-type-push-subitem-decimal swap + add-hex-output-metadata-entry + exit + } if + dup s" provide-hex" stringcmp 0 = { + ~ Create a new "push subitem hex" entry. + drop drop + 2 pick hex-output-metadata-entry-type-push-subitem-hex swap + add-hex-output-metadata-entry + exit + } if + dup s" provide-hex8" stringcmp 0 = { + ~ Create a new "push subitem hex8" entry. + drop drop + 2 pick hex-output-metadata-entry-type-push-subitem-hex8 swap + add-hex-output-metadata-entry + exit + } if + dup s" provide-hex16" stringcmp 0 = { + ~ Create a new "push subitem hex16" entry. + drop drop + 2 pick hex-output-metadata-entry-type-push-subitem-hex16 swap + add-hex-output-metadata-entry + exit + } if + dup s" provide-hex32" stringcmp 0 = { + ~ Create a new "push subitem hex32" entry. + drop drop + 2 pick hex-output-metadata-entry-type-push-subitem-hex32 swap + add-hex-output-metadata-entry + exit + } if + dup s" provide-hex64" stringcmp 0 = { + ~ Create a new "push subitem hex64" entry. + drop drop + 2 pick hex-output-metadata-entry-type-push-subitem-hex64 swap + add-hex-output-metadata-entry + exit + } if + dup s" provide-string" stringcmp 0 = { + drop drop + 2 pick hex-output-metadata-entry-type-push-subitem-string swap + add-hex-output-metadata-entry + exit + } if + dup s" provide-string-copy" stringcmp 0 = { + drop drop + 2 pick hex-output-metadata-entry-type-push-subitem-string swap + swap-transform-variables here @ swap-transform-variables dup 3unroll + ~ (copy, original, copy) + over stringlen 1+ memcopy + swap-transform-variables allocate-string swap-transform-variables + add-hex-output-metadata-entry + exit + } if + dup s" provide-keyword" stringcmp 0 = { + ~ Create a new "push subitem string" entry given a keyword pointer. + drop drop + 2 pick execution-token-to-entry entry-to-name + hex-output-metadata-entry-type-push-subitem-string swap + add-hex-output-metadata-entry + exit + } if + dup s" provide-data" stringcmp 0 = { + ~ Create a new "push subitem data" entry. + drop drop + 2 pick hex-output-metadata-entry-type-saved-data swap + add-hex-output-metadata-entry + exit + } if + + ~ Next are the subitem stack manipulation commands. These deal with the + ~ subitem entry stack, just as the provide-* commands do, but at this phase + ~ of processing the effect is just to create an entry in the entry array, + ~ not to do any actual stack manipulation, so their implementations are + ~ quite simple. + dup s" drop-subitem" stringcmp 0 = { + ~ Create a new "drop subitem" entry. + drop drop hex-output-metadata-entry-type-drop-subitem 0 + add-hex-output-metadata-entry + exit + } if + dup s" swap-subitems" stringcmp 0 = { + ~ Create a new "swap subitem" entry. + drop drop hex-output-metadata-entry-type-swap-subitems 0 + add-hex-output-metadata-entry + exit + } if + dup s" roll-subitems" stringcmp 0 = { + ~ Create a new "roll subitem" entry. The parameter goes in the content + ~ field, and the length is always zero. + drop drop 0 swap hex-output-metadata-entry-type-roll-subitems swap + add-hex-output-metadata-entry + exit + } if + + ~ If it's flagged as a suffix by the logic above, create a new + ~ suffix-comment entry. + swap { + hex-output-metadata-entry-type-suffix-comment swap + add-hex-output-metadata-entry + exit + } if + + ~ Create a new line-comment entry. + hex-output-metadata-entry-type-line-comment swap + add-hex-output-metadata-entry ; + + +~ We need to define a few helpers for use in +~ hex-read-magic-comment-introducer, below. The string pointers these return +~ are allocated in the manner described there. +~ +~ (first key -- body-string-pointer) +: hex-read-nonempty-comment-body + ~ This is the rare case in the transforms where we want to get into the + ~ nitty-gritty of lexing. Normally we rely on the upstream implementations, + ~ even when we have to go out of our way to do so, because we don't want to + ~ have to update the transforms every time there's a new syntax feature. + ~ Here, however, we've got a syntax that only has meaning to the transform, + ~ so there's no choice. + swap-transform-variables here @ swap-transform-variables + + ~ This little loop is a modified version of the original tilde loop. + swap { dup dup 0x0a != && } { pack8 key } while drop + + 0 pack8 + 8 packalign + + drop + swap-transform-variables here @ swap-transform-variables ; + +: hex-read-empty-comment-body + swap-transform-variables here @ swap-transform-variables + + 0 pack8 + 8 packalign + + drop + swap-transform-variables here @ swap-transform-variables ; + + +~ This is called from a tilde alternate (such as hex-tilde-alternate, but +~ there are several). Its purpose is to do the lexical analysis of the +~ sequence that introduces a magic comment; it assumes that the leading tilde +~ character has just been consumed from the main input, leaving the separating +~ space as the next character ready to be read. +~ +~ Recall that the delimiter for a regular comment is "~ " (the trailing +~ space is part of it for this purpose). The delimiter for a magic comment is +~ "~ : ". A regular comment may also have a linefeed instead of a space, +~ indicating that it's empty; magic comments likewise can be empty, by ending +~ the line immediately after the colon. +~ +~ This word does the necessary reading, then leaves the input pointing at +~ the body of the comment and returns a value indicating what type of comment +~ this is. A result code of 0 indicates it's a regular comment and the input +~ now points immediately after its ending delimiter; that is, to the start of +~ the next line of text. A value of 1 indicates it's a magic comment with +~ non-zero length and the input points immediately after the delimiter. A +~ value of 2 indicates it's a magic comment with zero length, and the input +~ again points to the start of the next line. +~ +~ Our caller will want to do things with the comment body, so we also return +~ a pointer to a string in temporary storage, for use in further processing. +~ This temporary storage is taken from the "real" log, outside the transform's +~ scope. It's not allocated, merely using the space right after "here", +~ because the caller may or may not wish to allocate it for the long term, and +~ may prefer that its long-term location be elsewhere. +~ +~ Some of the various magic-comment syntaxes care about spacing within the +~ comment body, so it's important to notice that only the first space after +~ the colon is consumed. Any subsequent spaces are part of the body. +~ +~ (-- body string pointer, result code) +: hex-read-magic-comment-introducer + ~ The original code goes byte-by-byte, checks that the value is nonzero + ~ and not equal to 0x0a (linefeed), and exits when either property fails. + ~ We want to do something different based on the very first characters of + ~ the comment body. So, we unroll the first few iterations of the loop so we + ~ can do that test before we fall back to the normal behavior. + ~ + ~ Specifically, we want to recognize a colon (0x3a) with a space (0x20) + ~ immediately after it. Since the interpeter peeked at the space delimiter, + ~ but didn't consume it, we also need to check for a space before the colon, + ~ for a total sequence of three bytes we need to match. We can't just skip + ~ over the space with "consume", because it might also be a linefeed, which + ~ would satisfy the exit criterion. We need to properly match it. + ~ + ~ We also recognize a colon followed immediately by a linefeed, as + ~ denoting an empty comment. + key dup dup 0x0a != && { + ~ If we got here, the first byte was not 0 or 0x0a. Now we check if it's + ~ colon. + 0x20 = { + ~ The first byte was 0x20. Now check the second. + key dup dup 0x0a != && { + dup 0x3a = { + ~ The second byte was 0x3a. Now check the third. + drop key dup { + ~ The third byte is not 0. + dup 0x20 = { + ~ We have a magic comment with a nonempty body. We must save + ~ the rest of it so we can return it. + drop key hex-read-nonempty-comment-body 1 + } { + ~ The third byte is not 0 or 0x20. + dup 0x0a = { + ~ We have a magic comment with an empty body, ending at a + ~ linefeed. As with nonempty ones, we prepare a temporary + ~ string to return. + drop hex-read-empty-comment-body 2 + } { + ~ The third byte was not 0, 0x0a, or 0x20. We got a space, + ~ a colon, and something else. So our special test failed, but + ~ the exit condition isn't met. So we're done unrolling. + hex-read-nonempty-comment-body 0 + } if-else + } if-else + } { + ~ We got a colon and a zero. This is a magic comment with an + ~ empty body, ending at the end of the file. + drop hex-read-empty-comment-body 2 + } if-else + } { + ~ Again, the special test failed but the exit condition isn't met. + hex-read-nonempty-comment-body 0 + } if-else + } { + ~ Again, the exit condition is met. + drop hex-read-empty-comment-body 0 + } if-else + } { + ~ Once more, the special test failed but the exit condition isn't met. + hex-read-nonempty-comment-body 0 + } if-else + } { + ~ For the last time, the exit condition is met. + drop hex-read-empty-comment-body 0 + } if-else ; + + + +~ The tilde alternate is a very important word for the hex transform; it has +~ the task of hooking comment processing to detect a special comment syntax +~ which can be used by words that generate binary output, to define comments +~ which will appear as part of the hex-dump version of that output, but which +~ will be ignored in normal execution. +~ +~ This mechanism allows us, for example, to avoid putting detailed knowledge +~ of the amd64 Evocation-assembly instructions in the hex transform; instead +~ all the details are kept in one place, the authoritative implementations of +~ those instructions. This will substantially improve the maintainability of +~ the code, especially when adding new architectures. +~ +~ This word hex-tilde-alternate only applies to code that is running +~ directly within the hex transform. When an inner transform runs within the +~ hex transform, the inner transform is responsible for lexical processing, so +~ this word never gets invoked. There is an additional variant word +~ hex-*-tilde-alternate for each inner transform that the hex transform +~ supports, responsible for conducting the analogous task within the rules of +~ that inner transform. +: hex-tilde-alternate + ~ The original tilde is already an immediate word, so we can take over + ~ those responsibilities directly without needing to do any extra work to + ~ check the mode we're in or anything like that. + hex-read-magic-comment-introducer + ~ (body string pointer, result code) + + { + ~ It's a magic comment, which may or may not be empty. We want to either + ~ compile or execute a call to hex-tilde-replacement. + interpreter-flags @ 0x01 & { + ~ We're in compile mode. + ~ + ~ Fortunately, we can pack the string directly into the output buffer, + ~ so we don't need to mess around with accumulate-string. + s" lit" find entry-to-execution-token , + has-non-space-this-input-line@ , + + s" litstring" find entry-to-execution-token , + here @ + swap packstring + 8 packalign + here ! + + ' hex-tilde-replacement entry-to-execution-token , + } { + ~ We're in interpret mode. + has-non-space-this-input-line@ swap + hex-tilde-replacement + } if-else + } { + ~ If it's a regular comment, we just ignore it. + drop + } if-else + + ~ Regardless of what path we took, we just consumed a newline, so make + ~ note of the fact. + 0 has-non-space-this-input-line! + ; make-immediate + + +~ This word runs whenever the label transform, running inside the hex +~ transform, encounters a tilde. It completely replaces the normal invocation +~ of tilde, just as hex-tilde-alternate does. It has the same responsibilities +~ around magic comments as hex-tilde-alternate, but for code which is +~ lexically processed by label-transform-one rather than by hex-transform-one. +~ +~ The actions it takes upon finding various sorts of comments differ from +~ hex-tilde-alternate, because the behavior of the label transform is +~ different from the behavior of the hex transform. +~ +~ This is installed by hex-label-word-replacement, bypassing the label +~ transform's normal precedence rules, so there's no need for it to be +~ immediate. +~ +~ Installing this word is the most significant change the hex transform +~ makes to the behavior of the label transform. +: hex-label-tilde-alternate + hex-read-magic-comment-introducer + ~ (body string pointer, result code) + + { + ~ It's a magic comment, which may or may not be empty. + interpreter-flags @ 0x01 & { + ~ We're in compile mode. We ignore magic comments in compile mode, + ~ because they shouldn't become part of the program being compiled. + drop + } { + ~ We're in interpret mode. We run the magic comment immediately. + has-non-space-this-input-line@ swap + swap-transform-variables allocate-string swap-transform-variables + hex-tilde-replacement + } if-else + } { + ~ If it's a regular comment, we just ignore it. + drop + } if-else + + ~ Regardless of what path we took, we just consumed a newline, so make + ~ note of the fact. + 0 has-non-space-this-input-line! ; + + +~ This word runs whenever the log-load transform, running inside the hex +~ transform, encounters a tilde. It completely replaces the normal invocation +~ of tilde, just as hex-tilde-alternate does. It has the same responsibilities +~ around magic comments as hex-tilde-alternate, but for code which is +~ lexically processed by log-load-transform-one rather than by +~ hex-transform-one. +~ +~ The actions it takes upon finding various sorts of comments differ from +~ hex-tilde-alternate, because the behavior of the log-load transform is +~ different from the behavior of the hex transform. +~ +~ This is installed by hex-log-load-word-replacement, bypassing the log-load +~ transform's normal precedence rules, so there's no need for it to be +~ immediate. +~ +~ Installing this word is the most significant change the hex transform +~ makes to the behavior of the log-load transform. +: hex-log-load-tilde-alternate + hex-read-magic-comment-introducer + ~ (body string pointer, result code) + + { + ~ It's a magic comment, which may or may not be empty. + interpreter-flags @ 0x01 & { + ~ We're in compile mode. We ignore magic comments in compile mode, + ~ because they shouldn't become part of the program being compiled. + drop + } { + ~ We're in interpret mode. We run the magic comment immediately. + has-non-space-this-input-line@ swap + swap-transform-variables allocate-string swap-transform-variables + hex-tilde-replacement + } if-else + } { + ~ If it's a regular comment, we just ignore it. + drop + } if-else + + ~ Regardless of what path we took, we just consumed a newline, so make + ~ note of the fact. + 0 has-non-space-this-input-line! ; + + +~ This input helper is a very unusual thing: It's used by hex-transform-one, +~ hex-label-word-replacement, and hex-log-load-word-replacement to skip the +~ space between words. That's necessary because we need to keep track of +~ whether we've had any non-blank stuff on each line yet, so that the tilde +~ replacement can figure out certain cases where a magic comment needs to be a +~ suffix comment rather than a line comment. +: hex-skip-space + { peek dup is-space } + { consume + ~ This will set the flag once for each newline, if we have multiple + ~ consecutive ones or something like that, but that's harmless. + 0x0a = { 0 has-non-space-this-input-line! } if + } while drop ; + + + +~ This word is installed into label-transform-one by the hex-word-alternate, +~ when the label transform runs inside the hex transform. It has the job of +~ performing substantive changes to the label transform's behavior, as +~ required by the hex transform. +~ +~ The most important changes to the base language that this word makes are +~ to call hex-label-tilde-alternate any time a tilde is encountered as a word, +~ and to call hex-skip-space before reading a word. +~ +~ This word is also responsible for invoking hex-memory-slide-replacement +~ when memcopy or memmove are encountered in immediate mode. Note that this +~ really pushes right up against the limit of what it's possible for a +~ transform to do without altering the compilation's output; invoking the +~ replacement in compile mode would do so. +: hex-label-word-replacement + hex-skip-space + word + + value@ + dup s" ~" stringcmp 0 = { + ~ We won't be going back to label-transform-one; clear away its state. + drop dropstring + + hex-label-tilde-alternate + + ~ Return directly to label-transform, faking the "not done" return value + ~ from label-transform-one. + 0 + 2 nexit + } if + dup s" memcopy" stringcmp 0 = { + interpreter-flags @ 0x01 & not { + drop hex-memory-slide-replacement exit + ~ Fall through, returning control to label-transform-one. + } if + } if + dup s" memmove" stringcmp 0 = { + interpreter-flags @ 0x01 & not { + drop hex-memory-slide-replacement exit + ~ Fall through, returning control to label-transform-one. + } if + } if + drop ; + + +~ This word is installed into log-load-transform-one by the +~ hex-word-alternate, when the log-load transform runs inside the hex +~ transform. It has the job of performing substantive changes to the log-load +~ transform's behavior, as required by the hex transform. +~ +~ The most important changes to the base language that this word makes are +~ to call hex-log-load-tilde-alternate any time a tilde is encountered as a +~ word, and to call hex-skip-space before reading a word. +: hex-log-load-word-replacement + hex-skip-space + word + + value@ + dup s" ~" stringcmp 0 = { + ~ We won't be going back to log-load-transform-one; clear away its state. + drop dropstring + + hex-log-load-tilde-alternate + + ~ Return directly to log-load-transform, faking the "not done" return + ~ value from log-load-transform-one. + 0 + 2 nexit + } if + drop ; + + +~ Just as hex-allocate-alternate has the job of installing +~ hex-allocate-replacement in only one single spot, so hex-word-alternate has +~ the job of installing hex-label-word-replacement. It gets installed when +~ "word" is called from lexically within label-transform-one, and at no other +~ time. +~ +~ Notice how the job of each transform's inner bits can be thought of as +~ calling "word" and doing something with the result, with all the lexical +~ processing delegated to "word". This makes "word" a good place to install a +~ modification to the transform's behavior. +~ +~ Its structure is highly similar to that of hex-allocate-alternate. +: hex-word-alternate + interpreter-flags @ 0x01 & { + ~ Compile mode. + latest @ entry-to-name + + dup s" label-transform-one" stringcmp 0 = { + drop + ' hex-label-word-replacement entry-to-execution-token , + exit + } if + + dup s" log-load-transform-one" stringcmp 0 = { + drop + ' hex-log-load-word-replacement entry-to-execution-token , + exit + } if + + ~ Not in a word we install stuff into. + drop + s" word" find entry-to-execution-token , + } { + ~ Immediate mode. + s" word" find + dup { entry-to-execution-token execute } { drop allocate } if-else + } if-else ; make-immediate + + +~ This runs instead of "interpret" in code under the hex transform. The +~ proximate effect of that is to make "include" work. Note, however, that this +~ replacement only applies directly under the hex transform. It isn't a +~ sensible approach under the log-load transform at all, and under the label +~ transform it would have such strange behavior that it isn't a good idea +~ either. +: hex-interpret-replacement + ~ This is a forward reference, so we need to look it up. + s" hex-transform-one" + swap-transform-variables find swap-transform-variables + entry-to-execution-token execute + + ~ We ignore the EOF indication, on the assumption something else will + ~ detect it and return to the outer loop, the one that's ours. For example, + ~ look at "include", in language/dynamic/files.e. + drop ; + + +~ This installs hex-interpret-replacement, compiling it in compile mode and +~ executing it in immediate mode. +: hex-interpret-alternate + interpreter-flags @ 0x01 & { + ' hex-interpret-replacement entry-to-execution-token , + } { + hex-interpret-replacement + } if-else ; make-immediate + + +~ This implements the hex transform for a single word. It is directly +~ analogous to "interpret", and reading language/dynamic/interpret.e may help +~ in understanding it, though it's meant to still make sense on its own. +~ +~ The hex transform runs code immediately. Whereas most transforms alter +~ what the transformed code compiles into, the hex transform alters what it +~ outputs. It's assumed that the output is a binary file of some sort; the +~ binary is output as hexadecimal, interspersed with comments describing the +~ code that produced it, including descriptions of what was executed, along +~ with any comments from the original code. +~ +~ The hex transform's alternates take priority over words defined under +~ the transform when running immediately, but for compilation, words defined +~ under the transform take precedence. In the event that neither an alternate +~ nor an entry in the inner dictionary is found, the outer dictionary is +~ checked; otherwise it's irrelevant. +~ +~ The point of this precedence rule is that string literals, which require +~ special treatment, will be using the inner implementation by the time it +~ gets to actually generating an executable image. It's important that there +~ not be any alternates altering the generated code, only the program actually +~ being compiled. Other literal syntaxes, such as tick, don't have alternates +~ at all (the hex transform manages to be more parsimonious with alternates +~ than other transforms), and will always be using inner versions. +~ +~ The precedence rule does fail to have the desired effect for label +~ references. For these, the alternates take the unusual step of manually +~ calling the inner versions when they exist. +~ TODO think through whether the precedence rule is actually doing anything... +~ +~ It expects to be called from "hex-transform", below, which loops. +~ +~ (-- done) +: hex-transform-one + hex-skip-space + word + + ~ If no word was returned, end the transformation. + dup 0 = { drop 1 exit } if + + ~ The string is on the top of the stack, so to get a pointer to it we get + ~ the stack address. + ~ (string) + value@ + + ~ If it's anything but tilde, set the input state flag accordingly. + dup s" ~" stringcmp 0 != { 1 has-non-space-this-input-line! } if + + ~ Check whether it's one of the words we have alternates for, and look up + ~ the alternate if so. + 0 swap + ~ (name as stack string, placeholder, name pointer) + dup s" self-codeword" stringcmp 0 = { + swap drop ' hex-self-codeword-alternate swap } if + ~ It is nontrivial to construct a string with a double-quote in it. + dup ' s" entry-to-name stringcmp 0 = { + swap drop ' hex-string-alternate swap } if + dup ' ." entry-to-name stringcmp 0 = { + swap drop ' hex-dot-string-alternate swap } if + dup s" create" stringcmp 0 = { swap drop ' hex-create-alternate swap } if + dup s" :" stringcmp 0 = { swap drop ' hex-colon-alternate swap } if + dup s" ;" stringcmp 0 = { swap drop ' hex-semicolon-alternate swap } if + dup s" ;asm" stringcmp 0 = { + swap drop ' hex-semicolon-assembly-alternate swap } if + dup s" L@'" stringcmp 0 = { swap drop ' hex-L@'-alternate swap } if + dup s" L!'" stringcmp 0 = { swap drop ' hex-L!'-alternate swap } if + dup s" bye" stringcmp 0 = { swap drop ' hex-bye-alternate swap } if + dup s" allocate" stringcmp 0 = { + swap drop ' hex-allocate-alternate swap } if + dup s" memcopy" stringcmp 0 = { swap drop ' hex-memcopy-alternate swap } if + dup s" memmove" stringcmp 0 = { swap drop ' hex-memmove-alternate swap } if + dup s" word" stringcmp 0 = { swap drop ' hex-word-alternate swap } if + dup s" ~" stringcmp 0 = { swap drop ' hex-tilde-alternate swap } if + dup s" interpret" stringcmp 0 = { + swap drop ' hex-interpret-alternate swap } if + ~ (name as stack string, 0 or alternate entry pointer, name pointer) + + find + ~ (stack string, 0 or alternate entry pointer, 0 or inner entry pointer) + + interpreter-flags @ 0x01 & { + ~ We're in compile mode. An alternate immediate entry has the highest + ~ precedence; an inner entry of any kind has second-highest. + over { + ~ An alternate entry exists; check its flags. + over entry-flags@ 0x01 & { + ~ It's an immediate entry, so it has precedence, regardless of + ~ what's up with the inner entry. Execute it. + drop dropstring-with-result + entry-to-execution-token execute + 0 exit + } { + ~ The alternate is not immediate, so check if there's an inner entry. + dup { + ~ There is also an inner entry. Check its flags. + dup entry-flags@ 0x01 & { + ~ The inner entry is immediate, so it has precedence. Execute it. + swap drop dropstring-with-result + entry-to-execution-token execute + 0 exit + } { + ~ The inner entry is not immediate, so the alternate has + ~ precedence. Compile it. + ~ + ~ Watch this space closely for correctness issues, it's a rare + ~ codepath. + drop dropstring-with-result + entry-to-execution-token , + 0 exit + } if-else + } { + ~ There's no inner entry. Compile the alternate. + ~ + ~ This path, too, is rare and should get close scrutiny for + ~ correctness. + drop dropstring-with-result + entry-to-execution-token , + 0 exit + } if-else + } if-else + } { + ~ There is no alternate entry; check for an inner entry. + dup { + ~ An inner entry exists; check its flags. + dup entry-flags@ 0x01 & { + ~ It's an immediate entry. Execute it. + swap drop dropstring-with-result + entry-to-execution-token execute + 0 exit + } { + ~ It's not an immediate entry. Compile it. + swap drop dropstring-with-result + entry-to-execution-token , + 0 exit + } if-else + } if + ~ If we got here, there's no inner or alternate entry; fall through. + } if-else + } { + ~ We're in immediate mode. An alternate entry of any kind has precedence. + over { + ~ There's an alternate entry. Execute it. + drop dropstring-with-result + entry-to-execution-token execute + 0 exit + } { + ~ There's no alternate entry. Check for an inner entry. + dup { + ~ An inner entry exists. Execute it. + swap drop dropstring-with-result + entry-to-execution-token execute + 0 exit + } if + ~ If we got here, there's no inner or alternate; fall through. + } if-else + } if-else + drop drop + ~ (stack string) + + ~ As a final fallback, we also check the outer dictionary, for immediate + ~ use only. This will allow things like assembly words to work. + value@ + swap-transform-variables + find + swap-transform-variables + dup { + ~ There's an outer entry; check the mode. + interpreter-flags @ 0x01 & { + ~ We're in compile mode; check the outer entry's flags. + dup entry-flags@ 0x01 & { + ~ It's an immediate word; execute it. + dropstring-with-result + entry-to-execution-token + execute + 0 exit + } { + ~ It's not an immediate word. Pretend it doesn't exist, and fall + ~ through. + drop + } if-else + } { + ~ We're in immediate mode. Execute the outer entry. + dropstring-with-result + entry-to-execution-token + execute + 0 exit + } if-else + } { + ~ There's no outer entry. Fall through. + drop + } if-else + ~ (stack string) + + ~ Check whether it's a number literal. + value@ read-integer 0 = { + ~ It's a number. + ~ + ~ (name as stack string, integer value) + dropstring-with-result + ~ (integer value) + + interpreter-flags @ 0x01 & { + ~ TODO this probably needs a lit alternate (!) + ' lit entry-to-execution-token , , + 0 exit + } if + + 0 exit + } if + ~ (stack string) + + ." No such word: " value@ emitstring newline dropstring 0 ; + + +~ This implements the hex transform for all words in a region given as +~ an input string. It is directly analogous to "quit", in +~ language/dynamic/interpret.e, but is more complex. +~ +~ (output metadata buffer start, output buffer start, output point, +~ input string pointer -- output buffer start, output point) +: hex-transform + main-input-buffer dup push-input-buffer + swap attach-string-to-input-buffer + ~ (output metadata buffer start, output buffer start, output point) + + ~ Save the old values of "here" and "latest", and set the initial values + ~ of the internal ones. These values need to persist across iterations, + ~ since client code will make its own updates to them and then rely on those + ~ updates having taken effect. So we do the swap just once, here outside the + ~ loop, and set it back when the loop ends. + ~ + ~ We also take this opportunity to initialize the other fields of + ~ transformation-state. While it might be nice to do these in some sort of + ~ logical order, it gets unreadable quickly, so we initialize them in the + ~ most convenient order given how we received our parameters. + here @ transformation-state transformation-state-saved-here ! + latest @ transformation-state transformation-state-saved-latest ! + here ! + 0 latest ! + transformation-state transformation-state-output-buffer-start ! + 0 transformation-state transformation-state-user-stack-depth ! + transformation-state transformation-state-output-metadata ! + ~ Now the stack has nothing of ours on it, so client code can do its thing. + + ~ We don't have the real values yet, so we can't create the initial + ~ entries in the output metadata, but we can and should zero it out to make + ~ sure everything else operating on it finds it well-formed. + zero-hex-output-metadata + + ~ The code under the hex transform never gets a real copy of warm-start, + ~ so it doesn't have the globals "here" and so on. It wants to have the + ~ globals, but it can't, not without our help. Why aren't we helping? + ~ + ~ Oh, right, we're helping right here. We just write them into the inner + ~ log and dictionary directly, by using our outer copy of "variable". These + ~ definitions don't have to do anything special; we point them to the same + ~ addresses as the outer ones. Note though that the actual values the + ~ transformed code sees sees are the wrapped, inner ones, since we swapped + ~ the values out elsewhere. + ~ + ~ If we provided these as alternates instead, that would work for code + ~ that doesn't have a nested transform, but the inner transforms want to be + ~ able to look them up by name. + swap-transform-variables log swap-transform-variables s" log" variable + swap-transform-variables s0 swap-transform-variables s" s0" variable + swap-transform-variables r0 swap-transform-variables s" r0" variable + swap-transform-variables latest swap-transform-variables s" latest" variable + swap-transform-variables here swap-transform-variables s" here" variable + + ~ It's important that the stack has nothing of ours on it that persists + ~ across iterations, so that client code can add and remove stuff there as + ~ it sees fit. + { hex-transform-one + ~ (done) + + ~ When the loop is done, get the real values of "here" and "latest" + ~ back. The internal "here" is also the output point, and will become our + ~ return value. The internal "latest" is discarded. + { transformation-state transformation-state-output-buffer-start @ + here @ + transformation-state transformation-state-saved-here @ here ! + transformation-state transformation-state-saved-latest @ latest ! + ~ (output point) + + ~ Though we don't actually use transformation-state outside of this + ~ invocation, for tidiness we zero it out. + 0 transformation-state transformation-state-saved-here ! + 0 transformation-state transformation-state-saved-latest ! + 0 transformation-state transformation-state-output-buffer-start ! + 0 transformation-state transformation-state-user-stack-depth ! + + ~ Also put the input source back how it was. + main-input-buffer pop-input-buffer + + exit } if } forever ; + -- cgit 1.4.1