From f3609dad9e7c78708d8049ed1629c22cb20d5580 Mon Sep 17 00:00:00 2001 From: Irene Knapp Date: Tue, 15 Sep 2026 12:48:19 -0700 Subject: refactor hex-tilde-alternate to split out lexing note that lexing of the body of a magic comment was already separate, part of hex-tilde-replacement. however, hex-tilde-alternate had been responsible for lexing the sequence that introduces a magic comment. because we want to make separate versions of the alternate for each inner transform, the lexing needed to be factored out. this does result in more copying, but it doesn't appear to noticeably affect runtime. Force-Push: yes Change-Id: I51a780d1dadb99f09bddb9eaccb1af387bd0ce87 --- transform.e | 269 ++++++++++++++++++++++++++++++++++++------------------------ 1 file changed, 160 insertions(+), 109 deletions(-) diff --git a/transform.e b/transform.e index 25442c2..8f01fed 100644 --- a/transform.e +++ b/transform.e @@ -3984,7 +3984,7 @@ allocate-transformation-state s" transformation-state" variable sys-write exit } if - postprocess-metadata-entries + ~ postprocess-metadata-entries ~ Importantly, postprocessing leaves the entries sorted, which means we ~ can go back and forth in a linear scan through the entry array while we @@ -4004,7 +4004,7 @@ allocate-transformation-state s" transformation-state" variable ~ it's important we not go beyond that point. { dup @ dup ~ (length remaining, current output address, metadata scan pointer, - ~ entry's data start, entry's data start) + ~ entry's first field, entry's first field) { ~ This is an nop, but for clarity's sake it's best to not depend on ~ details of the structure layout that perhaps not everyone has @@ -5152,22 +5152,73 @@ allocate-transformation-state s" transformation-state" variable add-hex-output-metadata-entry ; -~ The tilde alternate is a very important word for the hex transform; it has -~ the task of implementing a special comment syntax which can be used by words -~ that generate binary output, to define comments which will appear as part of -~ the hex-dump version of that output, but which will be ignored in normal -~ execution. +~ We need to define a few helpers for use in +~ hex-read-magic-comment-introducer, below. The string pointers these return +~ are allocated in the manner described there. ~ -~ This mechanism allows us, for example, to avoid putting detailed knowledge -~ of the amd64 Evocation-assembly instructions in the hex transform; instead -~ all the details are kept in one place, the authoritative implementations of -~ those instructions. This will substantially improve the maintainability of -~ the code, especially when adding new architectures. -: hex-tilde-alternate - ~ The original tilde is already an immediate word, so we can take over - ~ those responsibilities directly without needing to do any extra work to - ~ check the mode we're in or anything like that. +~ (first key -- body-string-pointer) +: hex-read-nonempty-comment-body + ~ This is the rare case in the transforms where we want to get into the + ~ nitty-gritty of lexing. Normally we rely on the upstream implementations, + ~ even when we have to go out of our way to do so, because we don't want to + ~ have to update the transforms every time there's a new syntax feature. + ~ Here, however, we've got a syntax that only has meaning to the transform, + ~ so there's no choice. + swap-transform-variables here @ swap-transform-variables + + ~ This little loop is a modified version of the original tilde loop. + swap { dup dup 0x0a != && } { pack8 key } while drop + + 0 pack8 + 8 packalign + + drop + swap-transform-variables here @ swap-transform-variables ; + +: hex-read-empty-comment-body + swap-transform-variables here @ swap-transform-variables + + 0 pack8 + 8 packalign + drop + swap-transform-variables here @ swap-transform-variables ; + + +~ This is called from a tilde alternate (such as hex-tilde-alternate, but +~ there are several). Its purpose is to do the lexical analysis of the +~ sequence that introduces a magic comment; it assumes that the leading tilde +~ character has just been consumed from the main input, leaving the separating +~ space as the next character ready to be read. +~ +~ Recall that the delimiter for a regular comment is "~ " (the trailing +~ space is part of it for this purpose). The delimiter for a magic comment is +~ "~ : ". A regular comment may also have a linefeed instead of a space, +~ indicating that it's empty; magic comments likewise can be empty, by ending +~ the line immediately after the colon. +~ +~ This word does the necessary reading, then leaves the input pointing at +~ the body of the comment and returns a value indicating what type of comment +~ this is. A result code of 0 indicates it's a regular comment and the input +~ now points immediately after its ending delimiter; that is, to the start of +~ the next line of text. A value of 1 indicates it's a magic comment with +~ non-zero length and the input points immediately after the delimiter. A +~ value of 2 indicates it's a magic comment with zero length, and the input +~ again points to the start of the next line. +~ +~ Our caller will want to do things with the comment body, so we also return +~ a pointer to a string in temporary storage, for use in further processing. +~ This temporary storage is taken from the "real" log, outside the transform's +~ scope. It's not allocated, merely using the space right after "here", +~ because the caller may or may not wish to allocate it for the long term, and +~ may prefer that its long-term location be elsewhere. +~ +~ Some of the various magic-comment syntaxes care about spacing within the +~ comment body, so it's important to notice that only the first space after +~ the colon is consumed. Any subsequent spaces are part of the body. +~ +~ (-- body string pointer, result code) +: hex-read-magic-comment-introducer ~ The original code goes byte-by-byte, checks that the value is nonzero ~ and not equal to 0x0a (linefeed), and exits when either property fails. ~ We want to do something different based on the very first characters of @@ -5189,127 +5240,112 @@ allocate-transformation-state s" transformation-state" variable 0x20 = { ~ The first byte was 0x20. Now check the second. key dup dup 0x0a != && { - 0x3a = { + dup 0x3a = { ~ The second byte was 0x3a. Now check the third. - key dup { + drop key dup { ~ The third byte is not 0. dup 0x20 = { - ~ The special test succeeded, so we want to either compile or - ~ execute a call to hex-tilde-replacement. First, though, we - ~ must save the rest of the comment body so we can provide it to - ~ the replacement at runtime. - ~ - ~ This is the rare case in the transforms where we want to get - ~ into the nitty-gritty of lexing. Normally we rely on the - ~ upstream implementations, even when we have to go out of our - ~ way to do so, because we don't want to have to update the - ~ transforms every time there's a new syntax feature. Here, - ~ however, we've got a syntax that only has meaning to the - ~ transform, so there's no choice. - drop - interpreter-flags @ 0x01 & { - ~ We're in compile mode. - ~ - ~ Fortunately, we can pack the string directly into the - ~ output buffer, so we don't need to mess around with - ~ accumulate-string. - s" lit" find entry-to-execution-token , - has-non-space-this-input-line@ , - s" litstring" find entry-to-execution-token , - here @ - key { dup dup 0x0a != && } { - pack8 key - } while drop - 0 pack8 - 8 packalign here ! - ' hex-tilde-replacement entry-to-execution-token , - } { - has-non-space-this-input-line@ - - swap-transform-variables here @ swap-transform-variables - - key { dup dup 0x0a != && } { pack8 key } while drop - 0 pack8 - 8 packalign - - swap-transform-variables - here @ swap here ! - swap-transform-variables - - transformation-state transformation-state-output-metadata @ - hex-output-metadata-latest-output-point @ - hex-tilde-replacement - ~ TODO this doesn't work yet - ~ the problem with running this in immediate mode during the - ~ label transform is that the latest output point is 0, - ~ because it's running as part of loading the copy of the - ~ compiler that sits directly inside the hex transform, not as - ~ part of reading the code to be compiled, which is inside - ~ the inner transforms. - } if-else + ~ We have a magic comment with a nonempty body. We must save + ~ the rest of it so we can return it. + drop key hex-read-nonempty-comment-body 1 } { ~ The third byte is not 0 or 0x20. - 0x0a = { - ~ The third byte is 0x0a. This is the alternate version of - ~ our special test, denoting a magic comment that happens to - ~ be empty. As with nonempty ones, we pack the string directly - ~ into the output buffer. - s" lit" find entry-to-execution-token , - has-non-space-this-input-line@ , - s" litstring" find entry-to-execution-token , - here @ - 0 pack8 - 8 packalign here ! - ' hex-tilde-replacement entry-to-execution-token , + dup 0x0a = { + ~ We have a magic comment with an empty body, ending at a + ~ linefeed. As with nonempty ones, we prepare a temporary + ~ string to return. + drop hex-read-empty-comment-body 2 } { - ~ The third byte was not 0, 0x0a, or 0x20. So our special - ~ test failed, but the exit condition isn't met. So we're done - ~ unrolling and can just do the original tilde loop for the - ~ rest. - key { dup dup 0x0a != && } { drop key } while drop + ~ The third byte was not 0, 0x0a, or 0x20. We got a space, + ~ a colon, and something else. So our special test failed, but + ~ the exit condition isn't met. So we're done unrolling. + hex-read-nonempty-comment-body 0 } if-else } if-else } { - ~ The exit condition is met, so we're done. - drop + ~ We got a colon and a zero. This is a magic comment with an + ~ empty body, ending at the end of the file. + drop hex-read-empty-comment-body 2 } if-else } { ~ Again, the special test failed but the exit condition isn't met. - key { dup dup 0x0a != && } { drop key } while drop + hex-read-nonempty-comment-body 0 } if-else } { ~ Again, the exit condition is met. - drop + drop hex-read-empty-comment-body 0 } if-else } { ~ Once more, the special test failed but the exit condition isn't met. - key { dup dup 0x0a != && } { drop key } while drop + hex-read-nonempty-comment-body 0 } if-else } { ~ For the last time, the exit condition is met. + drop hex-read-empty-comment-body 0 + } if-else ; + + + +~ The tilde alternate is a very important word for the hex transform; it has +~ the task of hooking comment processing to detect a special comment syntax +~ which can be used by words that generate binary output, to define comments +~ which will appear as part of the hex-dump version of that output, but which +~ will be ignored in normal execution. +~ +~ This mechanism allows us, for example, to avoid putting detailed knowledge +~ of the amd64 Evocation-assembly instructions in the hex transform; instead +~ all the details are kept in one place, the authoritative implementations of +~ those instructions. This will substantially improve the maintainability of +~ the code, especially when adding new architectures. +~ +~ This word hex-tilde-alternate only applies to code that is running +~ directly within the hex transform. When an inner transform runs within the +~ hex transform, the inner transform is responsible for lexical processing, so +~ this word never gets invoked. There is an additional variant word +~ hex-*-tilde-alternate for each inner transform that the hex transform +~ supports, responsible for conducting the analogous task within the rules of +~ that inner transform. +: hex-tilde-alternate + ~ The original tilde is already an immediate word, so we can take over + ~ those responsibilities directly without needing to do any extra work to + ~ check the mode we're in or anything like that. + hex-read-magic-comment-introducer + ~ (body string pointer, result code) + + { + ~ It's a magic comment, which may or may not be empty. We want to either + ~ compile or execute a call to hex-tilde-replacement. + interpreter-flags @ 0x01 & { + ~ We're in compile mode. + ~ + ~ Fortunately, we can pack the string directly into the output buffer, + ~ so we don't need to mess around with accumulate-string. + s" lit" find entry-to-execution-token , + has-non-space-this-input-line@ , + + s" litstring" find entry-to-execution-token , + here @ + swap packstring + 8 packalign + here ! + + ' hex-tilde-replacement entry-to-execution-token , + } { + ~ We're in interpret mode. + has-non-space-this-input-line@ swap + hex-tilde-replacement + } if-else + } { + ~ If it's a regular comment, we just ignore it. drop } if-else - ~ Regardless of what path we exited by, we just consumed a newline, so - ~ make note of the fact. + ~ Regardless of what path we took, we just consumed a newline, so make + ~ note of the fact. 0 has-non-space-this-input-line! ; make-immediate -~ This input helper is a very unusual thing: It's used by hex-transform-one -~ to skip the space between words. That's necessary because we need to keep -~ track of whether we've had any non-blank stuff on each line yet, so that the -~ tilde replacement can figure out certain cases where a magic comment needs -~ to be a suffix comment rather than a line comment. -: hex-skip-space - { peek dup is-space } - { consume - ~ This will set the flag once for each newline, if we have multiple - ~ consecutive ones or something like that, but that's harmless. - 0x0a = { 0 has-non-space-this-input-line! } if - } while drop ; - - : hex-label-word-replacement word @@ -5358,6 +5394,21 @@ allocate-transformation-state s" transformation-state" variable } if-else ; make-immediate +~ This input helper is a very unusual thing: It's used by hex-transform-one +~ to skip the space between words. That's necessary because we need to keep +~ track of whether we've had any non-blank stuff on each line yet, so that the +~ tilde replacement can figure out certain cases where a magic comment needs +~ to be a suffix comment rather than a line comment. +: hex-skip-space + { peek dup is-space } + { consume + ~ This will set the flag once for each newline, if we have multiple + ~ consecutive ones or something like that, but that's harmless. + 0x0a = { 0 has-non-space-this-input-line! } if + } while drop ; + + + ~ This implements the hex transform for a single word. It is directly ~ analogous to "interpret", and reading interpret.e may help in understanding ~ it, though it's meant to still make sense on its own. -- cgit 1.4.1