diff options
| -rw-r--r-- | transform.e | 269 |
1 files changed, 160 insertions, 109 deletions
diff --git a/transform.e b/transform.e index 25442c2..8f01fed 100644 --- a/transform.e +++ b/transform.e @@ -3984,7 +3984,7 @@ allocate-transformation-state s" transformation-state" variable sys-write exit } if - postprocess-metadata-entries + ~ postprocess-metadata-entries ~ Importantly, postprocessing leaves the entries sorted, which means we ~ can go back and forth in a linear scan through the entry array while we @@ -4004,7 +4004,7 @@ allocate-transformation-state s" transformation-state" variable ~ it's important we not go beyond that point. { dup @ dup ~ (length remaining, current output address, metadata scan pointer, - ~ entry's data start, entry's data start) + ~ entry's first field, entry's first field) { ~ This is an nop, but for clarity's sake it's best to not depend on ~ details of the structure layout that perhaps not everyone has @@ -5152,22 +5152,73 @@ allocate-transformation-state s" transformation-state" variable add-hex-output-metadata-entry ; -~ The tilde alternate is a very important word for the hex transform; it has -~ the task of implementing a special comment syntax which can be used by words -~ that generate binary output, to define comments which will appear as part of -~ the hex-dump version of that output, but which will be ignored in normal -~ execution. +~ We need to define a few helpers for use in +~ hex-read-magic-comment-introducer, below. The string pointers these return +~ are allocated in the manner described there. ~ -~ This mechanism allows us, for example, to avoid putting detailed knowledge -~ of the amd64 Evocation-assembly instructions in the hex transform; instead -~ all the details are kept in one place, the authoritative implementations of -~ those instructions. This will substantially improve the maintainability of -~ the code, especially when adding new architectures. -: hex-tilde-alternate - ~ The original tilde is already an immediate word, so we can take over - ~ those responsibilities directly without needing to do any extra work to - ~ check the mode we're in or anything like that. +~ (first key -- body-string-pointer) +: hex-read-nonempty-comment-body + ~ This is the rare case in the transforms where we want to get into the + ~ nitty-gritty of lexing. Normally we rely on the upstream implementations, + ~ even when we have to go out of our way to do so, because we don't want to + ~ have to update the transforms every time there's a new syntax feature. + ~ Here, however, we've got a syntax that only has meaning to the transform, + ~ so there's no choice. + swap-transform-variables here @ swap-transform-variables + + ~ This little loop is a modified version of the original tilde loop. + swap { dup dup 0x0a != && } { pack8 key } while drop + + 0 pack8 + 8 packalign + + drop + swap-transform-variables here @ swap-transform-variables ; + +: hex-read-empty-comment-body + swap-transform-variables here @ swap-transform-variables + + 0 pack8 + 8 packalign + drop + swap-transform-variables here @ swap-transform-variables ; + + +~ This is called from a tilde alternate (such as hex-tilde-alternate, but +~ there are several). Its purpose is to do the lexical analysis of the +~ sequence that introduces a magic comment; it assumes that the leading tilde +~ character has just been consumed from the main input, leaving the separating +~ space as the next character ready to be read. +~ +~ Recall that the delimiter for a regular comment is "~ " (the trailing +~ space is part of it for this purpose). The delimiter for a magic comment is +~ "~ : ". A regular comment may also have a linefeed instead of a space, +~ indicating that it's empty; magic comments likewise can be empty, by ending +~ the line immediately after the colon. +~ +~ This word does the necessary reading, then leaves the input pointing at +~ the body of the comment and returns a value indicating what type of comment +~ this is. A result code of 0 indicates it's a regular comment and the input +~ now points immediately after its ending delimiter; that is, to the start of +~ the next line of text. A value of 1 indicates it's a magic comment with +~ non-zero length and the input points immediately after the delimiter. A +~ value of 2 indicates it's a magic comment with zero length, and the input +~ again points to the start of the next line. +~ +~ Our caller will want to do things with the comment body, so we also return +~ a pointer to a string in temporary storage, for use in further processing. +~ This temporary storage is taken from the "real" log, outside the transform's +~ scope. It's not allocated, merely using the space right after "here", +~ because the caller may or may not wish to allocate it for the long term, and +~ may prefer that its long-term location be elsewhere. +~ +~ Some of the various magic-comment syntaxes care about spacing within the +~ comment body, so it's important to notice that only the first space after +~ the colon is consumed. Any subsequent spaces are part of the body. +~ +~ (-- body string pointer, result code) +: hex-read-magic-comment-introducer ~ The original code goes byte-by-byte, checks that the value is nonzero ~ and not equal to 0x0a (linefeed), and exits when either property fails. ~ We want to do something different based on the very first characters of @@ -5189,127 +5240,112 @@ allocate-transformation-state s" transformation-state" variable 0x20 = { ~ The first byte was 0x20. Now check the second. key dup dup 0x0a != && { - 0x3a = { + dup 0x3a = { ~ The second byte was 0x3a. Now check the third. - key dup { + drop key dup { ~ The third byte is not 0. dup 0x20 = { - ~ The special test succeeded, so we want to either compile or - ~ execute a call to hex-tilde-replacement. First, though, we - ~ must save the rest of the comment body so we can provide it to - ~ the replacement at runtime. - ~ - ~ This is the rare case in the transforms where we want to get - ~ into the nitty-gritty of lexing. Normally we rely on the - ~ upstream implementations, even when we have to go out of our - ~ way to do so, because we don't want to have to update the - ~ transforms every time there's a new syntax feature. Here, - ~ however, we've got a syntax that only has meaning to the - ~ transform, so there's no choice. - drop - interpreter-flags @ 0x01 & { - ~ We're in compile mode. - ~ - ~ Fortunately, we can pack the string directly into the - ~ output buffer, so we don't need to mess around with - ~ accumulate-string. - s" lit" find entry-to-execution-token , - has-non-space-this-input-line@ , - s" litstring" find entry-to-execution-token , - here @ - key { dup dup 0x0a != && } { - pack8 key - } while drop - 0 pack8 - 8 packalign here ! - ' hex-tilde-replacement entry-to-execution-token , - } { - has-non-space-this-input-line@ - - swap-transform-variables here @ swap-transform-variables - - key { dup dup 0x0a != && } { pack8 key } while drop - 0 pack8 - 8 packalign - - swap-transform-variables - here @ swap here ! - swap-transform-variables - - transformation-state transformation-state-output-metadata @ - hex-output-metadata-latest-output-point @ - hex-tilde-replacement - ~ TODO this doesn't work yet - ~ the problem with running this in immediate mode during the - ~ label transform is that the latest output point is 0, - ~ because it's running as part of loading the copy of the - ~ compiler that sits directly inside the hex transform, not as - ~ part of reading the code to be compiled, which is inside - ~ the inner transforms. - } if-else + ~ We have a magic comment with a nonempty body. We must save + ~ the rest of it so we can return it. + drop key hex-read-nonempty-comment-body 1 } { ~ The third byte is not 0 or 0x20. - 0x0a = { - ~ The third byte is 0x0a. This is the alternate version of - ~ our special test, denoting a magic comment that happens to - ~ be empty. As with nonempty ones, we pack the string directly - ~ into the output buffer. - s" lit" find entry-to-execution-token , - has-non-space-this-input-line@ , - s" litstring" find entry-to-execution-token , - here @ - 0 pack8 - 8 packalign here ! - ' hex-tilde-replacement entry-to-execution-token , + dup 0x0a = { + ~ We have a magic comment with an empty body, ending at a + ~ linefeed. As with nonempty ones, we prepare a temporary + ~ string to return. + drop hex-read-empty-comment-body 2 } { - ~ The third byte was not 0, 0x0a, or 0x20. So our special - ~ test failed, but the exit condition isn't met. So we're done - ~ unrolling and can just do the original tilde loop for the - ~ rest. - key { dup dup 0x0a != && } { drop key } while drop + ~ The third byte was not 0, 0x0a, or 0x20. We got a space, + ~ a colon, and something else. So our special test failed, but + ~ the exit condition isn't met. So we're done unrolling. + hex-read-nonempty-comment-body 0 } if-else } if-else } { - ~ The exit condition is met, so we're done. - drop + ~ We got a colon and a zero. This is a magic comment with an + ~ empty body, ending at the end of the file. + drop hex-read-empty-comment-body 2 } if-else } { ~ Again, the special test failed but the exit condition isn't met. - key { dup dup 0x0a != && } { drop key } while drop + hex-read-nonempty-comment-body 0 } if-else } { ~ Again, the exit condition is met. - drop + drop hex-read-empty-comment-body 0 } if-else } { ~ Once more, the special test failed but the exit condition isn't met. - key { dup dup 0x0a != && } { drop key } while drop + hex-read-nonempty-comment-body 0 } if-else } { ~ For the last time, the exit condition is met. + drop hex-read-empty-comment-body 0 + } if-else ; + + + +~ The tilde alternate is a very important word for the hex transform; it has +~ the task of hooking comment processing to detect a special comment syntax +~ which can be used by words that generate binary output, to define comments +~ which will appear as part of the hex-dump version of that output, but which +~ will be ignored in normal execution. +~ +~ This mechanism allows us, for example, to avoid putting detailed knowledge +~ of the amd64 Evocation-assembly instructions in the hex transform; instead +~ all the details are kept in one place, the authoritative implementations of +~ those instructions. This will substantially improve the maintainability of +~ the code, especially when adding new architectures. +~ +~ This word hex-tilde-alternate only applies to code that is running +~ directly within the hex transform. When an inner transform runs within the +~ hex transform, the inner transform is responsible for lexical processing, so +~ this word never gets invoked. There is an additional variant word +~ hex-*-tilde-alternate for each inner transform that the hex transform +~ supports, responsible for conducting the analogous task within the rules of +~ that inner transform. +: hex-tilde-alternate + ~ The original tilde is already an immediate word, so we can take over + ~ those responsibilities directly without needing to do any extra work to + ~ check the mode we're in or anything like that. + hex-read-magic-comment-introducer + ~ (body string pointer, result code) + + { + ~ It's a magic comment, which may or may not be empty. We want to either + ~ compile or execute a call to hex-tilde-replacement. + interpreter-flags @ 0x01 & { + ~ We're in compile mode. + ~ + ~ Fortunately, we can pack the string directly into the output buffer, + ~ so we don't need to mess around with accumulate-string. + s" lit" find entry-to-execution-token , + has-non-space-this-input-line@ , + + s" litstring" find entry-to-execution-token , + here @ + swap packstring + 8 packalign + here ! + + ' hex-tilde-replacement entry-to-execution-token , + } { + ~ We're in interpret mode. + has-non-space-this-input-line@ swap + hex-tilde-replacement + } if-else + } { + ~ If it's a regular comment, we just ignore it. drop } if-else - ~ Regardless of what path we exited by, we just consumed a newline, so - ~ make note of the fact. + ~ Regardless of what path we took, we just consumed a newline, so make + ~ note of the fact. 0 has-non-space-this-input-line! ; make-immediate -~ This input helper is a very unusual thing: It's used by hex-transform-one -~ to skip the space between words. That's necessary because we need to keep -~ track of whether we've had any non-blank stuff on each line yet, so that the -~ tilde replacement can figure out certain cases where a magic comment needs -~ to be a suffix comment rather than a line comment. -: hex-skip-space - { peek dup is-space } - { consume - ~ This will set the flag once for each newline, if we have multiple - ~ consecutive ones or something like that, but that's harmless. - 0x0a = { 0 has-non-space-this-input-line! } if - } while drop ; - - : hex-label-word-replacement word @@ -5358,6 +5394,21 @@ allocate-transformation-state s" transformation-state" variable } if-else ; make-immediate +~ This input helper is a very unusual thing: It's used by hex-transform-one +~ to skip the space between words. That's necessary because we need to keep +~ track of whether we've had any non-blank stuff on each line yet, so that the +~ tilde replacement can figure out certain cases where a magic comment needs +~ to be a suffix comment rather than a line comment. +: hex-skip-space + { peek dup is-space } + { consume + ~ This will set the flag once for each newline, if we have multiple + ~ consecutive ones or something like that, but that's harmless. + 0x0a = { 0 has-non-space-this-input-line! } if + } while drop ; + + + ~ This implements the hex transform for a single word. It is directly ~ analogous to "interpret", and reading interpret.e may help in understanding ~ it, though it's meant to still make sense on its own. |