create a special comment syntax for the hex transform's use

right now all we do is recognize the special comments and save their body text for later handling. in a future CL, hex-tilde-replacement will turn them into output metadata entries.

Change-Id: I53cc9d0cd781c2fc43af180ae98606efc14c3147
Force-Push: yes

1 files changed, 120 insertions, 4 deletions

diff --git a/transform.e b/transform.e
index 5475db4..1598d19 100644
--- a/transform.e
+++ b/transform.e
@@ -2828,9 +2828,6 @@ allocate-transformation-state s" transformation-state" variable
   0 swap ! ~ End-of-entries delimiter
   drop ;
 
-: hex-tilde-alternate [ ' ~ entry-to-execution-token , ]
-  ; make-immediate
-
 : hex-self-codeword-alternate self-codeword ;
 
 : hex-string-alternate
@@ -3352,6 +3349,10 @@ allocate-transformation-state s" transformation-state" variable
 ~ hex-colon-alternate prepends its stuff first and then
 ~ hex-semicolon-alternate slides that all forward to prepend the call to
 ~ hex-trace.
+~
+~ TODO this probably doesn't serve a purpose in the end, and should go away,
+~ but the actual stack-walking code is useful and should be kept until it's
+~ been refactored into something worth keeping.
 : hex-trace
   exit
   ~ value@ here @ swap 0 hex-output-metadata-entry-type-line-comment
@@ -3608,6 +3609,121 @@ allocate-transformation-state s" transformation-state" variable
   } if-else ; make-immediate
 
 
+~   Similar to how we handle "allocate" as a replacement which is installed by
+~ an alternate, we do the same for tilde, but for a different reason. The
+~ semantics of ordinary, non-transformed tilde skip over comment text while
+~ compiling, and discard it. We need to do processing of that text later, when
+~ the comment's position would be reached in running the word it's part of, so
+~ the alternate compiles it appropriately.
+~
+~   Importantly, the replacement is only used for comments that have a special
+~ comment syntax unique to the hex transform, as described in more detail in
+~ the alternate, below. If it were used for all comments, it would interfere
+~ with the distance of branches, which could get quite complex to work around.
+~ It is the responsibility of words that use the hex-transform comment syntax
+~ to avoid placing such comments within the scope of a branch. This applies
+~ only to explicit, precomputed branches using the words "branch" and
+~ "branch0". Fortunately, implicit branches that are part of high-level
+~ flow-control words such as "if" and "while" will work fine, since the
+~ substitution occurs before the flow control words start sliding code around,
+~ and it updates "here" appropriately.
+~
+~ (string pointer)
+: hex-tilde-replacement
+  ." hmmm " emitstring newline ;
+
+
+~   The tilde alternate is a very important word for the hex transform; it has
+~ the task of implementing a special comment syntax which can be used by words
+~ that generate binary output, to define comments which will appear as part of
+~ the hex-dump version of that output, but which will be ignored in normal
+~ execution.
+~
+~   This mechanism allows us, for example, to avoid putting detailed knowledge
+~ of the amd64 Evocation-assembly instructions in the hex transform; instead
+~ all the details are kept in one place, the authoritative implementations of
+~ those instructions. This will substantially improve the maintainability of
+~ the code, especially when adding new architectures.
+: hex-tilde-alternate
+  ~   The original tilde is already an immediate word, so we can take over
+  ~ those responsibilities directly without needing to do any extra work to
+  ~ check the mode we're in or anything like that.
+
+  ~   The original code goes byte-by-byte, checks that the value is nonzero
+  ~ and not equal to 0x0a (linefeed), and exits when either property fails.
+  ~ We want to do something different based on the very first characters of
+  ~ the comment body. So, we unroll the first few iterations of the loop so we
+  ~ can do that test before we fall back to the normal behavior.
+  ~
+  ~   Specifically, we want to recognize a colon (0x3a) with a space (0x20)
+  ~ immediately after it. Since the interpeter peeked at the space delimiter,
+  ~ but didn't consume it, we also need to check for a space before the colon,
+  ~ for a total sequence of three bytes we need to match. We can't just skip
+  ~ over the space with "consume", because it might also be a linefeed, which
+  ~ would satisfy the exit criterion. We need to properly match it.
+  key dup dup 0x0a != && {
+    ~ If we got here, the first byte was not 0 or 0x0a. Now we check if it's
+    ~ colon.
+    0x20 = {
+      ~ The first byte was 0x20. Now check the second.
+      key dup dup 0x0a != && {
+        0x3a = {
+          ~ The second byte was 0x3a. Now check the third.
+          key dup dup 0x0a != && {
+            0x20 = {
+              ~   The special test succeeded, so we want to insert a call to
+              ~ hex-tilde-replacement. First, though, we must save the rest of
+              ~ the comment body so we can provide it to the replacement at
+              ~ runtime.
+              ~
+              ~   This is the rare case in the transforms where we want to get
+              ~ into the nitty-gritty of lexing. Normally we rely on the
+              ~ upstream implementations, even when we have to go out of our
+              ~ way to do so, because we don't want to have to update the
+              ~ transforms every time there's a new syntax feature. Here,
+              ~ however, we've got a syntax that only has meaning to the
+              ~ transform, so there's no choice.
+              ~
+              ~   Fortunately, we can pack the string directly into the output
+              ~ buffer, so we don't need to mess around with
+              ~ accumulate-string.
+              s" litstring" find entry-to-execution-token ,
+              here @
+              key { dup dup 0x0a != && } {
+                pack8 key
+              } while drop
+              0 pack8
+              8 packalign here !
+              ' hex-tilde-replacement entry-to-execution-token ,
+            } {
+              ~   The third byte was not 0, 0x0a, or 0x20. So our special
+              ~ test failed, but the exit condition isn't met. So we're done
+              ~ unrolling and can just do the original tilde loop for the
+              ~ rest.
+              key { dup dup 0x0a != && } { drop key } while drop
+            } if-else
+          } {
+            ~ The exit condition is met, so we're done.
+            drop
+          } if-else
+        } {
+          ~ Again, the special test failed but the exit condition isn't met.
+          key { dup dup 0x0a != && } { drop key } while drop
+        } if-else
+      } {
+        ~ Again, the exit condition is met.
+        drop
+      } if-else
+    } {
+      ~ Once more, the special test failed but the exit condition isn't met.
+      key { dup dup 0x0a != && } { drop key } while drop
+    } if-else
+  } {
+    ~ For the last time, the exit condition is met.
+    drop
+  } if-else ; make-immediate
+
+
 ~   This implements the hex transform for a single word. It is directly
 ~ analogous to "interpret", and reading interpret.e may help in understanding
 ~ it, though it's meant to still make sense on its own.
@@ -3660,7 +3776,6 @@ allocate-transformation-state s" transformation-state" variable
   ~ the alternate if so.
   0 swap
   ~ (name as stack string, placeholder, name pointer)
-  dup s" ~" stringcmp 0 = { swap drop ' hex-tilde-alternate swap } if
   dup s" self-codeword" stringcmp 0 = {
     swap drop ' hex-self-codeword-alternate swap } if
   ~ It is nontrivial to construct a string with a double-quote in it.
@@ -3683,6 +3798,7 @@ allocate-transformation-state s" transformation-state" variable
   dup s" bye" stringcmp 0 = { swap drop ' hex-bye-alternate swap } if
   dup s" allocate" stringcmp 0 = {
     swap drop ' hex-allocate-alternate swap } if
+  dup s" ~" stringcmp 0 = { swap drop ' hex-tilde-alternate swap } if
   ~ (name as stack string, 0 or alternate entry pointer, name pointer)
 
   find