add a constant string "boot-source" which is run at startup

this will allow us to have compiled-in code that wouldn't run under the transforms, though we don't add anything of substance just yet

this strategy was used early in bring-up, in the historical flatassembler version of Evocation, so the design space of it is already well-explored in Irenes' head. how pleasant. :)

Force-Push: yes
Change-Id: I722cddb77657df9ca518c0d9841aa587bf63feb3

5 files changed, 193 insertions, 26 deletions

diff --git a/README.txt b/README.txt
index ce7a254..8657dc6 100644
--- a/README.txt
+++ b/README.txt
@@ -245,10 +245,10 @@ expectations on yourself around understanding the transformation facility.
 
   It's okay, you can benefit from it before you understand it: Transformation
 provides the core tricks that make it possible to compile Forth code into
-standalone executables. The call to label-transform in evoke.e, and the call
-to log-load-transform in compilation/execution.e, are the two spots where
-compilation is handed off to the transformation facility, and you can pretty
-much just take it for granted that it works, until you feel ready.
+standalone executables. The calls to label-transform and log-load-transform in
+evoke.e are the two spots where compilation is handed off to the
+transformation facility, and you can pretty much just take it for granted that
+it works, until you feel ready.
 
   If you want examples of programs that are smaller than Evocation itself,
 quine.e is a tiny program written in proper Evocation that outputs its own
diff --git a/compilation/execution.e b/compilation/execution.e
index ecf5ec4..e2cb2b2 100644
--- a/compilation/execution.e
+++ b/compilation/execution.e
@@ -419,9 +419,15 @@
 ~ pointers. It was directly jumped to from cold-start. There's nowhere to
 ~ return to, so it needs to never return.
 ~
-~ (output buffer start, output point, input string pointer
-~  -- output buffer start, output point)
-: output-warm-start
+~   There are a few optional pieces of the warm-start routine, so here in the
+~ compile-time code that builds it, we break the routine into logical pieces
+~ which are assembled by calling the appropriate helpers sequentially.
+~
+~   Outputting the warm-start routine should always begin by calling
+~ output-warm-start-header.
+~
+~ (output buffer start, output point -- output buffer start, output point)
+: output-warm-start-header
   ~ : blank-line
   ~ : blank-line
   ~ : ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
@@ -453,7 +459,6 @@
   ~   While it's not actually a requirement that codeword pointers be
   ~ word-aligned, it's highly likely that it helps performance. (Whether it
   ~ does is up to Intel's microcode.)
-  3unroll
   8 packalign
   current-offset L!' warm-start
   ~ (input string pointer, output buffer start, output point)
@@ -550,23 +555,107 @@
   ~ : -8 log-load-variable #-- (codeword pointer)
 
   ~   Having done that, nothing else needs to be defined in an unusual way, so
-  ~ we can go ahead and dispatch to the log-load transform, and do the rest of
-  ~ the code through that.
+  ~ at this point the warm-start routine can go ahead and dispatch to the
+  ~ log-load transform, and do the rest of the code through that. However, to
+  ~ allow customization, we here in output-warm-start will let our caller take
+  ~ responsibility for making sure that happens.
+  ;
 
-  ~ (input string pointer, output buffer start, output point)
-  3roll log-load-transform
-  ~ (output buffer start, output point)
 
+~   This helper doesn't have any actual compile-time responsibilities, it
+~ exists solely for the sake of the magic comments.
+~
+~   We could have it be responsible for calling log-load-transform, but
+~ honestly it's nicer letting the caller be responsible for that, since the
+~ caller is also responsible for calling label-transform.
+~
+~ (output buffer start, output point -- output buffer start, output point)
+: output-warm-start-post-log-load-transform
   ~ : blank-line
   ~ :   We're now done with the log-load routine; the next little bit of code
   ~ : here is the final bit of the warm-start routine.
+  ;
+
 
-  ~   Now everything we need has been added to the log and we're almost ready
-  ~ to jump into it. It's inconvenient for code under the log-load transform
-  ~ to interact with the label system, so if we want to point
-  ~ main-input-buffer at a string, this is the place to do it.
 
-  ~ The next layer is built now, so let's move on to it.
+~   After the log-load routine finishes, everything we need has been added to
+~ the log and we're almost ready to jump into dynamic code running from the
+~ log. That code still needs main-input-buffer to be set up in some way. The
+~ simplest approach is to attach it directly to standard input. This helper
+~ outputs a snippet that does that, structured to be used as part of the
+~ warm-start routine.
+~
+~   Executables that don't call output-warm-start-attach-stdin will need to
+~ instead call output-warm-start-attach-boot-source. Exactly one of these must
+~ be used.
+~
+~ (output buffer start, output point -- output buffer start, output point)
+: output-warm-start-attach-stdin
+  L@' litstring offset-to-target-address-space pack64
+  ~ : -8 litstring #-- (codeword pointer)
+  s" relink-main-input-buffer-to-stdin" packstring 8 packalign
+  L@' log-load-find-execution-token offset-to-target-address-space pack64
+  ~ : -8 log-load-find-execution-token #-- (codeword pointer)
+  L@' execute offset-to-target-address-space pack64
+  ~ : -8 execute #-- (codeword pointer)
+  ;
+
+
+~   After the log-load routine finishes, everything we need has been added to
+~ the log and we're almost ready to jump into dynamic code running from the
+~ log. It's inconvenient for code under the log-load transform to interact
+~ with the label system, so if we want to point main-input-buffer at a string,
+~ that's the place to do it.
+~
+~   That's optional functionality, which any given executable may or may not
+~ want to use, so it's here in this helper. If your executable is built with
+~ a call to output-warm-start-attach-boot-source, it must also at some point
+~ include a call to output-boot-source.
+~
+~ (output buffer start, output point, input string pointer
+~  -- output buffer start, output point)
+: output-warm-start-attach-boot-source
+  L@' litstring offset-to-target-address-space pack64
+  ~ : -8 litstring #-- (codeword pointer)
+  s" main-input-buffer" packstring 8 packalign
+  L@' log-load-find-execution-token offset-to-target-address-space pack64
+  ~ : -8 log-load-find-execution-token #-- (codeword pointer)
+  L@' execute offset-to-target-address-space pack64
+  ~ : -8 execute #-- (codeword pointer)
+  L@' lit offset-to-target-address-space pack64
+  ~ : -8 lit #-- (codeword pointer)
+  L@' boot-source offset-to-target-address-space pack64
+  ~ : -8 boot-source #-- (data pointer)
+  L@' 3roll offset-to-target-address-space pack64
+  ~ : -8 3roll #-- (codeword pointer)
+  L@' litstring offset-to-target-address-space pack64
+  ~ : -8 litstring #-- (codeword pointer)
+  s" attach-string-to-input-buffer" packstring 8 packalign
+  L@' log-load-find-execution-token offset-to-target-address-space pack64
+  ~ : -8 log-load-find-execution-token #-- (codeword pointer)
+  L@' swap offset-to-target-address-space pack64
+  ~ : -8 swap #-- (codeword pointer)
+  L@' lit offset-to-target-address-space pack64
+  ~ : -8 lit #-- (codeword pointer)
+  4 ~ : provide-hex
+  pack64
+  ~ : -8 # #-- (integer literal)
+  L@' unroll offset-to-target-address-space pack64
+  ~ : -8 unroll #-- (codeword pointer)
+  L@' execute offset-to-target-address-space pack64
+  ~ : -8 execute #-- (codeword pointer)
+  ;
+
+
+~   Once the log-load routine has been run and main-input-buffer has been
+~ prepared, the next layer is built. This helper outputs the final little bit
+~ of the warm-start routine, which moves on to that layer.
+~
+~   Outputting the warm-start routine should always end by calling
+~ output-warm-start-footer.
+~
+~ (output buffer start, output point -- output buffer start, output point)
+: output-warm-start-footer
   L@' litstring offset-to-target-address-space pack64
   ~ : -8 litstring #-- (codeword pointer)
   s" quit" packstring 8 packalign
@@ -583,6 +672,67 @@
   ;
 
 
+~   This helper generates a single word entry named "boot-source", flagged as
+~ hidden, and outputs a string as its contents. This string is meant to be
+~ used by the code generated by output-warm-start-attach-boot-source, and will
+~ have no significant effects otherwise.
+~
+~   The helper is expecting to be called at a point where the generated output
+~ already has a dictionary, and isn't in the middle of generating any
+~ routines. The usual place to call it is immediately after the label
+~ transform.
+~
+~ (output buffer start, output point, previous entry pointer,
+~  input string pointer -- output buffer start, output point)
+: output-boot-source
+  ~ : blank-line
+  ~ : blank-line
+  ~ : ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
+  ~ :  ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
+  ~ : ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
+  ~ : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+  ~ : blank-line
+  ~ :   This is the start of the boot-source, which is Forth code included
+  ~ : as text, and invoked by the warm-start routine. At the time it runs,
+  ~ : the warm-start routine and the log-load routine have both completed and
+  ~ : the Forth execution environment is fully ready.
+  ~ :
+  ~ :   The routine is stored as a dictionary entry with the "hidden" flag
+  ~ : set. It has a regular entry header up to, but not including, the
+  ~ ; codeword. Then it has text, with a null terminator and alignment
+  ~ : padding at the end.
+  ~ :
+  ~ :   Unfortunately, this winds up being somewhat hard to read in the hex
+  ~ : dump. Sorry about that.
+  ~ : blank-line
+  ~ : indent
+  3unroll pack64 ~ : 8 #-- (previous entry pointer)
+  ~ : 1 #-- (entry flags)
+  0x80 pack8
+  ~ : 2 bidirectional-null-terminated entry name
+  0 pack8
+  ~ (output buffer start, input string pointer, output point)
+
+  s" boot-source"
+  ~   We do a spurious stringlen just so the magic comments can see the
+  ~ string's length and use it for an appropriate adjustment.
+  dup stringlen
+  ~ : provide-data
+  ~ : data-adjust-length
+  drop
+  ~ : 1 suppress
+  packstring
+  8 packalign
+  ~ : fresh-line
+  ~ (output buffer start, input string pointer, output point)
+
+  dup 3 pick - L!' boot-source
+
+  swap packstring 8 packalign
+  ~ : deindent
+  ;
+
+
 ~ Where next?
 ~ ~~~~~~~~~~~
 ~
diff --git a/compilation/transform.e b/compilation/transform.e
index 80ef98b..5b84009 100644
--- a/compilation/transform.e
+++ b/compilation/transform.e
@@ -1447,7 +1447,7 @@ allocate-transformation-state s" transformation-state" variable
 ~ more complex.
 ~
 ~ (output buffer start, output point, input string pointer
-~  -- output buffer start, output point)
+~  -- output buffer start, output point, final entry pointer)
 : label-transform
   ~ : blank-line
   ~ : blank-line
@@ -1499,12 +1499,13 @@ allocate-transformation-state s" transformation-state" variable
 
     ~  When the loop is done, get the real values of "here" and "latest"
     ~ back. The internal "here" is also the output point, and will become our
-    ~ return value. The internal "latest" is discarded.
+    ~ return value. The internal "latest" is returned as a result.
     { transformation-state transformation-state-output-buffer-start @
       here @
+      latest @ 3unroll
       transformation-state transformation-state-saved-here @ here !
       transformation-state transformation-state-saved-latest @ latest !
-      ~ (output buffer start, output point)
+      ~ (final entry pointer, output buffer start, output point)
 
       ~   Though we don't actually use transformation-state outside of this
       ~ invocation, for tidiness we zero it out.
@@ -1515,6 +1516,9 @@ allocate-transformation-state s" transformation-state" variable
       ~  Also put the input source back how it was.
       main-input-buffer pop-input-buffer
 
+      3roll 2 pick -
+      ~ (output buffer start, output point, final entry offset)
+
       exit } if } forever ;
 
 
diff --git a/evoke.e b/evoke.e
index 1b1453b..cade5e4 100644
--- a/evoke.e
+++ b/evoke.e
@@ -30,7 +30,10 @@ s" compilation/execution.e" include
   s" language/dynamic/linux.e" pack-file-contents
   s" language/dynamic/files.e" pack-file-contents
   0 pack8 8 packalign here !
-  s" source-to-copy-to-log" variable ;
+  s" source-to-copy-to-log" variable
+
+  s" relink-main-input-buffer-to-stdin"
+  s" source-to-textually-include" variable ;
 read-inputs
 
 
@@ -56,8 +59,15 @@ read-inputs
   elf-program-header
 
   output-cold-start
-  source-to-copy-to-log output-warm-start
+
+  output-warm-start-header
+  source-to-copy-to-log log-load-transform
+  output-warm-start-post-log-load-transform
+  output-warm-start-attach-boot-source
+  output-warm-start-footer
+
   source-to-precompile label-transform
+  source-to-textually-include output-boot-source
 
   ~   If we wanted words in the log to be able to call statically-linked
   ~ words, we could set this to something nonzero. We don't, so we leave it
diff --git a/language/dynamic/interpret.e b/language/dynamic/interpret.e
index 0af0dab..02f3365 100644
--- a/language/dynamic/interpret.e
+++ b/language/dynamic/interpret.e
@@ -593,7 +593,10 @@ dup entry-flags@ 0x80 invert & swap entry-flags!
   main-input-buffer input-buffer-refill !
   ;
 
-~   Of course it is perfectly reasonable to change this, but for now it's
-~ hardcoded.
-relink-main-input-buffer-to-stdin
+~   We don't actually call relink-main-input-buffer-to-stdin here. It would
+~ work if we did (the log-load transform is capable of making that happen),
+~ but some executables will wish to use it during warm-start while others will
+~ prefer to first attach input to a string, often called boot-source, and only
+~ later relink it. So, we leave those calls to helpers in
+~ compilation/execution.e.