refactor the compilation harness

it's more declarative now. however, at the moment it only produces the one type of executable, so it's only for building "evoke", not "hex". hopefully that can be fixed soon.

Change-Id: Id970b9952fcd73ba6da4fa503ae2f784afe1b8a7
Force-Push: yes

1 files changed, 237 insertions, 0 deletions

diff --git a/compilation/top-level.e b/compilation/top-level.e
new file mode 100644
index 0000000..959711b
--- /dev/null
+++ b/compilation/top-level.e
@@ -0,0 +1,237 @@
+~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~ ~~ Top-level executable builders ~~
+~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~
+~   Although it's perfectly possible to manually invoke the pieces of the
+~ compiler, it's inconvenient to do so. This file has helper words which kick
+~ off the process of compilation from the top level.
+~
+~   The two most important words in the top-level facility are "enchant",
+~ which produces an executable binary, and "inscribe", which produces a
+~ commented hex dump corresponding to that same binary.
+~
+~   Significant effort has been expended to make sure that the interfaces to
+~ "enchant" and "inscribe" are the same, despite them notionally existing at
+~ different levels of abstraction.
+
+
+~   This accepts a pointer to an existing buffer for file contents to be
+~ appended to, and another pointer to the start of a packed list of filenames.
+~ The filename list consists of null-terminated strings, all directly abutting
+~ each other in memory, terminated by a zero-length string (which can also
+~ be thought of as a freestanding null byte).
+~
+~   It reads the contents of the named files into the buffer, with no
+~ separation between them, and returns a pointer to the end of the contents.
+~
+~ (buffer point, filename list point -- buffer point)
+: read-inputs-from-filename-list
+  ~ (buffer point, filename list point)
+  { dup 8@ } {
+    dup 3roll swap pack-file-contents swap
+    dup stringlen 1+ +
+  } while drop ;
+
+
+~ (static source filename list, dynamic source filename list, boot source
+~  filename list -- source to precompile, source to copy to log,
+~  source to copy textually)
+: read-inputs
+  here @ dup
+  ~ (static source filename list, dynamic source filename list,
+  ~  boot source filename list, source to precompile, buffer point)
+  4 pick read-inputs-from-filename-list
+  0 pack8 8 packalign here !
+  ~ (static source filename list, dynamic source filename list,
+  ~  boot source filename list, source to precompile)
+
+  here @ dup
+  ~ (static source filename list, dynamic source filename list,
+  ~  boot source filename list, source to precompile, source to copy to log,
+  ~  buffer point)
+  5 pick read-inputs-from-filename-list
+  4 pick read-inputs-from-filename-list
+  0 pack8 8 packalign here !
+  ~ (static source filename list, dynamic source filename list,
+  ~  boot source filename list, source to precompile, source to copy to log)
+
+  here @ dup
+  ~ (static source filename list, dynamic source filename list,
+  ~  boot source filename list, source to precompile, source to copy to log,
+  ~  source to copy textually, buffer point)
+  4 pick read-inputs-from-filename-list
+  0 pack8 8 packalign here !
+  ~ (static source filename list, dynamic source filename list,
+  ~  boot source filename list, source to precompile, source to copy to log,
+  ~  source to copy textually)
+
+  4 roll drop 4 roll drop 4 roll drop ;
+
+
+~ (output memory start, current output point,
+~  compile-header execution token, compile-footer execution token,
+~  source to precompile, source to copy to log, source to include textually
+~  -- output memory start, current output point)
+: output-contents
+  7 roll 7 roll
+  ~ (compile-header execution token, compile-footer execution token,
+  ~  source to precompile, source to copy to log, source to include textually,
+  ~  output memory start, current output point)
+
+  ~ Invoke the provided header callback.
+  7 roll execute
+  ~ (compile-footer execution token, source to precompile,
+  ~  source to copy to log, source to include textually, output memory start,
+  ~  current output point)
+
+  elf-file-header
+  elf-program-header
+
+  output-cold-start
+
+  ~ Warm start and the log-loaded source.
+  output-warm-start-header
+  4 roll log-load-transform
+  ~ (compile-footer execution token, source to precompile,
+  ~  source to include textually, output memory start, current output point)
+  output-warm-start-post-log-load-transform
+  output-warm-start-attach-boot-source
+  output-warm-start-footer
+
+  ~ The label-transformed source.
+  4 roll label-transform
+  ~ (compile-footer execution token, source to include textually,
+  ~  output memory start, current output point, final entry pointer)
+
+  ~ The boot source.
+  4 roll output-boot-source
+  ~ (compile-footer execution token, output memory start,
+  ~  current output point)
+
+  ~ Invoke the provided footer callback.
+  3roll execute
+  ~ (output memory start, current output point)
+  ;
+
+
+~   Given a description of an executable, compile and output that executable
+~ as runnable binary, to standard output.
+~
+~ (static source filename list, dynamic source filename list,
+~  boot source filename list, compile-header word name,
+~  compile-footer word name --)
+: enchant-from-description
+  5 roll 5 roll 5 roll read-inputs
+  ~ (compile-header word name, compile-footer word name, source to precompile,
+  ~  source to copy to log, source to copy textually)
+
+  ~   We need an execution token to pass to label-loop. There's no reason to
+  ~ create a dictionary entry for it; we just allocate space for the
+  ~ executable word body directly.
+  here @
+  docol ,
+  6 roll find entry-to-execution-token literal
+  5 roll find entry-to-execution-token literal
+  4 roll literal
+  3 roll literal
+  swap literal
+  ' output-contents entry-to-execution-token ,
+  ' exit entry-to-execution-token ,
+
+  label-loop
+  1 3unroll sys-write
+  bye ;
+
+
+~   Given a filename containing the top-level source of an executable, compile
+~ and output that executable as runnable binary, to standard output.
+~
+~ (top-level source filename --)
+: enchant
+  include drop
+  s" latest-executable-data" find entry-to-execution-token execute
+  enchant-from-description ;
+
+
+~   Given a filename containing the top-level source of an executable, compile
+~ and output that executable as a commented hex dump, to standard output.
+~ Processing the hex dump through "hex" will give an identical binary to what
+~ "enchant" would have produced, with the same inputs.
+~
+~   Please notice that if the thing we're compiling is "evoke" and its list
+~ of source files has changed, we'll be working with the new list, not
+~ the old list. This is the same as what "enchant" does, and it's strictly
+~ preferable behavior. We match the behavior of enchant as closely as
+~ possible in terms of what's taken from the host compiler vs. the compiler
+~ being compiled; see the description of "enchant", above, for more details
+~ and discussion of the implications.
+~
+~ (top-level source filename --)
+: inscribe
+  dup here @ = { allocate-string } if
+  dup include drop
+  s" latest-executable-data" find entry-to-execution-token execute
+  ~ (top-level source filename, static source filename list,
+  ~  dynamic source filename list, boot source filename list,
+  ~  compile-header word name, compile-footer word name --)
+
+  ~   We begin by constructing the full text of all the source that will be
+  ~ run inside the scope of the hex transform. That necessarily includes not
+  ~ only the compilation machinery, but the transformation system. Notably
+  ~ it does not need to include "inscribe", but does need to include
+  ~ "enchant", and having a stray unused copy of inscribe is harmless, so we
+  ~ don't try to exclude it.
+  here @ dup
+  ~ (top-level source filename, static source filename list,
+  ~  dynamic source filename list, boot source filename list,
+  ~  compile-header word name, compile-footer word name,
+  ~  hex-transformed source pointer, output point)
+
+  6 pick read-inputs-from-filename-list
+  5 pick read-inputs-from-filename-list
+  4 pick read-inputs-from-filename-list
+  8 roll pack-file-contents
+
+  ~   We need to create a call to "enchant" in the code under transformation.
+  ~ That call doesn't need to be arrived at in exactly the same way as an
+  ~ independent call would be, since it doesn't become part of the output, it
+  ~ only needs to have the same effect.
+  ~
+  ~   So, we can take the filename list pointers that will become parameters
+  ~ to "enchant" and pass them in directly, heedless of the fact they
+  ~ reference locations in the outer log.
+  s"  0x" pack-raw-string 7 roll pack-hex64
+  s"  0x" pack-raw-string 6 roll pack-hex64
+  s"  0x" pack-raw-string 5 roll pack-hex64
+  ~ (compile-header word name, compile-footer word name,
+  ~  hex-transformed source pointer, output point)
+
+  ~   We do not have that luxury with the two compile hooks; we need copies of
+  ~ them that are under the hex transform. We produce code that looks them up
+  ~ by name. Notice that this is possible because we included the top-level
+  ~ source in our full source text.
+  ~
+  ~   Outputting quotation marks is, as always, a pain.
+  s"  s" pack-raw-string 0x22 pack8 s"  " pack-raw-string
+  4 roll pack-raw-string
+  0x22 pack8
+  s"  allocate-string" pack-raw-string
+
+  s"  s" pack-raw-string 0x22 pack8 s"  " pack-raw-string
+  3roll pack-raw-string
+  0x22 pack8
+  s"  allocate-string" pack-raw-string
+  ~ (hex-transformed source pointer, output point)
+
+  s"  enchant-from-description" pack-raw-string
+  ~ (hex-transformed source pointer, output point)
+  0 pack8 8 packalign here !
+
+  ~ (hex-transformed source pointer)
+  1024 1024 4 * * allocate
+  1024 1024 8 * * allocate
+  ~ (hex-transformed source pointer, binary buffer pointer,
+  ~  metadata buffer pointer)
+
+  swap dup 4 roll hex-transform bye ;
+