Making an Editor

Recently, I was working on tools, considering how I was going to improve my theorem-proving experience with Coq. I’m an Atom user. I like it quite a lot. And before I liked Atom, I liked LightTable (I still do, to an extent). So naturally my first instinct was to write a plugin for Atom. I’d never done it before, but hey no time like the present, right?

I started writing a plugin for Atom and immediately hated the experience. I could go off on a rant about Javascript and the many reasons I find it distasteful, but that’s not what this post is about. I’ll just say I think Javascript is showing its age and it’s not my cup of tea. I didn’t want to be spending my time learning Javascript and Node.js when I could be learning about a language I liked and was likely to want to use in the future.

I suppose I could have tried writing my plugin for LightTable. Clojurescript isn’t half bad, and there are many things to like about that project. But a thought occurred to me. Why just write a plugin? Why not make an editor? It was a good idea, and it kills several birds with one stone. Most notably, I need to make some shitty pots.

I chose to write an editor in Racket. I’d like to share my journey so far. Perhaps I’ll share the rest as I continue forward. I’ve learned some very interesting things! As for now, by the end of this post, we will have a fully-functional source code editor that is capable of not only editing its own source code, but also of reloading itself dynamically to reflect those changes to its code!

All code developed in these posts shall be licensed under the terms of the GNU General Public License, version 3. The full text of that license can be found here.

The Simplest Editor in Racket

Racket has a GUI library, unimaginitively but helpfully named racket/gui. Have a peek at its excellent documentation. It has an editor widget built-in. Let’s use it to build the world’s simplest editor. After a raco pkg new and a bit of typing, I have a simple main.rkt:

#lang racket/base

;; Scrividh is a text editor with some IDE features.
;; Copyright (C) 2017  Lucas Adam Michael Paul
;;
;; This program is free software: you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.
;;
;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.
;;
;; You should have received a copy of the GNU General Public License
;; along with this program.  If not, see <http://www.gnu.org/licenses/>.

(module+ test
  (require rackunit))

(require "editor-main.rkt")

(module+ test
  ;; Tests to be run with raco test
  )

(module+ main
  ;; Main entry point, executed when run with the `racket` executable or DrRacket.
  (editor-main)
  )

I want the actual code of the editor itself to be in a separate file. It’ll make some things easier later – particularly refactoring. So all functionality is delegated to editor-main in editor-main.rkt:

#lang racket/gui

;; Scrividh is a text editor with some IDE features.
;; Copyright (C) 2017  Lucas Adam Michael Paul
;;
;; This program is free software: you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.
;;
;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.
;;
;; You should have received a copy of the GNU General Public License
;; along with this program.  If not, see <http://www.gnu.org/licenses/>.

(provide editor-main)

(define (editor-main)
  (define f (new frame%
                 [label "Scrividh"]
                 [width 200]
                 [height 200]))
  (define c (new editor-canvas% [parent f]))
  (define t (new text%))
  (send c set-editor t)
  (send f show #t))

It’s mostly copyright notice. Sorry about that. I’ll fold it in future code blocks. (Caution: do not run if you don’t have an external way to kill or close the program. It has no “Quit” button yet!) But look! It runs!

The world’s simplest editor

This much anyone could have done by looking no farther than a page into the Editor documentation. In fact, they could easily get copy/paste working by reading just a few lines further. But even that isn’t really very good. After all, this editor cannot even load or save files!

Let’s fix that. The racket/gui library makes this easy. I’ll just show the editor-main function, since that’s all that changed:

(define (editor-main)
  (define f (new frame%
                 [label "Scrividh"]
                 [width 200]
                 [height 200]))
  (define c (new editor-canvas% [parent f]))
  (define t (new text%))
  (define mb (new menu-bar% [parent f]))
  (define m-file (new menu% [label "File"] [parent mb]))
  (define m-edit (new menu% [label "Edit"] [parent mb]))
  (define mi-open
    (new menu-item%
         [label "Open"]
         [parent m-file]
         [callback
          (λ (i e)
            (define path (get-file #f f))
            (when path
              (send t load-file path 'text)))]
         [shortcut #\o]
         [shortcut-prefix '(ctl)]))
  (define mi-save
    (new menu-item%
         [label "Save"]
         [parent m-file]
         [callback
          (λ (i e)
            (send t save-file #f 'text))]
         [shortcut #\s]
         [shortcut-prefix '(ctl)]))
  (append-editor-operation-menu-items m-edit #f)
  (send t set-max-undo-history 100)
  (send c set-editor t)
  (send f show #t))

We add menu items for Open and Save, and give them callbacks that ask the text% object to do the actual work of opening and saving files. Adding menu items even has a way of giving us keyboard shortcuts, which is just beautiful. This is more like it!

An actual text editor!

We can actually use this to edit its own source code! It’s ugly, and doesn’t do syntax highlighting or proper indentation… yet. Let’s fix that RIGHT NOW!

As it happens, there is another library we can leverage for this. It’s called framework, and its documentation is here. It’s maybe not quite as polished as racket/gui’s, but still very good. It has a racket:text% class that will do nicely as a drop-in replacement for text%. I won’t show the whole code again, because there are only two small changes: Adding (require framework) before the provide, and replacing text% with racket:text% on line 27.

A source-code editor!

Wow. It is positively mind-blowing how much we can do in less than 100 lines of Racket. Granted, this is standing on the shoulders of geniuses who wrote some fantastic library code, which does far more than it has any business doing. Also, there are some serious limitations to how far we can keep going in this direction without starting to write some brilliant (and lengthy) code of our own.

But there’s still something we can do before we wrap up this post. Something I promised early on. I haven’t forgotten. There are still a couple of serious flaws with this program. First of all, it still doesn’t have a quit button. Second of all, it’s really irritating that you have to quit and then restart the editor every time you want to test a change. What’s the point of being able to edit your own source code if you have to restart every time you save?

Nano-Incremental Iteration

Racket has a cool little function hidden away in the less-utilized corners of its library: dynamic-rerequire. That’s not a typo. There are two “re”s in there. Yes, dynamic-require is a thing. I’m talking about its lesser-known partner. Let’s have a look at the documentation:

Like (dynamic-require module-path 0), but with reloading support. The dynamic-rerequire function is intended for use in an interactive environment, especially via enter!.

If invoking module-path requires loading any files, then modification dates of the files are recorded. If the file is modified, then a later dynamic-rerequire re-loads the module from source; see also Module Redeclarations. Similarly if a later dynamic-rerequire transitively requires a modified module, then the required module is re-loaded. Re-loading support works only for modules that are first loaded (either directly or indirectly through transitive requires) via dynamic-rerequire.

… ooOOOOOOOOOOOOOOOOOOOOOooooooooooooooooh! That sounds like fun. It’s time to rewrite main.rkt a bit. Get rid of (require "editor-main.rkt"), and replace it with (require racket/rerequire). Now we need a good loop:

(define editor-main-module "editor-main.rkt")

(define (main-loop)
  (dynamic-rerequire editor-main-module)
  (let/ec break
    (let loop ([hot-swap-state #f])
      (let* ([editor-main (dynamic-require editor-main-module 'editor-main)]
             [editor-custodian (make-custodian)]
             [hot-swap-state-prime
              (parameterize ([current-custodian editor-custodian])
                (call/ec (editor-main hot-swap-state break)))])
        (custodian-shutdown-all editor-custodian)
        (dynamic-rerequire editor-main-module)
        (loop hot-swap-state-prime)))))

This is slightly dense, so let’s do some unpacking. First, we abstract away the file location of editor-main.rkt. That’s just so that in case we ever want to move the file, we know there’s only one place to change it in the code. The main loop first does a dynamic-rerequire of that editor-main module. This call comes first so that it’s dynamic-rerequire that first pulls in editor-main.rkt as opposed to dynamic-require, which will not properly record the modification time of the source file.

Next, we record an escape continuation with let/ec. Why? So that it can later be used to quit the program. There will be basically no other way out of this loop, and this is the easiest way (for me) to “break” out. Now we need to get our hands on the editor-main function itself. We do this with a call to dynamic-require. It gives us whatever provided identifier we ask for, so we ask for editor-main, which is going to have a few changes of its own later. We also get ourselves a new custodian, which is a Racket thing you can read about here. Basically, if we execute some function with a custodian and then shut that custodian down, we can clean up after just about anything that code did, including its threads and GUI.

Finally, we call editor-main with the new custodian, the break continuation, and another escape continuation made on the spot right there with call/ec. Why two escape continuations? One to be able to Quit, and the other to be able to go back to the beginning of the loop from anywhere within editor-main without necessarily needing to return properly.

We expect editor-main to return with something we name hot-swap-state-prime. That’s funny; editor-main didn’t accept any arguments before, and it certainly didn’t return anything. What’s this about? Well, it’d suck if reloading the editor caused you to lose all your unsaved work, or made the editor forget what file it was editing.

The hot-swap-state is going to keep that around for us, so we can put it back once the GUI is rebuilt. That’s why we call loop with the new, hot-swap-state-prime, which becomes hot-swap-state in the next iteration. On the first run through, there is no hot-swap-state, so we call editor-main with #f to let it know it’s beginning fresh. But on every subsequent call, it gets whatever it gave itself when it was shut down before. So we can pass whatever we need to keep between instantiations.

Before looping, we need to call dynamic-rerequire again, so that the new contents of the source file are reloaded before dynamic-require pulls in the editor-main function for the next call.

We should call this loop from the main module:

(module+ main
  ;; Main entry point, executed when run with the `racket` executable or DrRacket.
  (main-loop)
  (exit)
  )

Now we need to fix up editor-main.rkt a bit. First, rename the editor-main function to build-gui, and make it take two arguments:

(define (build-gui main-thread hot-swap-state)

We’ll need auxiliary functions within the definition of build-gui. Call them build-hot-swap-state and restore-hot-swap-state, and put them right after the definition of the racket:text% object, t:

(define (build-hot-swap-state)
  `(,(send t get-flattened-text)
    ,(send t get-filename)
    ,(send t is-modified?)))
(define (restore-hot-swap-state)
   (match hot-swap-state
     [(list text-contents filename modified) ;=>
      (send t insert text-contents)
      (send t set-filename filename)
      (send t set-modified modified)]))

We also need two new menu items after “Save”:

  (new separator-menu-item% [parent m-file])
  (define mi-reinstantiate-editor
    (new menu-item%
         [label "Reinstantiate editor"]
         [parent m-file]
         [callback
          (λ (i e)
            (thread-send
             main-thread
             `(reinstantiate-editor ,(build-hot-swap-state))))]
         [shortcut #\r]
         [shortcut-prefix '(ctl alt)]))
  (define mi-quit
    (new menu-item%
         [label "Quit"]
         [parent m-file]
         ; TODO: Prompt to save modified files
         [callback (λ (i e) (thread-send main-thread 'end))]
         [shortcut #\q]
         [shortcut-prefix '(ctl)]))

And we need to call restore-hot-swap-state when it is not false at the end of the function:

  (when hot-swap-state
    (restore-hot-swap-state))

And finally, we need a new editor-main that will set things up nicely:

(define ((editor-main hot-swap-state end-program) reload-program)
  (let ([main-window-es (make-eventspace)]
        [you-got-mail (thread-receive-evt)])
    (parameterize ([current-eventspace main-window-es])
      (build-gui (current-thread) hot-swap-state))
    (match (sync ;; Don't change the code inconsistently
            (wrap-evt main-window-es (λ (v) 'closed))
            (wrap-evt you-got-mail (λ (v) 'mail)))
      ['mail
       (match (thread-receive)
         [`(reinstantiate-editor ,hot-swap-state-prime)
          ;=>
          (reload-program hot-swap-state-prime)]
         ['end (end-program)]
         ;; Unknown message
         [m (displayln m)])]
      ['closed (end-program)]
      ;; Can't happen unless the code is changed incosistently
      [s (displayln s)])))

This is also a bit thick, so let’s unpack it. The make-eventspace is making a new Racket thread, with a few extra trimmings that help keep GUI things nicely separated out. This allows the build-gui function to return and the GUI to remain responsive while the editor-main function sits and listens for messages. you-got-mail is an event that is ready for synchronization when the thread recieves a message from another thread via thread-send.

The build-gui function is called with the current thread (so that it can send messages back) and the hot-swap-state as arguments.

Then we synchronize on two possible Events. If we get 'closed, it means the window was closed. If we get 'mail, it means the thread was sent a message. In the former case, we can just end-program, using the continuation that pulls us totally out of the loop in main.rkt. However, if we got mail, we need to open it to figure out the user’s intentions.

If the user wants to reinstantiate the editor, then we got a new hot-swap-state from the GUI callback. We’ll just invoke our reload-program escape continuation with that value to send it back to the loop in main.rkt, where it’s expected. If the user wants to quit, we invoke the end-program escape continuation, breaking us totally out of the loop.

And there you have it. Try running the result. It doesn’t look any different except for the “Reinstantiate editor” and “Quit” buttons. But try changing editor-main.rkt, saving, and then hitting “Reinstantiate editor”. Isn’t that fun?

Future Episodes

Next time, tabs! Probably.

(edited on 2017-08-01 to fix a typo and be ever so slightly more professional in the intro)