← forbear

forbear log, entry 003: Rendering JSX without a browser by embedding Node.js in a native program

This is realtime layouting happening every single frame, without any caching, running in both languages.

Both windows render in sync with my 240 Hz monitor, so the 4.2 ms you see on them is the time between frames, not the time the work takes.

To properly support JavaScript, I've decided to run it through Node.js. This doesn't mean I spawn a separate subprocess and then have to communicate through a socket or IPC or whatever. It's in the same process, by embedding Node.js and accessing the V8 memory mostly directly, which means we can keep the high bar for performance since most of the work is still done through jai.

The upside of running on Node.js is that users have access to all that they have there. They can use any libraries they're already used to, and they can use (almost) any Node APIs they might want. The missing part around this is asynchronous work, which with the way we run things, is still not supported.

Node.js is not going to linked/included on any build of forbear, only when the user actually wants to use JavaScript, so it doesn't affect anyone else.

Including a way to run from JavaScript might look odd when coming from the first entry of this log, but the reality is that it's better to provide a way to do things from wherever everyone prefers. I believe that users can build a better app using JavaScript here, than they would if they were to build with Electron. And the native language way of doing thigns will always be available instead.

The actual JavaScript code looks like this:

To actually support this JSX flavour, we're compiling this JSX with a TypeScript transformer that converts the element definitions into the open and close function calls. It also keeps track of source code locations so that it can replicate the keying that's done by the element macro in jai. You can see the result in the dist/index.js tab above.

open, close and text are native functions, defined from C++, that call straight into jai. jai then reads each style field it needs out of the JavaScript object through V8, so there's no serialization step in between.

The central part of this is element definition. Nodes are not defined in JavaScript here, as it might seem. Nodes are all defined in jai, but the parameters all come from JavaScript. This also means that, unlike in React, elements are not values you can pass around and return. You just "define" the element and it gets pushed into the Node_Tree in jai.

This is both a blessing and a curse. I think the pros outweigh the cons, particularly because this makes it more viable to rerun everything every single frame instead of just reacting when things change, like React does. This means you can do animations and all sorts of weird things across frames, since you have access to each and every one of them.

The curse is the fact that this is different from React. You can't store an element in a variable or pass it as a prop, which I believe leads to a sligthly worse DX.

But, if you've read any of the other posts, you'll know this is missing some of the important setup code that the previous applications already had, so here it is:

#import "Basic";
#import "File";
#import,file "../../forbear.jai"(JAVASCRIPT_ADAPTER = true);

#run {
    #import "Process";
    result := run_command("npm", "run", "build", working_directory = #filepath);
    assert(result.type == .EXITED && result.exit_code == 0, "npm run build failed");
}

JAVASCRIPT_SOURCE :: #run read_entire_file(tprint("%dist/index.js", #filepath));
INTER_TTF :: #run read_entire_file(tprint("%../Inter.ttf", #filepath));

main :: () {
    backend: Backend;
    window: Window;
    #if OS == .LINUX {
        init_backend(*backend);
        init_window(*backend.connection, *window, 800, 600, "playground");
    } else #if OS == .MACOS {
        init_backend(*backend);
        init_window(*backend, *window, 800, 600, "playground");
    }

    inter: Font;
    init_font(*inter, INTER_TTF);

    forbear: Forbear;
    init_forbear(*forbear, *window, *backend, Base_Style.{
        font = *inter,
    });

    javascript_adapter: Javascript_Adapter;
    init_adapter(*javascript_adapter, JAVASCRIPT_SOURCE, "javascript-example");

    while window.running {
        #if OS == .LINUX {
            while true {
                event := next_event(*backend.connection, !can_draw(window));
                if !event.something {
                    break;
                }
                update_backend(*backend, event);
                update_window(*backend, *window, event);
            }
        } else #if OS == .MACOS {
            update_backend(*backend);
            update_window(*backend, *window);
        }

        render_with_adapter(*javascript_adapter);
        layout();
        draw_and_reset_node_tree(rgb(0.0, 0.0, 0.0));
        reset_temporary_storage();
    }
}

If you've seen the other code samples, you can see that this is actually not that different. The main difference is that jai doesn't render the elements directly, but rather calls this render_with_adapter.

The other difference is at the top. The #run block runs npm run build while jai compiles, and JAVASCRIPT_SOURCE reads the output into a constant. So the JavaScript ends up inside the executable, with no file to read at startup.

The long-term ideal is that every JavaScript application built with forbear will have this "escape hatch" of native code where they can do just about whatever they want. The reasoning for this is basically twofold:

  1. so that nothing about the native side is hidden: the window, the events and the frame loop are a short file in your own project that you can read and change, and working next to it is how the fundamentals become familiar,
  2. and so that advanced users of the library can have full control of everything without having to give up their preferred convenience.

Among the details, we're also defining some .d.ts definitions so that the code feels comfortable for JavaScript developers, and many utilities that are defined using v8::FunctionTemplate from C++, sometimes calling jai code as well.

layout

In other news, you can clearly see we also have layouting implemented! This already has fitting, growing, shrinking, text wrapping, fixed, minimum, maximum sizes, padding, margins, horizontal and vertical placement, justification horizontally and vertically, flow, fixed and relative placement.

Overall, it's actually not an extremely complicated algorithm. It's heavily based on Clay, implemented with improvements over the old Zig version, which was written from scratch based on Nic's awesome YouTube video. It's only ~700 LOC in a single function.

The most interesting part about it is the fact that it runs in multiple passes over the entire Node_Tree:

  1. fit every node to its children, going from children to parents
  2. grow and shrink the children of every node, going from parents to children
  3. wrap the text, now that the widths are known
  4. fit the heights again, since wrapping changed them
  5. grow the heights again
  6. place every node and justify its children

That's done for two reasons: 1) because it makes the algorithm much simpler (as Nic explains in much more detail and depth than I possibly can) and 2) because each pass is a plain loop over the nodes, which is faster than the alternative of going back to retroactively update them.

The algorithm is also made simpler because we use a single, flat array to store all nodes, which I call the Node_Tree. Since nodes are always inserted in the order parent -> children, to iterate the entire tree we can literally just loop over it. If we want to go from children to parents, we can loop over it in reverse, which makes this extremely cache coherent. In the long term, this layouting pass will most likely be optimized with a structure-of-arrays approach.

In the future, I will write an explanation of this algorithm and keep it updated so that users of the library can also debug issues and understand layouting without having to read the code.

remaining fixes

Alongside all of these changes, I've also made a bunch of improvements to the Wayland client implementation and the way we define buffers, so that this library can start actually creating things that someone would actually use. It no longer crashes when screenshotting, it respects the monitor's refresh rate through the frame event from the Wayland compositor, and it uses a double buffer instead of just one.

Besides this, all forbear allocations within a frame are done through a Flat_Pool, which is a kind of allocator in jai that keeps a reserved space in RAM and is reset and reused across boundaries, frames in our case. This also fixes a few leaks we had.

what's next

I'm really excited to start building some real applications with this to stress test the algorithms and, most importantly, the JSX API. So, I plan on working on a calculator program. It should be simple, instant to open, beautiful and better than the status quo for calculators. A simple app to start, then we can start working up to more things.

Before I can do this, I'm planning on working on event handling first, then state management and components.

Also, you can now subscribe to entries on the website, so I can notify you when I write a new one!

https://github.com/gabrielmfern/forbear