dillon1000/react

[compiler] Port React Compiler to Rust (#36173)

This is an experimental, work-in-progress port of React Compiler to
Rust. Key points:

* Work-in-progress - we are sharing early, prior to testing internally
at Meta, to get feedback from partners in parallel with continued
development.
* No builds available yet, you'll have to do some hacking if you want to
try this.
* All fixtures pass, no known gaps but there may be lurking bugs.
* The architecture was heavily guided by humans (me, @josephsavona) but
majority coded by AI. I was very hands-on in setting the architecture,
the testing and verification strategy, incremental migration approach,
etc. I also kept a close eye on the code and spent a decent amount of
time going back and forth to get code quality to a decent level.
* The public API is basically "Rust Babel AST" + Scope Info in, Rust
Babel AST out. We use a Rust representation of the Babel AST as our
"public API", as it were, and then each integration (Babel, OXC, SWC)
converts to/from their native representation. For now integrations must
also provide scope information - in the future React Compiler may
compute bindings and references itself from the AST.
* Internally, the Rust version uses the same architecture as the
TypeScript version. The compiler converts from the AST into our own
intermediate representation (HIR, short for High-level Intermediate
Representation) which uses a control-flow graph (CFG) and single-static
assignment (SSA). We go through the same series of passes, with the same
overall algorithms. It's very much a pass-by-pass port. The main
differences are in the data representation - using arena-like structures
(and indices into these arenas) to work within Rust's borrowing system.
* Early performance numbers are derived from AI and i haven't spent much
time validating the benchmark setup, beyond the fact that the
optimization opportunities it discovered made complete sense and the
fixes were right. With that caveat, itt does appear that the Rust
version is quite fast already: 3x faster when operating as a Babel
plugin. The serialization cost is quite high, but the actual
transformation logic is ~10x faster, so it's net faster. Native
integrations (oxc, swc) should be even faster.
* There are 3 integrations right now: an alternative Babel plugin (which
will eventually get removed as we integrate into
babel-plugin-react-compiler), and examples of what OXC and SWC
integrations could look like (see react_compiler_oxc and
react_compiler_swc crates).

correctness:
* all 1725 fixtures pass in snap when comparing the temporary rust
version of the plugin with the main version. this compares generated
code output as well as errors.
* all fixtures also pass a full comparison of the per-pass compiler
intermediate representation — the intermediate state (including log
events and errors) are ~identical after every single pass (modulo some
normalization of ids)
* The OXC and SWC example integrations seem to be working well, though i
haven't manually verified this to the same extent as i have the Babel
integration.

development:
* `yarn snap --rust` is the primary test suite, testing that we error or
compile as expected. It does not test the inner state of the compiler
along the way, though, making it less suitable for finding subtle logic
gaps btw the TS and Rust versions. It's also Babel based, making it less
easy to test OXC and SWC integrations.
* `compiler/scripts/test-e2e.sh` is an e2e test of all 3 variants (babel
wrapper around Rust, OXC/SWC integrations) against the TS
implementation. This does a partial comparison, focused on final output
code only (doesn't test error details etc). Useful for getting the swc
and oxc integrations closer to parity.
* `compiler/script/test-rust-port.sh` does detailed testing of the
internal compiler state after each pass, in addition to checking the
final output code. This is the key script used to port the compiler,
ensuring not just that the output was the same but that each pass was
capturing all the same detail. This script can be pointed at *any*
directory of JS files, which we expect to use for internal testing at
Meta.


## For Partners

We're excited to partner with teams to integrate the Rust version of
React Compiler into other tools, like OXC and SWC. If you're interested
in working with us on this, the best place to start is by taking a look
at the react_compiler_swc and react_compiler_oxc crates. These give you
an idea of the API shape that we're thinking of.

Note that the conversion from any AST into our HIR is complex, and we
can only maintain one version. Hence we've aligned on using a Babel-like
AST as our public API. Another key point is that we don't yet implement
our own scope analysis (since the TS version of the compiler relied on
Babel's scope analysis), so for now we require that the scope data be
serialized. It's a denormalized graph, and some metadata has to be
stored to associate nodes with scopes. We're open to feedback about the
AST and scope representation - we iterated a bit just to get things to
work, but it can be more optimal.

Key changes that we are considering:
* Currently the compiler returns `Option<Program>`, which is `Some` if
anything changed. This requires replacing the entire program. We plan to
change this to return a series of patches to apply, in a form that is
reasonably usable and efficient for all the integrations we care about
(Babel, OXC, SWC, etc).
* The Rust representation of the Babel AST is fine enough, but we could
make it more optimal by doing arena allocation. We also plan to change
the string representation to smol_str.
* The scope representation, and association of data btw AST and scope,
is very much a first pass approach that is good enough. We expect to
implement our own scope resolution, though, so we hopefully won't need
to iterate on the scope representation and can just throw it away.

In terms of the shape of the integration, we anticipate that each
integration would have the following:

* Implementor repo (OXC, SWC, etc): lightweight code transform and lint
pipeline integration that delegates to `crates/react_compiler_<name>`
from our repo
* Our repo: one crate per implementor, eg react_compiler_swc,
react_compiler_oxc, where most of the logic lives.

This setup lets us make changes to the integration layer easily within
our repo. Feedback appreciated!

---------

Co-authored-by: Joe Savona <[email protected]>
Co-authored-by: Mike Vitousek <[email protected]>
Co-authored-by: Mike Vitousek <[email protected]>
Co-authored-by: lauren <[email protected]>
Co-authored-by: lauren <[email protected]>