https://slack.engineering/balancing-old-tricks-with-new-feats-ai-powered-conversion-from-enzyme-to-react-testing-library-at-slack/
Skip to content
Slack Engineering
Search for: [ ]
Search
[featured_i]
Balancing Old Tricks with New Feats: AI-Powered Conversion From
Enzyme to React Testing Library at Slack
Accelerate code conversion with AI
[IMG_7728-e]
Sergii Gorbachov Senior Software Engineer
*
*
*
16 minutes * Written 28 days ago
In the world of frontend development, one thing remains certain:
change is the only constant. New frameworks emerge, and libraries can
become obsolete without warning. Keeping up with the ever-changing
ecosystem involves handling code conversions, both big and small. One
significant shift for us was the transition from Enzyme to React
Testing Library (RTL), prompting many engineers to convert their test
code to a more user-focused RTL approach. While both Enzyme and RTL
have their own strengths and weaknesses, the absence of native
support for React 18 by Enzyme presented a compelling rationale for
transitioning to RTL. It's so compelling that we at Slack decided to
convert more than 15,000 of our frontend unit and integration Enzyme
tests to RTL, as part of the update to React 18.
We started by exploring the most straightforward avenue of seeking
out potential Enzyme adapters for React 18. Unfortunately, our search
yielded no viable options. In his article titled "Enzyme is dead. Now
what?", Wojciech Maj, the author of the React 17 adapter,
unequivocally suggested, "you should consider looking for Enzyme
alternative right now."
complicated adapterAdapter illustrating the mismatch and
incompatibility of React 18 and Enzyme
Considering our ultimate goal of updating to React 18, which Enzyme
does not support, we started with a thorough analysis of the
problem's scope and ways to automate this process. Our initiative
began with a monumental task of converting more than 15,000 Enzyme
test cases, which translated to more than 10,000 potential
engineering hours. At that scale with that many engineering hours
required, it was almost obligatory to optimize and automate that
process. Despite thorough reviews of existing tools and extensive
Google searches, we found no suitable solutions for this very common
problem. In this blog, I will walk you through our approach to
automating the Enzyme-to-RTL conversion process. It includes
analyzing and scoping the challenge, utilizing traditional Abstract
Syntax Tree (AST) transformations and an AI Large Language Model
(LLM) independently, followed by our custom hybrid approach of
combining AST and LLM methodologies.
Abstract Syntax Tree (AST) transformations
Our initial approach centered around a more conventional way of
performing automated code conversions -- Abstract Syntax Tree (AST)
transformations. These transformations enable us to represent code as
a tree structure with nodes and create targeted queries with
conversions from one code node to another. For example, wrapper.find
('selector'); can can be represented as:
ast representationAST representation of `wrapper.find('selector');`
Naturally, we aimed to create rules to address the most common
conversion patterns. Besides focusing on the rendering methods, such
as mount and shallow, and various helpers utilizing them, we
identified the most frequently used Enzyme methods to prioritize the
conversion efforts. These are the top 10 methods in our codebase:
[
{ method: 'find', count: 13244 },
{ method: 'prop', count: 3050 },
{ method: 'simulate', count: 2755 },
{ method: 'text', count: 2181 },
{ method: 'update', count: 2147 },
{ method: 'instance', count: 1549 },
{ method: 'props', count: 1522 },
{ method: 'hostNodes', count: 1477 },
{ method: 'exists', count: 1174 },
{ method: 'first', count: 684 },
... and 55 more methods
]
One important requirement for our conversion was achieving
100%-correct transformations, because any deviation would result in
incorrect code generation. This challenge was particularly pronounced
with AST conversions, where in order to create transformations with
100% accuracy we needed to painstakingly create highly specific rules
for each scenario manually. Within our codebase, we found 65 Enzyme
methods, each with its own quirks, leading to a rapidly expanding
rule set and growing concerns about the feasibility of our efforts.
Take, for example, the Enzyme method find, which accepts a variety of
arguments like selector strings, component types, constructors, and
object properties. It also supports nested filtering methods like
first or filter, offering powerful element targeting capabilities but
adding complexity to AST manipulation.
In addition to the large number of manual rules needed for method
conversions, certain logic was dependent on the rendered component
Document Object Model (DOM) rather than the mere presence or absence
of comparable methods in RTL. For instance, the choice between
getByRole and getByTestId depended on the accessibility roles or test
IDs present in the rendered component. However, AST lacks the
capability to incorporate such contextual information. Its
functionality is confined to processing the conversion logic based
solely on the contents of the file being transformed, without
consideration for external sources such as the actual DOM or React
component code.
With each new transformation rule we tackled, the difficulty seemed
to escalate. After establishing patterns for 10 Enzyme methods and
addressing other obvious patterns related to our custom Jest matchers
and query selectors, it became apparent that AST alone couldn't
handle the complexity of this conversion task. Consequently, we opted
for a pragmatic approach: we achieved relatively satisfactory
conversions for the most common cases while resorting to manual
intervention for the more complex scenarios. For every line of code
requiring manual adjustments, we added comments with suggestions and
links to relevant documentation. This hybrid method yielded a modest
success rate of 45% automatically converted code across the selected
files used for evaluation. Eventually, we decided to provide this
tool to our frontend developer teams, advising them to run our
AST-based codemod first and then handle the remaining conversions
manually.
Exploring the AST provided useful insights into the complexity of the
problem. We faced the challenge of different testing methodologies in
Enzyme and RTL with no straightforward mapping between them.
Additionally, there were no suitable tools available to automate this
process effectively. As a result, we had to seek out alternative
approaches to address this challenge.
Large Language Models (LLMs) transformations
use of LLMsTeam members enthusiastically discussing AI applications
Amidst the widespread conversations on AI solutions and their
potential applications across the industry, our team felt compelled
to explore their applicability to our own challenges. Collaborating
with the DevXP AI team at Slack, who specialize in integrating AI
into the developer experience, we integrated the capabilities of
Anthropic's AI model, Claude 2.1, into our workflows. We created the
prompts and sent the test code along with them to our
recently-implemented API endpoint.
Despite our best efforts, we encountered significant variation and
inconsistency. Conversion success rates fluctuated between 40-60%.
The outcomes ranged from remarkably effective conversions to
disappointingly inadequate ones, depending largely on the complexity
of the task. While some conversions proved impressive, particularly
in transforming highly Enzyme-specific methods into functional RTL
equivalents, our attempts to refine prompts had limited success. Our
efforts to fine-tune prompts may have complicated matters, possibly
perplexing the AI model rather than aiding it. The scope of the task
was too large and multifaceted, so the standalone application of AI
failed to provide the consistent results we sought, highlighting the
complexities inherent in our conversion task.
The realization that we had to resort to manual conversions with
minimal automation was disheartening. It meant dedicating a
substantial amount of our team's and company's time to test
migration, time that could otherwise be invested in building new
features for our customers or enhancing developer experience.
However, at Slack, we highly value creativity and craftsmanship and
we didn't halt our efforts there. Instead, we remained determined to
explore every possible avenue available to us.
AST + LLM transformations
We decided to observe how real humans perform test conversions and
identify any aspects we might have overlooked. One notable advantage
in the comparison between manual human conversion and automated
processes was the wealth of information accessible to humans during
conversion tasks. Humans benefit from valuable insights taken from
various sources, including the rendered React component DOM, React
component code (often authored by the same individuals), AST
conversions, and extensive experience with frontend technologies.
Recognizing the significance of this, we reviewed our workflows and
integrated most of this relevant information into our conversion
pipeline. This is our final pipeline flowchart:
pipeline chartProject pipeline flowchart
This strategic pivot, and the integration of both AST and AI
technologies, helped us achieve the remarkable 80% conversion success
rate, based on selected files, demonstrating the complementary nature
of these approaches and their combined efficacy in addressing the
challenges we faced.
In our pursuit of optimizing our conversion process, we implemented
several strategic decisions that led to a notable 20-30% improvement
beyond the capabilities of our LLM model out-of-the-box. Among these,
two innovative approaches stood out that I will write about next:
1. DOM tree collection
2. LLM control with prompts and AST
DOM tree collection
One crucial aspect of our approach was the collection of the DOM tree
of React components. This step proved critical because RTL testing
relies heavily on the DOM structure of a component rather than its
internal structure. By capturing the actual rendered DOM for each
test case, we provided our AI model with essential contextual
information that enabled more accurate and relevant conversions.
This collection step was essential because each test case might have
different setups and properties passed to the component, resulting in
varying DOM structures for each test case. As part of our pipeline,
we ran Enzyme tests and extracted the rendered DOM. To streamline
this process, we developed adaptors for Enzyme rendering methods and
stored the rendered DOM for each test case in a format consumable by
the LLM model. For instance:
// Import original methods
import enzyme, { mount as originalMount, shallow as originalShallow } from 'enzyme';
import fs from 'fs';
let currentTestCaseName: string | null = null;
beforeEach(() => {
// Set the current test case name before each test
const testName = expect.getState().currentTestName;
currentTestCaseName = testName ? testName.trim() : null;
});
afterEach(() => {
// Reset the current test case name after each test
currentTestCaseName = null;
});
// Override mount method
enzyme.mount = (node: React.ReactElement, options?: enzyme.MountRendererProps) => {
const wrapper = originalMount(node, options);
const htmlContent = wrapper.html();
if (process.env.DOM_TREE_FILE) {
fs.appendFileSync(
process.env.DOM_TREE_FILE,
`${currentTestCaseName} and ${htmlContent};\n`,
);
}
return wrapper;
};
...
LLM control with prompts and AST
The second creative change we had to integrate was a more robust and
strict controlling mechanism for hallucinations and erratic responses
from our LLM. We achieved this by employing two key mechanisms:
prompts and in-code instructions made with the AST codemod. Through a
strategic combination of these approaches, we created a more coherent
and reliable conversion process, ensuring greater accuracy and
consistency in our AI-driven transformations.
We initially experimented with prompts as the primary means of
instructing the LLM model. However, this proved to be a
time-consuming task. Our attempts to create a universal prompt for
all requests, including initial and feedback requests, were met with
challenges. While we could have condensed our code by employing a
single, comprehensive prompt, we found that this approach led to a
significant increase in the complexity of requests made to the LLM.
Instead, we opted to streamline the process by formulating a prompt
with the most critical instructions that consisted of three parts:
introduction and general context setting, main request (10 explicit
required tasks and seven optional), followed by the instructions on
how to evaluate and present the results:
Context setting:
`I need assistance converting an Enzyme test case to the React Testing Library framework.
I will provide you with the Enzyme test file code inside