https://blog.appsignal.com/2021/10/05/under-the-hood-of-macros-in-elixir.html Logo of AppSignal * Home * Tour Features + Error tracking Catch errors and make sure they don't happen again. + Performance monitoring Find performance issues before they find you. + Host monitoring Run healthy apps on healthy hardware. + Metric dashboards Collect metrics and visualize them with a few lines of code. + Workflow features AppSignal keeps your team focused on building great apps. + Anomaly detection Get alerted in real-time when metrics go over their limits. + Uptime monitoring Get alerted in real-time when your application is down. Supported platforms + Elixir APM + Ruby APM + NodeJS APM + JavaScript error tracking * Docs * Blog * Pricing * Go to app * Log in * Start free trial Menu Features * Error tracking * Performance monitoring * Host monitoring * Metric dashboards * Workflow features * Anomaly detection * Uptime monitoring Platforms * Elixir APM * Ruby APM * NodeJS APM * JavaScript error tracking Product * Pricing Resources * Documentation * Blog * Go to app * Log in * Start free trial Under the Hood of Macros in Elixir Jia Jia Hao Woo on Oct 5, 2021 Lines "I absolutely love AppSignal." David David Becerra Happy developer --------------------------------------------------------------------- "I absolutely love AppSignal. It's helped me identify errors quickly and has provided some great insight on performance." Discover AppSignal Welcome back to part two of this series on metaprogramming in Elixir. In part one, we introduced metaprogramming and gave a brief overview of macros. In this part, we will explore the inner workings and behaviors of macros in more depth. As discussed in the previous post, macros are compile-time constructs in Elixir. So, before diving into how macros work, it is important to understand where macros lie within Elixir's compilation process. Stages of Elixir's Compilation Process We can boil down the Elixir compilation process to the following basic stages: Compilation process of an Elixir program (Note: the actual compilation process of an Elixir program is more intricate than above.) The compilation process can be broken down into the following phases: 1. Parsing -- The Elixir source code (program) is parsed into an AST, which we will call the initial AST. 2. Expansion -- The initial AST is scanned and macro calls are identified. Macros are executed. Their output (AST) is injected and expanded into the callsite. Expansion occurs recursively, and a final AST is generated. 3. Bytecode generation phase -- After the final AST is generated, the compiler performs an additional set of operations that eventually generate and execute BEAM VM bytecode. As you can see, macros sit at the expansion phase, right before code is converted into bytecode. Therefore, a good knowledge of the expansion phase helps us understand how macros work. Expansion Phase Let's start by first examining the expansion phase on a general level. The compiler will expand macros (as per Macro.expand) to become part of the program's pre-generated AST. Macro expansion occurs recursively, meaning that Elixir will continue expanding a macro until it reaches its most fundamental AST form. As macros expand right before bytecode is generated, they can modify a program's behavior during compile-time. If we dig a little deeper, we will find that the compiler first injects the output AST of a macro at its callsite. Then the AST is expanded recursively. We can observe this behavior as follows: 1 defmodule Foo do 2 defmacro foo do 3 quote do 4 IO.inspect("hello") 5 end 6 end 7 end 8 9 iex(1)> require Foo 10 iex(2)> ast = quote do: Foo.foo 11 {{:., [], [{:__aliases__, [alias: false], [:Foo]}, :foo]}, [no_parens: true], 12 []} 13 iex(3)> ast |> Macro.expand(__ENV__) 14 {{:., [], 15 [ 16 {:__aliases__, [counter: -576460752303423391, alias: false], [:IO]}, 17 :inspect 18 ]}, [], ["hello"]} ast represents the initial AST generated by the compiler before expansion. It holds a reference to the macro Foo.foo but it is not expanded as the macro has not been evaluated yet. When we call Macro.expand on the given AST, the compiler begins by injecting the behavior of the macro into the callsite. We can expand the AST one step at a time using Macro.expand_once. Contexts in Macros Now that we understand the basics of the expansion phase, we can investigate the parts of a macro. Macros contain two contexts -- a macro context and a caller context: 1 defmacro foo do 2 # This is the macro's context, this is executed when the macro is called 3 4 # This is the return value of the macro (AST) 5 quote do 6 # This is the caller's context, this is executed when the callsite is called 7 end 8 end As you can see, the macro's context is any expression declared before the quote. The caller's context is the behavior declared in the quote. The quote generated AST is the macro's output and is injected into and expanded at the callsite. The behavior defined under the caller's context 'belongs' to the caller, not the module where the macro is defined. The following example, taken from Metaprogramming Elixir, illustrates this: 1 defmodule Mod do 2 defmacro definfo do 3 IO.puts "definfo :: Macro's context #{__MODULE__}" 4 5 quote do 6 IO.puts "definfo :: Caller's context #{__MODULE__}" 7 8 def friendly_info do 9 IO.puts "friend_info :: Module name #{__MODULE__}" 10 end 11 end 12 end 13 end 14 15 defmodule MyModule do 16 require Mod 17 Mod.definfo 18 end 19 20 iex(1)> c "context.exs" 21 definfo :: Macro's context Elixir.Mod 22 definfo :: Caller's context Elixir.MyModule 23 [Mod, MyModule] 24 iex(2)> MyModule.friendly_info 25 friend_info :: Module name Elixir.MyModule 26 :ok As you can see, the module that executes the macro's context is Mod. But the module that executes the caller's context is MyModule -- the callsite where the macro is injected and expanded. Similarly, when we declare friendly_info, we inject this function into the callsite of the macro, which is MyModule. So the function now 'belongs' to MyModule. But why does there need to be two different contexts? What exactly makes the macro context and caller context different from one another? Order of Evaluation in Macro and Caller Contexts The key difference between the macro context and the caller context is that behavior is evaluated at different times. Let's look at an example: 1 defmodule Foo do 2 defmacro foo do 3 IO.puts("macro") 4 quote do 5 IO.puts("caller") 6 end 7 end 8 end 9 10 iex(1)> require Foo 11 iex(2)> ast = quote do: Foo.foo 12 {{:., [], [{:__aliases__, [alias: false], [:Foo]}, :foo]}, [no_parens: true], 13 []} 14 15 iex(3)> ast |> Macro.expand(__ENV__) 16 macro 17 {{:., [], 18 [{:__aliases__, [counter: -576460752303423293, alias: false], [:IO]}, :puts]}, 19 [], ["caller"]} When we expand the AST of a macro call, the macro context is evaluated and the caller context is injected and expanded into the callsite. However, we can go a level deeper. Let's look again at the first component of the expansion phase: Macros are executed. Their output (AST) is injected and expanded into the callsite. For a compiler to know what AST needs to be injected into the callsite, it has to retrieve the output of the macro during compilation (when the expansion phase occurs). The macro call is parsed as an AST during the parsing phase. The compiler identifies and executes these macro call ASTs prior to the expansion phase. If we think of macros as regular functions, the macro context is the function body and the caller context is the result of the function. During compilation, a macro is executed and evaluates the macro context. Then the quote is evaluated, returning the results of the function. The caller context is injected and expanded into the callsite of the macro. The macro context is evaluated during compile-time and treated as a regular function body. It executes within and is 'owned' by its containing module. The caller context is injected into the callsite, so it is 'owned' by the caller. It is evaluated whenever the callsite is evaluated. The example above showcases what happens when a macro is invoked at the module level. But what if we attempt to invoke the macro in a function? When do the macro and caller contexts evaluate? Well, we can use another example here: 1 defmodule Foo do 2 defmacro foo do 3 IO.puts("macro") 4 5 quote do 6 IO.puts("caller") 7 end 8 end 9 end 10 11 defmodule Bar do 12 require Foo 13 14 def execute do 15 IO.puts("execute") 16 Foo.foo 17 end 18 end 19 20 macro 21 iex(1)> Bar.execute 22 execute 23 caller The caller context evaluates when the function is called (as we established earlier). However, something interesting happens with our macro context -- it evaluates when the module compiles. This evaluation only happens once when Bar compiles, as evidenced by the lack of "macro" in our output when we call Bar.execute. Why is this the case? Well, we only need to evaluate the macro once to retrieve its output (caller context) and inject it into the callsite (which is a function in this case). The caller context behavior evaluates every time the function is called. This difference in the order and time of evaluation helps guide us on when to use the macro and the caller contexts. We use the macro context when we want the behavior to be evaluated during compile-time. This is regardless of when the caller context is evaluated or where the macro is called in the code. We use the caller context when we want to invoke behavior injected into the callsite at evaluation. Now that we have a better grasp of the Elixir compilation process, macros, and the order of evaluation, we can revisit unquote. Revisiting unquote In part one of this series, we established that unquote evaluates a given expression and injects the result (as an AST) into the AST built from quote. This is only a piece of the puzzle. Let's dig deeper to understand the behavior of unquote during compilation and the necessity of using it. While the rest of the quote body is evaluated at the same time as the callsite, unquote is evaluated (immediately) during compile-time -- when the macro is evaluated. unquote aims to evaluate and inject the result of a given expression. This expression might contain information that is only available during the macro evaluation, including variables that are initialized during this process. unquote must be evaluated during compile-time along with the macro, so that the AST of the result injects into the quote that we build. But why do we need to unquote the expression to inject it into the AST? To answer this, let's compare the expanded AST of a macro using unquote against one that does not: 1 defmodule Foo do 2 defmacro foo(x) do 3 quote do 4 IO.inspect(x) 5 end 6 end 7 end 8 9 defmodule Bar do 10 defmacro bar(y) do 11 quote do 12 IO.inspect(unquote(y)) 13 end 14 end 15 end 16 17 iex(1)> require Foo 18 iex(2)> require Bar 19 iex(3)> ast_foo = quote do: Foo.foo(1 + 2 * 3) 20 iex(4)> ast_bar = quote do: Bar.bar(1 + 2 * 3) 21 iex(5)> ast_foo |> Macro.expand(__ENV__) 22 {{:., [], 23 [ 24 {:__aliases__, [counter: -576460752303423448, alias: false], [:IO]}, 25 :inspect 26 ]}, [], [{:x, [counter: -576460752303423448], Foo}]} 27 28 iex(6)> ast_bar |> Macro.expand(__ENV__) 29 {{:., [], 30 [ 31 {:__aliases__, [counter: -576460752303423384, alias: false], [:IO]}, 32 :inspect 33 ]}, [], 34 [ 35 {:+, [context: Elixir, import: Kernel], 36 [1, {:*, [context: Elixir, import: Kernel], [2, 3]}]} 37 ]} Observe that the expanded Foo.foo AST is vastly different from the Bar.bar AST even though they are both given the same variable. This is because Elixir is quite literal with variable references. If a variable is referenced without unquote, an AST of that variable reference injects into the AST. Using unquote ensures that the underlying AST of the variable's value injects into the quote body. Now you may ask: What is the difference in variable scoping between the evaluation of macros and the execution of the callsite? Why does it matter? The scoping of variables in macros can be a confusing subject, so let's demystify it. Variable Scoping in Macros Now that we understand how macros are evaluated and expanded, we can look at the scoping of variables in macros, and when to use the options unquote and bind_quoted in quote. Due to function clause scoping, the arguments of a variable are initialized and 'in scope' during the macro evaluation of a function. Similarly, variables declared and assigned within a function body remain in scope until the function ceases. The same behavior applies to macros. When the macro context is evaluated, its arguments and any initialized variables are 'in scope.' This is why unquote can evaluate variable references declared as arguments of the macro or any variables initialized in the macro context. Any evaluation of variable initialization in the caller context will initialize these variables within the callsite during execution. To understand this difference better, let's look at a few examples: 1 defmacro foo do 2 quote do 3 x = 1 + 1 4 def bar, do: IO.inspect(unquote(x)) 5 end 6 end In this first example, unquote will not work. The variable x has not yet been initialized, but should have been initialized during the execution of the callsite. The immediate evaluation of unquote runs too early, so we cannot reference our variable x when we need to. When unquote evaluates during compile-time, it attempts to evaluate the variable reference expression of x and finds that it is not in scope. How can we fix this? By disabling unquoting. This means disabling the immediate evaluation of unquote. We only want unquote to evaluate when our caller context evaluates. This ensures that unquote can properly reference a variable in scope (x) as variable initialization would have occurred during the evaluation of the callsite. 1 defmacro foo do 2 quote unquote: false do 3 x = 1 + 1 4 def bar, do: IO.inspect(unquote(x)) 5 end 6 end This example highlights the impact of scoping in macros. If we attempt to access a variable that is available during the evaluation of the macro context, unquote as-is is perfect for us. However, suppose we try to access a variable that is only available during the evaluation of the callsite. In that case, we must disable the immediate unquoting behavior to initialize variables in scope before unquote attempts to reference them. Let's apply this understanding to two other examples. 1 defmacro foo(opts) do 2 quote bind_quoted: [opts: opts] do 3 x = Keyword.get(opts, :x) 4 def bar, do: IO.inspect(unquote(x)) 5 end 6 end In this example, we have initialized the variable x from a keyword list. As the keyword list is initialized during compile-time (along with the evaluation of the macro context), we first have to bind it to the caller context to: 1. Generate an initialization of the variable during the evaluation of the callsite, and 2. Disable unquoting behavior. We have to bind opts to the caller context, as the variable is no longer in scope during the evaluation of the callsite. Finally, we have: 1 defmacro foo(x) do 2 quote do 3 def bar, do: IO.inspect(unquote(x)) 4 end 5 end In this last example, x remains a variable in scope during the evaluation of the macro context -- i.e. when the macro is called. The immediate evaluation of unquote works in our favor. It renders unquote(x) valid, as x is in scope when unquote is evaluated. Macro Hygiene in Elixir While we are on the topic of scopes in macros, let's discuss macro hygiene. According to tutorialspoint.dev: Hygienic macros are macros whose expansion is guaranteed not to cause the accidental capture of identifiers. This means that if we inject and expand a macro into the callsite, we need not worry about the macro's variables (defined in the caller context) conflicting with the caller's variables. Elixir ensures this by maintaining a distinction between a caller variable and macro variable. You can explore this further using an example from the official tutorial. A macro variable is declared within the body of quote, while a caller variable is declared within the callsite of the macro. 1 defmodule Foo do 2 defmacro change do 3 quote do: a = 13 4 end 5 end 6 7 defmodule Bar do 8 require Foo 9 10 def go do 11 a = 1 12 Foo.change 13 a 14 end 15 end 16 17 Bar.go 18 # => 1 In this example, a referenced in change is not the same variable a in the scope of go. Even when we attempt to change the value of a, the value of a in the scope of go remains untouched. However, there may be a time where you have to reference a variable from the caller's scope in the macro's scope. Elixir provides the macro var! to bridge the gap between these two scopes: 1 defmodule Foo do 2 defmacro change do 3 quote do: var!(a) = 13 4 end 5 end 6 7 defmodule Bar do 8 require Foo 9 10 def go do 11 a = 1 12 Foo.change 13 a 14 end 15 end 16 17 Bar.go 18 # => 13 This distinction ensures no unintended changes to a variable due to changes within a macro (whose source code we may not have access to). You can apply the same hygiene to aliases and imports. Understanding require In the code examples shown in this section, we have always used require before we invoke the macros within the module. Why is that? This is the perfect segue into how the compiler resolves modules containing macros -- particularly the order in which modules are compiled. In Elixir, modules compile in parallel, and usually -- for regular modules -- the compiler is smart enough to compile dependencies of functions in the proper order of use. The parallel compilation process pauses the compilation of a file until the dependency is resolved. This behavior is replicated when handling modules that contain macro invocations. However, as macros must be available during compile-time, the module these macros belong to must be compiled beforehand. Here's where require comes into the picture. require explicitly informs the compiler to compile and load the module containing the macro first. We can use an example to illustrate this behavior: 1 defmodule Foo do 2 IO.puts("Compiling Foo") 3 4 defmacro foo do 5 IO.puts("Foo.foo := macro") 6 quote do 7 IO.puts("Foo.foo := caller") 8 end 9 end 10 end 11 12 defmodule Bar do 13 IO.puts("Compiling Bar") 14 15 require Foo 16 17 IO.puts("Bar := before Foo.foo") 18 Foo.foo() 19 end 20 21 Compiling Foo 22 Foo.foo := macro 23 Compiling Bar 24 Bar := before Foo.foo 25 Foo.foo := caller (Note that this is just an approximation of the actual compilation process, but it aims to paint a clearer picture of how require works.) Let's try to understand why the outputs are in this order. Firstly, Bar tries to compile. The compiler scans and finds a require for Foo before evaluating any module-level expressions within the module (such as IO.puts). So it pauses the compilation of Bar and compiles the Foo module first. As Foo is compiled, module-level code -- like IO.puts -- is evaluated, and the compiler prints the first line of the output. Once Foo is compiled, the compiler returns to Bar to resume compilation. Bar is parsed, macro calls are executed, and the macro context is evaluated. Even though Foo.foo is called after IO.puts ("Bar := before Foo.foo"), the evaluation of the macro call takes precedence over the evaluation of module-level code. During expansion, Foo.foo's caller context is injected and expanded into the callsite in Bar. It then behaves just like a regular module-level function call, printing the last three output lines in order of declaration. In essence, require instructs the compiler on the order of compilation that each module should go through if there are macro dependencies. This ensures that the macros are available during compile-time. Escaping Macros in Elixir Before explaining what Macro.escape does, let's look at an example: 1 iex(1)> x = {1, 2, 3} 2 iex(2)> quote do: unquote(x) 3 {1, 2, 3} 4 iex(3)> ast = quote do: IO.inspect(unquote(x)) 5 {{:., [], [{:__aliases__, [alias: false], [:IO]}, :inspect]}, [], [{1, 2, 3}]} 6 iex(4)> Code.eval_quoted(ast) 7 ** (CompileError) nofile: invalid quoted expression: {1, 2, 3} 8 9 Please make sure your quoted expressions are made of valid AST nodes. If you would like to introduce a value into the 10 AST, such as a four-element tuple or a map, make sure to call Macro.escape/1 before 11 (stdlib 3.15) lists.erl:1358: :lists.mapfoldl/3 12 (elixir 1.11.3) lib/code.ex:706: Code.eval_quoted/3 That is a strange error. Based on our understanding of unquote and macros, the code should work as intended, but it doesn't. Why is that? Well, the answer is found on iex(2). When we attempt to unquote x, we return not an AST, but a tuple -- the same one initially assigned to x. The error then points to the fact that the tuple is an invalid quoted expression. When we unquote(x) as-is, we inject a raw tuple into the AST, which cannot be evaluated as it is not a valid AST and throws an error. So, how do we fix it? We need to convert the raw tuple referenced by x into a valid AST. This can be achieved by escaping this value using Macro.escape. Let's understand what Macro.escape does: 1 iex(1)> a = {1, 2, 3} 2 {1, 2, 3} 3 iex(2)> Macro.escape(a) 4 {:{}, [], [1, 2, 3]} 5 iex(3)> quote do: unquote(a) 6 {1, 2, 3} 7 iex(4)> quote do: unquote(Macro.escape(a)) 8 {:{}, [], [1, 2, 3]} In iex(2), we see that Macro.escape(a) returns an AST of the tuple, not the raw tuple -- and this is exactly what we are looking for. By combining Macro.escape's behavior with unquote, we can inject the AST of the tuple into the quote as seen in iex(4). Let's test this: 1 iex(1)> x = {1, 2, 3} 2 {1, 2, 3} 3 iex(2)> quote do: unquote(Macro.escape(x)) 4 {:{}, [], [1, 2, 3]} 5 iex(3)> ast = quote do: IO.inspect(unquote(Macro.escape(x))) 6 {{:., [], [{:__aliases__, [alias: false], [:IO]}, :inspect]}, [], 7 [{:{}, [], [1, 2, 3]}]} 8 iex(4)> Code.eval_quoted(ast) 9 {1, 2, 3} 10 {{1, 2, 3}, []} As you can see, the code works just as intended because we escape the tuple. Often when working with data structures like tuples and dictionaries, you may find that the injected data from unquote does not inject a valid AST. In these cases, you should use Macro.escape before unquote. Guard Clauses and Pattern Matching Finally, it's worth mentioning that, much like regular functions defined through def, macros can use guard clauses and pattern matching: 1 defmacro foo(x) when x == 4 do 2 IO.inspect("macro with x being 4") 3 end 4 5 defmacro foo(_) do 6 IO.inspect("macro with any other value") 7 end Up Next: Applications of Macros in Elixir Congratulations, you've made it to the end of this part! You should now have a better grasp of how macros work internally. In part three, we will look at the many applications of macros in Elixir. Happy coding! P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post! Our guest author Jia Hao Woo is a developer from the little red dot -- Singapore! He loves to tinker with various technologies and has been using Elixir and Go for about a year. Follow his programming journey at his blog and on Twitter. AppSignal monitors your apps AppSignal provides insights for Ruby, Rails, Elixir, Phoenix, Node.js, Express and many other frameworks and libraries. We are located in beautiful Amsterdam. We love stroopwafels. If you do too, let us know. We might send you some! Discover AppSignal 5 favorite Elixir articles * Nov 17 2020 Announcing AppSignal for Elixir 2.0 * May 19 2020 Using Mnesia in an Elixir Application * Mar 10 2020 Building Compile-time Tools With Elixir's Compiler Tracing Features * Feb 4 2020 Monitoring the Erlang VM With AppSignal's Magic Dashboard * Apr 16 2019 Routing in Phoenix Umbrella Apps 10 latest Elixir articles * Sep 28 2021 Real-Time Form Validation with Phoenix LiveView * Sep 21 2021 What Are Atoms in Elixir and How To Monitor Them With AppSignal * Sep 14 2021 Application Code Upgrades in Elixir * Sep 7 2021 An Introduction to Metaprogramming in Elixir * Aug 31 2021 LiveView Integration Tests in Elixir * Aug 23 2021 Using Supervisors to Organize Your Elixir Application * Aug 17 2021 An Introduction to Testing LiveView in Elixir * Jul 27 2021 A Guide to Hot Code Reloading in Elixir * Jul 13 2021 Building Aggregates in Elixir and PostgreSQL * Jun 30 2021 How to Monitor and Optimize Your Database Performance: A Practical Guide Go back Elixir alchemy icon Subscribe to Elixir Alchemy A true alchemist is never done exploring. And neither are we. Sign up for our Elixir Alchemy email series and receive deep insights about Elixir, Phoenix and other developments. [ ] [ ] Sign me up! [ ] We'd like to set cookies, read why. I accept