Naming `eval` Scripts with the `//# sourceURL` Directive

December 5th, 2014

In Firefox 36, SpiderMonkey (Firefox's JavaScript engine) now supports the //# sourceURL=my-display-url.js directive. This allows developers to give a name to a script created by eval or new Function, and get better stack traces.

To demonstrate this, let's use a minimal version of John Resig's micro templater. The micro templater compiles template strings into JS source code that it passes to new Function, thus transforming the template string into a function.

function tmpl(str) {
  return new Function("obj",
    "var p=[],print=function(){p.push.apply(p,arguments);};" +

    // Introduce the data as local variables using with(){}
    "with(obj){p.push('" +

    // Convert the template into pure JavaScript
    str
      .replace(/[\r\t\n]/g, " ")
      .split("<%").join("\t")
      .replace(/((^|%>)[^\t]*)'/g, "$1\r")
      .replace(/\t=(.*?)%>/g, "',$1,'")
      .split("\t").join("');")
      .split("%>").join("p.push('")
      .split("\r").join("\\'")

    + "');}return p.join('');");
};

The details of how the template is converted into JavaScript source code isn't of import; what is important is that it dynamically creates new scripts via code evaluated in new Function.

We can define a new templater function:

var hello = tmpl("<h1>Hello, <%=name%></h1>");

And use it like this:

hello({ name: "World!" });
// "<h1>Hello, World!</h1>"

When we get an error, SpiderMonkey will generate a name for the evaled (or in this case, new Functioned) script based on the location where the call to eval (or new Function) occurred. For our concrete example, this is the generated name for the hello templater function's frame:

file:///Users/fitzgen/scratch/foo.js line 2 > Function

And here it is in the context of an error with the whole stack trace:

hello({ name: Object.create(null) });
// TypeError: can't convert Object to string
// Stack trace:
// anonymous@file:///Users/fitzgen/scratch/foo.js line 2 > Function:1:107
// @file:///Users/fitzgen/scratch/foo.js:28:3

Despite being a solid improvement over just "eval frame" or something of that sort, these stack traces can still be difficult to read. If there are many different templater functions, the value of the eval script's introduction location is further diminished. It is difficult to determine which of the many functions created by tmpl contains the thrown error, because they all end up with the same name, because they were all created at the same location.

We can improve this situation with the //# sourceURL directive.

Consider this version of the tmpl function adapted to use the //# sourceURL directive:

function tmpl(name, str) {
  return new Function("obj",
    "var p=[],print=function(){p.push.apply(p,arguments);};" +

    // Introduce the data as local variables using with(){}
    "with(obj){p.push('" +

    // Convert the template into pure JavaScript
    str
      .replace(/[\r\t\n]/g, " ")
      .split("<%").join("\t")
      .replace(/((^|%>)[^\t]*)'/g, "$1\r")
      .replace(/\t=(.*?)%>/g, "',$1,'")
      .split("\t").join("');")
      .split("%>").join("p.push('")
      .split("\r").join("\\'")

    + "');}return p.join('');"
    + "//# sourceURL=" + name);
};

Note that the function takes a new parameter called name and appends //# sourceURL=<name> to the end of the generated JS code passed to new Function.

With this new version of tmpl, we create our templater function like this:

var hello = tmpl("hello.template", "<h1>Hello <%=name%></h1>");

Now SpiderMonkey will use the name given by the //# sourceURL directive, instead of using a name based on the introduction site of the script:

hello({ name: Object.create(null) });
// TypeError: can't convert Object to string
// Stack trace:
// anonymous@hello.template:1:107
// @file:///Users/fitzgen/scratch/foo.js:25:3

Giving the eval script a name makes it easier for us to debug errors originating from it, and we can give a different name to different scripts created at the same location.

The //# sourceURL directive is also particularly useful for dynamic code- and module-loaders, which fetch source text over the network and then eval it.

Additionally, in Firefox 37, evaled sources with a //# sourceURL directive will be debuggable and labeled with the name specified by the directive in the debugger. (Update: //# sourceURL support in the debugger is also available in Firefox 36!)

wu.js 2.0

August 7th, 2014

On May 21st, 2010, I started experimenting with lazy, functional streams in JavaScript with a library I named after the Wu-Tang Clan.

commit 9d3c5b19a088f6e33888c215f44ab59da4ece302
Author: Nick Fitzgerald <fitzgen@gmail.com>
Date:   Fri May 21 22:56:49 2010 -0700

    First commit

Four years later, the feature-complete, partially-implemented, and soon-to-be-finalized ECMAScript 6 supports lazy streams in the form of generators and its iterator protocol. Unfortunately, ES6 iterators are missing the higher order functions you expect: map, filter, reduce, etc.

Today, I'm happy to announce the release of wu.js version 2.0, which has been completely rewritten for ES6.

wu.js aims to provide higher order functions for ES6 iterables. Some of them you already know (filter, some, reduce) and some of them might be new to you (reductions, takeWhile). wu.js works with all ES6 iterables, including Arrays, Maps, Sets, and generators you write yourself. You don't have to wait for ES6 to be fully implemented by every JS engine, wu.js can be compiled to ES5 with the Tracuer compiler.

Here's a couple small examples:

const factorials = wu.count(1).reductions((last, n) => last * n);
// (1, 2, 6, 24, ...)

const isEven = x => x % 2 === 0;
const evens = wu.filter(isEven);
evens(wu.count());
// (0, 2, 4, 6, ...)

Check out the wu.js documentation here.

Come work with me on Firefox Developer Tools

July 8th, 2014

My team at Mozilla, the half of the larger devtools team that works on JavaScript and performance tools, is looking to hire another software engineer.

We have members of the devtools team in our San Francisco, London, Vancouver, Paris, Toronto, and Portland offices, but many also work remotely.

We are responsible for writing the full stack of the tools we create, from the C++ platform APIs exposing SpiderMonkey and Gecko internals, to the JavaScript/CSS/HTML based frontend that you see when you open the Firefox Developer Tools.

Some of the things we're working on:

One of the most important things for me is that every line of code we write at Mozilla is Free and Open Source from day one, and we're dedicated to keeping the web open.

Apply here!

Debugging Web Performance with Firefox DevTools - Velocity 2014

June 26th, 2014

On Tuesday, June 3rd, 2014, I gave a presentation on debugging web performance with Firefox DevTools to the Velocity Conf 2014, Santa Clara. I'm not sure how useful the slides are without me talking, but here they are:

Debugging Web Performance with Firefox DevTools - Velocity 2014

Beyond Source Maps

March 12th, 2014

There's been some recent talk on es-discuss about standardizing source maps in ECMAScript 7. Before that happens, we should take a moment to reflect on what source maps have done well, where they are lacking, and meditate on what a more perfect debug format for compilers targeting JavaScript might be. My response quickly outgrew an email reply, and so I am collecting my thoughts and posting them here.

Before diving in, it makes sense to tell you where I am coming from. I implemented the mozilla/source-map library and the source map support in the Firefox Developer Tools' debugger. I'm a coauthor of the document specifying source maps; although the actual design of the format was done by John Lenz. I mostly documented and polished the parts that weren't about the format itself: like how to find a script's source map. On top of that, I've debugged scripts with source maps compiled from browserify, the r.js AMD module optimizer, CoffeeScript, ClojureScript, Emscripten, UglifyJS, Closure Compiler, and more.

When do source maps work well?

Source maps work pretty well for:

When do source maps fall short?

Whenever you start doing "real compilation" instead of "transpiling" and your source language's semantics don't match JavaScript's semantics, source maps break down:

A stepping debugger isn't useful without the ability to inspect bindings and values in the environment, and the above issues create humungous hurdles for doing that!

Moreover, using the console as a REPL, adding a watch expression, or setting a new conditional breakpoint must be done with JavaScript, not the source language.

SourceMap.next?

Requirements

An ideal SourceMap.next format should support the existing use of source maps:

Additionally, any SourceMap.next format should support debugging JavaScript object code that the existing source map format fails to satisfy:

A Kernel of a Solution

The ideas I present here aren't fully baked yet, but I have confidence in the approach.

Because a SourceMap.next must do more than translate file, line, and column locations, it no longer makes sense to build the format on location mappings. Instead it should annotate the abstract syntax tree of the JavaScript object code with debugging information. Instead of adding debugging information to a line and column location in the JavaScript object code which in turn can be approximately translated into a specific JavaScript statement or expression, annotating the AST adds debugging information directly on a specific JavaScript statement or expression. It cuts out the middle man while simultaneously simplifying the architecture. Many existing tools are already using this approach: they create a JavaScript AST, annotate it with source language locations, and use escodegen to generate both the JavaScript object code and a source map!

Annotating the AST enables us to take advantage of the hierarchy present in the tree. If you are searching for a specific annotation and the current AST node does not have it, recurse on the node's parent. This makes it easy to provide very detailed information or a less fine-grained summary. For example, in the following AST, rather than requiring source location annotations for each AssignmentExpression, Identifier, BinaryExpression, and Literal AST nodes, one could simply add the annotation to the AssignmentExpression node. Then, any queries for source location information on any of the other nodes would bubble up to the AssignmentExpression and yield its annotated source location information.

AssignmentExpression
  operator: "="
  left: Identifier
    name: "answer"
  right: BinaryExpression
    operator: "*"
    left: Literal
      value: 6
    right: literal
      value: 7

Here are a few debugging annotations that should cover the requirements listed above:

"Solving the SourceMap.next problem once and for all" reduces to defining a compact binary encoding of JavaScript's AST that supports an arbitrary number of arbitrary debugging annotations on each node. The beauty of this approach is that as long as we take care that this format is future-extensible with new types of debugging annotations, we can resolve any future issues without needing to update the binary format. We just add new debugging annotations and keep the old annotations for legacy consumers until they die off. The only time the binary format changes is when a new version of the ECMAScript standard is released and defines new syntactic forms that the AST must incorporate.

The final affordance of having an extensible debugging format is that compilers which target JavaScript can progressively add more debugging information over time. Since specific annotations are optional, the compiler can simply skip those for which it can't provide information. When a later version of the compiler can provide that information, it can start adding those annotations.

« Older Entries


Recent Entries

Naming `eval` Scripts with the `//# sourceURL` Directive on December 5th, 2014

wu.js 2.0 on August 7th, 2014

Come work with me on Firefox Developer Tools on July 8th, 2014

Debugging Web Performance with Firefox DevTools - Velocity 2014 on June 26th, 2014

Beyond Source Maps on March 12th, 2014

Memory Tooling in Firefox Developer Tools in 2014 on March 4th, 2014

Hiding Implementation Details with ECMAScript 6 WeakMaps on January 13th, 2014

Re-evaluate Individual Functions in Firefox Developer Tools' Scratchpad on November 22nd, 2013

Testing Source Maps on October 2nd, 2013

Destructuring Assignment in ECMAScript 6 on August 15th, 2013

Creative Commons License

Fork me on GitHub