Source maps

[natevw]

SMASH transmorgrifies an input and outputs JavaScript. Anything that transmorgrifies an input and outputs JavaScript should generate a source map. Therefore, SMASH should generate a source map. ∎

[natevw]

Started looking to this today. It's not particularly trivial, but of course certainly doable.

For input that doesn't have a map, AFAICT we'll still need to map each individual output line explicitly to its input. I don't quite see a shorthand way of allowing subsequent output lines to just keep matching up with input (although perhaps just a semicolon per line ;;;;;;;;;?)

For input that has already been processed (i.e. already having a map file of its own), there is a source map "index" format. This would primarily be detected via the //# sourceMappingURL= comment I suppose, perhaps also sniffing for a input.js.map file?

Potential gotchas

To be honest, the source map proposal still has a very inelegant feel to it. Or put more nicely, it's a very practical way to start dealing with the problem from the browser's view, with very little to facilitate implementation of the generator's workflow/tooling IMO.

• one of the fields in the map JSON (mercifully optional at least) is the map "file" name itself!
• when transforming already-mapped files, their sourceMappingURL may be relative to the hosted script, and very unlikely to be relative to the development filesystem
• also needs a sourceRoot within the file, which will usually be different between generating filesystem and hosting webserver — this would need to be user-configurable
• handling mixed ("original" vs. "already mapped") sources, I'm not sure this will be a problem but could be

References

• Helpful introduction/overview
• Source map proposal
• potentially useful helper library (may include way more than we need)

[natevw]

Hah, after all that I think I just realized a much much simpler way to deal with this in practice!

.SECONDEXPANSION:
static/%.js static/%.js.map: js/%.build.js $$(shell node_modules/.bin/smash --ignore-missing --list js/%.build.js > /dev/null)
    @echo Combine and compress: $^
    node_modules/.bin/uglifyjs --source-map [email protected] --source-map-url $*.js.map --source-map-root dbg-js -p 1 -c warnings=false -m -b beautify=false,semicolons=false $^ > $@

I.e. have smash figure out the dependencies and their order, but then defer directly to Uglify for the rest.

[natevw]

Dang it, of course I can't just skip smash, because that leaves all the "import" statements (and using sed or some other hack to strip them out pre-uglify would mess up line numbering).

update: so I guess I could leave /blank/ lines in place of the stripped import statements…hmmm…

[natevw]

Oy, this is ugly but, it works!

Backstory: this assumes source directory js/ and served directory static/ (which contains a symlink dbg-js back to ../js). I mark my "primary" source files (in root of js/) which I want to serve out of static/ with a .build.js extension. Clear as mud? I then need this chunk in my Makefile:

.PHONY: js
listfiles = $(shell ls -1 js/*.build.js)
js: $(listfiles:js/%.build.js=static/%.js)

.SECONDEXPANSION:
static/%.js static/%.js.map: $$(shell node_modules/.bin/smash --ignore-missing --list js/%.build.js | sed 's/^js/js\/.noimport/')
    node_modules/.bin/uglifyjs --source-map [email protected] --source-map-url $*.js.map --source-map-root dbg-js -p 2 -c warnings=false -m -b beautify=false,semicolons=false $^ > $@
js/.noimport/%.js: js/%.js
    mkdir -p $(dir $@)
    sed 's/^import[[:space:]]\{1,\}"[^"]\{1,\}"[[:space:]]*;\{0,1\}[[:space:]]*\(\/\/.*\)\{0,1\}//' $< > $@

Voilà! Every time I modify any relevant source file or even make a new js/foo_bar.build.js, make js uses that lovely bit of PDP-11 golf to magically Do The Right Thing™ [disclaimer: On My Machine In Testing So Far®]

To deal with the import "blahblah"; statements I've made the target depend on intermediary files (by munging the output of smash --list). These files get hackily barfed into a hidden/mirror folder structure; this mirrored structure is needed so the relative source map path matches, and it needs to be hidden otherwise Nate will remember his hatred for it forever.

The lovely import-truncating regex is a ported version of

smash/lib/smash/emit-imports.js

Line 26 in d5efeea

var match = /^import\s+"([^"]+)"\s*;?\s*(?:\/\/.*)?$/.exec(line);

so hopefully it matches smash ± exactly.

Note that make will remove the intermediary files automatically, but the directories just get left as litter…neither one is particularly ideal (could make the files .SECONDARY but…enough on this for the day).

[natevw]

I'm finding a couple of caveats:

• UglifyJS expects each file to be self-contained JS, e.g. gets upset when trying to import "d3/src/start.js", whereas this would have been fine w/Smash (xref Option to concatenate *before* parsing mishoo/UglifyJS#469)
• I'm not sure if smash supports "import" statements dumping their output into the middle of the calling file, but this alternative usage certainly won't
• TBD…

[natevw]

Perhaps this could be best solved by splitting into three separate stages:

1. dependency resolution (smash --list)
2. concatenation w/source map (sed truncation hack + maybe https://github.com/pmuellr/cat-source-map or similar)
3. minification w/source map (Uglify2 should be able to transform map from previous step)

Then I could use the same Makefile trick for an app-specific custom D3 build as well.

E.g. js/d3.custom.build.js

import "d3/start.js";
// … dependencies …
import "d3/end.js";

[natevw]

Here's a "behind the scenes" look at how browserify got source map support, some interesting tips/strategies: http://thlorenz.com/blog/browserify-sourcemaps