Persistent caches

[joliss]

Talking with @stefanpenner, here's an idea of how to do it:

In a plugin that wants persistent caches, call the base constructor:

Plugin.call(this, ..., { persistentCache: __dirname })

Broccoli will scan the source files of the calling plugin and its npm dependencies and hash over their mtimes using hash-for-dep. (We cannot just hash over version numbers, because people might be editing some of the sources.)

Multiple instances (nodes) of the same plugin share the same cache space.

During the build the plugin gets and sets keys:

this.persistentCache.getAndRetain(computeHash([relPath, mtime, otherOptions])) // => buffer or null
this.persistentCache.setAndRetain(computeHash([relPath, mtime, otherOptions]), buffer)

Broccoli implicitly adds the hash-for-dep hash to the key.

At the end of each build, all the keys in the cache that haven't been retained by a "get" or "set" call are evicted from the cache.

Different environments (dev, test, production) might use slightly different sets of nodes. We don't want them to evict each other. So we scope on the environment so they're independent.

/tmp/broccoli-persistent-cache-HOSTNAME-_home_jo_myproject/dev/deadbeef
     ^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^ ^^^^^^^^^^^^^^^^^^ ^^^ ^^^^^^^^
     fixed                     allow    project path       env key hash
                               for NFS  (likely hashed
                                        to deal with long 
                                        paths)

Multiple processes can use the same persistent cache at the same time. We need to be a bit careful that concurrent setting or eviction doesn't break things.

[rwjblue]

👍

[BryanCrotaz]

I'm writing a broccoli plugin - what is best practice for caching files?

[mariokostelac]

It would worth a shot to think abstracting mtime to fileVersion, calculated by a passed function, defaulted to getting mtime (or not). That would allow reusing persisted cache on repositories based on version controls that do not preserve file metadata (like git, for example).