Refactor to separate bash and python scripts.

[aidanheerdegen]

Use parallel to allow simultaneous indexing and shuffling inputs.

Set file chunks to 1000.

Only here for information purposes in case it is useful. Requires update to COSIMA Cookbook

[aidanheerdegen]

This refactor generated a list of experiment directories to index, and then used parallel to run a separate python updating script. As the idea of parallel execution is probably not going to work it is perhaps not that useful, but I think the general idea is good, as it is then easier to replace the bash script with some other way of supplying these experiment directories, like a yaml file for instance.

[micaeljtoliveira]

@aidanheerdegen I think these changes are great, but I would go even further and move the python scrip to the COSIMA cookbook. That way we can ensure that the script is up-to-date with the database API. If you agree, I can take care of creating a PR for that.

[aidanheerdegen]

Yes, agreed. I think a proper CLI tool, with some options and, say, a config.yaml file would be great. I'd suggest creating an Issue first, where we can dump some ideas/suggestions and discuss them before writing the code.

[aidanheerdegen]

See COSIMA/cosima-cookbook#299 (would not have to be this complex in the first instance)

[micaeljtoliveira]

I'm closing this PR, as this has been implemented in COSIMA/cosima-cookbook#300 and #25