Implement SWH compatible SHA-1s

[gousiosg]

Is your feature request related to a problem? Please describe.

We currently compute by SHA1s by calling the SHA1 function on the file contents. To link with SWH's software archive, we need to compute git compatible SHA1s

Describe the solution you'd like

Add a column to the files table (e.g. swh_sha1) and fill it in with git salted SHA1s. As per @zvr:

Git computes the SHA of a file object (blob) by "salting" (prepending) to its contents the word "blob", a space, the size of the file in decimal and a NUL byte.

$ echo 'Hello, World!' | git hash-object -w --stdin
8ab686eafeb1f44702738c8b0f24f2567c36da6d
$ echo -e 'blob 14\0Hello, World!' | shasum
8ab686eafeb1f44702738c8b0f24f2567c36da6d

Additional context

• Information about SWH persistent ids

[amottier]

@gousiosg you mention that we are currently computing SHA1 but in the wiki I found a reference to SHA256.

As I'm not very familiar with the overall FASTEN architecture I have trouble locating the code that is actually computes the SHA.
Can you point me to the appropriate direction so I can double check if we are using SHA-1 or SHA-256 and update (if need) the wiki?

Thanks.

[proksch]

Adding a small comment just to raise this issue to the top. Should be an easy fix and should be implemented before the next restart of the whole pipeline.

[proksch]

@amottier In our call some weeks ago, Cedric said that he will talk to Stefano to understand the details. I am not sure whether this happened before he left the project... do you have any further insights or would you be so nice to initiate the contact with Stefano to understand what exactly is required from our side?

[amottier]

Documentation of the Python tool (swh-identify) to generate the SWHID: https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html#computing

[mir-am]

@proksch, @amottier
To implement this feature, I have looked into the Metadata DB code. I have two findings:
1- ATM, we do NOT insert checksum for the files table. Therefore, its value is always null. Instead of adding a new column, we can insert SWH-compatible SHA into this field.
2- It seems that we do not have access to contents/source code for files of a package version. Therefore, it may not be possible to implement what Georgios described above.