Skip to content

Commit

Permalink
feat: Add archive-file externals
Browse files Browse the repository at this point in the history
  • Loading branch information
@twpayne
twpayne committed Jul 4, 2023
1 parent 5d71e82 commit 850a848
Show file tree
Hide file tree
Showing 4 changed files with 206 additions and 53 deletions.
57 changes: 34 additions & 23 deletions ...ezmoi.io/docs/reference/special-files-and-directories/chezmoiexternal-format.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,32 +13,33 @@ externals to be included on different machines.

Entries are indexed by target name relative to the directory of the
`.chezmoiexternal.$FORMAT` file, and must have a `type` and a `url` field.
`type` can be either `file`, `archive`, or `git-repo`. If the entry's parent
directories do not already exist in the source state then chezmoi will create
them as regular directories.
`type` can be either `file`, `archive`, `archive-file`, or `git-repo`. If the
entry's parent directories do not already exist in the source state then chezmoi
will create them as regular directories.

Entries may have the following fields:

| Variable | Type | Default value | Description |
| ----------------- | -------- | ------------- | ------------------------------------------------------------- |
| `type` | string | *none* | External type (`file`, `archive`, or `git-repo`) |
| `encrypted` | bool | `false` | Whether the external is encrypted |
| `exact` | bool | `false` | Add `exact_` attribute to directories in archive |
| `exclude` | []string | *none* | Patterns to exclude from archive |
| `executable` | bool | `false` | Add `executable_` attribute to file |
| `format` | string | *autodetect* | Format of archive |
| `include` | []string | *none* | Patterns to include from archive |
| `refreshPeriod` | duration | `0` | Refresh period |
| `stripComponents` | int | `0` | Number of leading directory components to strip from archives |
| `url` | string | *none* | URL |
| `checksum.sha256` | string | *none* | Expected SHA256 checksum of data |
| `checksum.sha384` | string | *none* | Expected SHA384 checksum of data |
| `checksum.sha512` | string | *none* | Expected SHA512 checksum of data |
| `checksum.size` | int | *none* | Expected size of data |
| `clone.args` | []string | *none* | Extra args to `git clone` |
| `filter.command` | string | *none* | Command to filter contents |
| `filter.args` | []string | *none* | Extra args to command to filter contents |
| `pull.args` | []string | *none* | Extra args to `git pull` |
| Variable | Type | Default value | Description |
| ----------------- | -------- | ------------- | ---------------------------------------------------------------- |
| `type` | string | *none* | External type (`file`, `archive`, `archive-file`, or `git-repo`) |
| `encrypted` | bool | `false` | Whether the external is encrypted |
| `exact` | bool | `false` | Add `exact_` attribute to directories in archive |
| `exclude` | []string | *none* | Patterns to exclude from archive |
| `executable` | bool | `false` | Add `executable_` attribute to file |
| `format` | string | *autodetect* | Format of archive |
| `path` | string | *none* | Path to file in archive |
| `include` | []string | *none* | Patterns to include from archive |
| `refreshPeriod` | duration | `0` | Refresh period |
| `stripComponents` | int | `0` | Number of leading directory components to strip from archives |
| `url` | string | *none* | URL |
| `checksum.sha256` | string | *none* | Expected SHA256 checksum of data |
| `checksum.sha384` | string | *none* | Expected SHA384 checksum of data |
| `checksum.sha512` | string | *none* | Expected SHA512 checksum of data |
| `checksum.size` | int | *none* | Expected size of data |
| `clone.args` | []string | *none* | Extra args to `git clone` |
| `filter.command` | string | *none* | Command to filter contents |
| `filter.args` | []string | *none* | Extra args to command to filter contents |
| `pull.args` | []string | *none* | Extra args to `git pull` |

If any of the optional `checksum.sha256`, `checksum.sha384`, or
`checksum.sha512` fields are set, chezmoi will verify that the downloaded data
Expand Down Expand Up @@ -84,6 +85,12 @@ determine whether an archive member is included:
Excluded archive members do not generate source state entries, and, if they are
directories, all of their children are also excluded.

If `type` is `archive-file` then the target is a file or symlink with the
contents of the entry `path` in the archive at `url`. The optional integer field
`stripComponents` will remove leading path components from the members of the
archive before comparing them with `path`. The behavior of `format` is the same
as for `archive`.

If `type` is `git-repo` then chezmoi will run `git clone $URL $TARGET_NAME`
with the optional `clone.args` if the target does not exist. If the target
exists, then chezmoi will run `git pull` with the optional `pull.args` to
Expand Down Expand Up @@ -120,6 +127,10 @@ re-download unless forced. To force chezmoi to re-download URLs, pass the
url = "https://github.com/romkatv/powerlevel10k/archive/v1.15.0.tar.gz"
exact = true
stripComponents = 1
[".local/bin/age"]
type = "archive-file"
url = "https://github.com/FiloSottile/age/releases/download/v1.1.1/age-v1.1.1-{{ .chezmoi.os }}-{{ .chezmoi.arch }}.tar.gz"
path = "age/age"
["www/adminer/plugins"]
type = "archive"
url = "https://api.github.com/repos/vrana/adminer/tarball"
Expand Down
16 changes: 6 additions & 10 deletions assets/chezmoi.io/docs/user-guide/include-files-from-elsewhere.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -113,24 +113,20 @@ to include

## Extract a single file from an archive

You can extract a single file from an archive using the `$ENTRY.filter.command`
and `$ENTRY.filter.args` variables in `.chezmoiexternal.$FORMAT`, for example:
You can extract a single file from an archive using the `archive-file` type in
`.chezmoiexternal.$FORMAT`, for example:

```toml title="~/.local/share/chezmoi/.chezmoiexternal.toml"
{{ $ageVersion := "1.0.0" -}}
{{ $ageVersion := "1.1.1" -}}
[".local/bin/age"]
type = "file"
type = "archive-file"
url = "https://github.com/FiloSottile/age/releases/download/v{{ $ageVersion }}/age-v{{ $ageVersion }}-{{ .chezmoi.os }}-{{ .chezmoi.arch }}.tar.gz"
executable = true
refreshPeriod = "168h"
[".local/bin/age".filter]
command = "tar"
args = ["--extract", "--file", "/dev/stdin", "--gzip", "--to-stdout", "age/age"]
path = "age/age"
```

This will extract the single archive member `age/age` from the given URL (which
is computed for the current OS and architecture) to the target
`./local/bin/age` and set its executable bit.
`./local/bin/age`.

## Import archives

Expand Down
174 changes: 154 additions & 20 deletions pkg/chezmoi/sourcestate.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,10 @@ type ExternalType string

// ExternalTypes.
const (
ExternalTypeArchive ExternalType = "archive"
ExternalTypeFile ExternalType = "file"
ExternalTypeGitRepo ExternalType = "git-repo"
ExternalTypeArchive ExternalType = "archive"
ExternalTypeArchiveFile ExternalType = "archive-file"
ExternalTypeFile ExternalType = "file"
ExternalTypeGitRepo ExternalType = "git-repo"
)

var (
Expand Down Expand Up @@ -79,9 +80,10 @@ type External struct {
Command string `json:"command" toml:"command" yaml:"command"`
Args []string `json:"args" toml:"args" yaml:"args"`
} `json:"filter" toml:"filter" yaml:"filter"`
Format ArchiveFormat `json:"format" toml:"format" yaml:"format"`
Include []string `json:"include" toml:"include" yaml:"include"`
Pull struct {
Format ArchiveFormat `json:"format" toml:"format" yaml:"format"`
Include []string `json:"include" toml:"include" yaml:"include"`
ArchivePath string `json:"path" toml:"path" yaml:"path"`
Pull struct {
Args []string `json:"args" toml:"args" yaml:"args"`
} `json:"pull" toml:"pull" yaml:"pull"`
RefreshPeriod Duration `json:"refreshPeriod" toml:"refreshPeriod" yaml:"refreshPeriod"`
Expand Down Expand Up @@ -2149,6 +2151,14 @@ func (s *SourceState) readExternal(
switch external.Type {
case ExternalTypeArchive:
return s.readExternalArchive(ctx, externalRelPath, parentSourceRelPath, external, options)
case ExternalTypeArchiveFile:
return s.readExternalArchiveFile(
ctx,
externalRelPath,
parentSourceRelPath,
external,
options,
)
case ExternalTypeFile:
return s.readExternalFile(ctx, externalRelPath, parentSourceRelPath, external, options)
case ExternalTypeGitRepo:
Expand All @@ -2167,19 +2177,11 @@ func (s *SourceState) readExternalArchive(
external *External,
options *ReadOptions,
) (map[RelPath][]SourceStateEntry, error) {
data, err := s.getExternalData(ctx, externalRelPath, external, options)
data, format, err := s.readExternalArchiveData(ctx, externalRelPath, external, options)
if err != nil {
return nil, err
}

url, err := url.Parse(external.URL)
if err != nil {
return nil, fmt.Errorf("%s: %s: %w", externalRelPath, external.URL, err)
}
urlPath := url.Path
if external.Encrypted {
urlPath = strings.TrimSuffix(urlPath, s.encryption.EncryptedSuffix())
}
dirAttr := DirAttr{
TargetName: externalRelPath.Base(),
Exact: external.Exact,
Expand All @@ -2199,11 +2201,6 @@ func (s *SourceState) readExternalArchive(
externalRelPath: {sourceStateDir},
}

format := external.Format
if format == ArchiveFormatUnknown {
format = GuessArchiveFormat(urlPath, data)
}

patternSet := newPatternSet()
for _, includePattern := range external.Include {
if err := patternSet.add(includePattern, patternSetInclude); err != nil {
Expand Down Expand Up @@ -2331,6 +2328,143 @@ func (s *SourceState) readExternalArchive(
return sourceStateEntries, nil
}

// readExternalArchiveData reads an external archive's data and returns its data
// and format.
func (s *SourceState) readExternalArchiveData(
ctx context.Context,
externalRelPath RelPath,
external *External,
options *ReadOptions,
) ([]byte, ArchiveFormat, error) {
data, err := s.getExternalData(ctx, externalRelPath, external, options)
if err != nil {
return nil, ArchiveFormatUnknown, err
}

url, err := url.Parse(external.URL)
if err != nil {
return nil, ArchiveFormatUnknown, fmt.Errorf(
"%s: %s: %w",
externalRelPath,
external.URL,
err,
)
}
urlPath := url.Path
if external.Encrypted {
urlPath = strings.TrimSuffix(urlPath, s.encryption.EncryptedSuffix())
}

format := external.Format
if format == ArchiveFormatUnknown {
format = GuessArchiveFormat(urlPath, data)
}

return data, format, nil
}

// readExternalArchiveFile reads a file from an external archive and returns its
// SourceStateEntries.
func (s *SourceState) readExternalArchiveFile(
ctx context.Context,
externalRelPath RelPath,
parentSourceRelPath SourceRelPath,
external *External,
options *ReadOptions,
) (map[RelPath][]SourceStateEntry, error) {
if external.ArchivePath == "" {
return nil, fmt.Errorf("%s: missing path", externalRelPath)
}

data, format, err := s.readExternalArchiveData(ctx, externalRelPath, external, options)
if err != nil {
return nil, err
}

var sourceStateEntry SourceStateEntry
if err := WalkArchive(data, format, func(name string, fileInfo fs.FileInfo, r io.Reader, linkname string) error {
if external.StripComponents > 0 {
components := strings.Split(name, "/")
if len(components) <= external.StripComponents {
return nil
}
name = path.Join(components[external.StripComponents:]...)
}
switch {
case name == "":
return nil
case name != external.ArchivePath:
// If this entry is a directory and it cannot contain the file we
// are looking for then skip this directory.
if fileInfo.IsDir() && !strings.HasPrefix(external.ArchivePath, name) {
return fs.SkipDir
}
return nil
case fileInfo.Mode()&fs.ModeType == 0:
contents, err := io.ReadAll(r)
if err != nil {
return fmt.Errorf("%s: %w", name, err)
}
lazyContents := newLazyContents(contents)
fileAttr := FileAttr{
TargetName: fileInfo.Name(),
Type: SourceFileTypeFile,
Empty: fileInfo.Size() == 0,
Executable: isExecutable(fileInfo),
Private: isPrivate(fileInfo),
ReadOnly: isReadOnly(fileInfo),
}
sourceRelPath := parentSourceRelPath.Join(NewSourceRelPath(fileAttr.SourceName(s.encryption.EncryptedSuffix())))
targetStateEntry := &TargetStateFile{
lazyContents: lazyContents,
empty: fileAttr.Empty,
perm: fileAttr.perm() &^ s.umask,
sourceAttr: SourceAttr{
External: true,
},
}
sourceStateEntry = &SourceStateFile{
lazyContents: lazyContents,
Attr: fileAttr,
origin: external,
sourceRelPath: sourceRelPath,
targetStateEntry: targetStateEntry,
}
return Break
case fileInfo.Mode()&fs.ModeType == fs.ModeSymlink:
fileAttr := FileAttr{
TargetName: fileInfo.Name(),
Type: SourceFileTypeSymlink,
}
sourceRelPath := parentSourceRelPath.Join(NewSourceRelPath(fileAttr.SourceName(s.encryption.EncryptedSuffix())))
targetStateEntry := &TargetStateSymlink{
lazyLinkname: newLazyLinkname(linkname),
sourceAttr: SourceAttr{
External: true,
},
}
sourceStateEntry = &SourceStateFile{
Attr: fileAttr,
origin: external,
sourceRelPath: sourceRelPath,
targetStateEntry: targetStateEntry,
}
return Break
default:
return fmt.Errorf("%s: unsupported mode %o", name, fileInfo.Mode()&fs.ModeType)
}
}); err != nil {
return nil, err
}
if sourceStateEntry == nil {
return nil, fmt.Errorf("%s: path not found in %s", external.ArchivePath, external.URL)
}

return map[RelPath][]SourceStateEntry{
externalRelPath: {sourceStateEntry},
}, nil
}

// ReadExternalDir returns all source state entries in an external_ dir.
func (s *SourceState) readExternalDir(
rootSourceAbsPath AbsPath, rootSourceRelPath SourceRelPath, rootTargetRelPath RelPath,
Expand Down
12 changes: 12 additions & 0 deletions pkg/cmd/testdata/scripts/external.txtar
View file
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,12 @@ chhome home11/user
stderr 'MD5 mismatch'
stderr 'SHA256 mismatch'

chhome home12/user

# test that chezmoi reads archive-file externals
exec chezmoi apply
cmp $HOME/.file golden/dir/file

-- archive/dir/file --
# contents of dir/file
-- golden/.file --
Expand Down Expand Up @@ -130,6 +136,12 @@ stderr 'SHA256 mismatch'
url = "{{ env "HTTPD_URL" }}/.corrupt-file"
checksum.md5 = "49fe9018f97349cdd0a0ac7b7f668b05"
checksum.sha256 = "634a4dd193c7b3b926d2e08026aa81a416fd41cec52854863b974af422495663"
-- home12/user/.local/share/chezmoi/.chezmoiexternal.toml --
[".file"]
type = "archive-file"
url = "{{ env "HTTPD_URL" }}/archive.tar.gz"
path = "dir/file"
stripComponents = 1
-- home2/user/.local/share/chezmoi/.chezmoiexternal.toml --
[".file"]
type = "file"
Expand Down

0 comments on commit 850a848

Please sign in to comment.