Skip to content

Commit

Permalink
Finish architecture section of README; Add missing doc comment in coriam
Browse files Browse the repository at this point in the history
  • Loading branch information
bcongdon committed Apr 27, 2018
1 parent 43703e7 commit 819845b
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 1 deletion.
10 changes: 9 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,10 +201,18 @@ Mappers may maintain state if desired (though not encouraged).

### Partition / Shuffle

Records emitted from a Mapper are partitioned by key.
Key/value pairs emitted during the map stage are written to intermediate files. Keys are partitioned into one `N` buckets, where `N` is the number of reducers. As a result, each mapper may write to as many as `N` separate files.

This results in a set of files labeled `map-binX-Y` where `X` is a number between 0 and N-1, and `Y` is the mapper's ID (a number between 0 and the number of mappers).

### Reducers / Output

Currently, reducer input must be able to fit in memory. This is because keys are only partitioned, not sorted. The reducer performs an in-memory per-key partition.

Reducers receive per-key values in an arbitrary order. It is guaranteed that all values for a given key will be provided in a single call to Reduce by-key.

Values emitted from a reducer will be stored in tab separated format (i.e. `KEY\tVALUE`) in files labeled `output-X` where `X` is the reducer's ID (a number between 0 and the number of reducers).

Reducers may maintain state if desired (though not encouraged).

## Contributing
Expand Down
1 change: 1 addition & 0 deletions internal/pkg/coriam/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,7 @@ func (iamClient *IAMClient) DeployPermissions(roleName string) (roleARN string,
return roleARN, err
}

// DeletePermissions deletes corral's IA role policy and IAM role.
func (iamClient *IAMClient) DeletePermissions(roleName string) error {
log.Debugf("Deleting role policy")
deletePolicyParams := &iam.DeleteRolePolicyInput{
Expand Down

0 comments on commit 819845b

Please sign in to comment.