Skip to content

Commit

Permalink
Add --include/-y to tile-join (#86)
Browse files Browse the repository at this point in the history 
* Add --include/-y to tile-join

* Update documentation and changelog
  • Loading branch information
@e-n-f
e-n-f authored May 4, 2023
1 parent 5bd46e7 commit 605640e
Show file tree
Hide file tree
Showing 5 changed files with 29 additions and 13 deletions.
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# 2.25.0

* Add `--include`/`-y` option to tile-join

# 2.24.0

* Add --cluster-maxzoom option to limit zoom levels that receive clustering
Expand Down
3 changes: 2 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -784,8 +784,9 @@ The options are:

### Filtering features and feature attributes

* `-x` *key* or `--exclude=`*key*: Remove attributes of type *key* from the output. You can use this to remove the field you are matching against if you no longer need it after joining, or to remove any other attributes you don't want.
* `-x` *key* or `--exclude=`*key*: Remove attributes named *key* from the output. You can use this to remove the field you are matching against if you no longer need it after joining, or to remove any other attributes you don't want. You can use multiple `-x` options to remove multiple attributes.
* `-X` or `--exclude-all`: Remove all attributes from the output.
* `-y` *key* or `--include=`*key*: Remove all attributes except for those named *key* from the output. You can use multiple `-y` options to retain multiple attributes.
* `-i` or `--if-matched`: Only include features that matched the CSV.
* `-j` *filter* or `--feature-filter`=*filter*: Check features against a per-layer filter (as defined in the [Mapbox GL Style Specification](https://docs.mapbox.com/mapbox-gl-js/style-spec/#other-filter)) and only include those that match. Any features in layers that have no filter specified will be passed through. Filters for the layer `"*"` apply to all layers.
* `-J` *filter-file* or `--feature-filter-file`=*filter-file*: Like `-j`, but read the filter from a file.
Expand Down
4 changes: 3 additions & 1 deletion man/tippecanoe.1
View file
Original file line number Diff line number Diff line change
Expand Up @@ -992,10 +992,12 @@ The options are:
.SS Filtering features and feature attributes
.RS
.IP \(bu 2
\fB\fC\-x\fR \fIkey\fP or \fB\fC\-\-exclude=\fR\fIkey\fP: Remove attributes of type \fIkey\fP from the output. You can use this to remove the field you are matching against if you no longer need it after joining, or to remove any other attributes you don't want.
\fB\fC\-x\fR \fIkey\fP or \fB\fC\-\-exclude=\fR\fIkey\fP: Remove attributes named \fIkey\fP from the output. You can use this to remove the field you are matching against if you no longer need it after joining, or to remove any other attributes you don't want. You can use multiple \fB\fC\-x\fR options to remove multiple attributes.
.IP \(bu 2
\fB\fC\-X\fR or \fB\fC\-\-exclude\-all\fR: Remove all attributes from the output.
.IP \(bu 2
\fB\fC\-y\fR \fIkey\fP or \fB\fC\-\-include=\fR\fIkey\fP: Remove all attributes except for those named \fIkey\fP from the output. You can use multiple \fB\fC\-y\fR options to retain multiple attributes.
.IP \(bu 2
\fB\fC\-i\fR or \fB\fC\-\-if\-matched\fR: Only include features that matched the CSV.
.IP \(bu 2
\fB\fC\-j\fR \fIfilter\fP or \fB\fC\-\-feature\-filter\fR=\fIfilter\fP: Check features against a per\-layer filter (as defined in the Mapbox GL Style Specification \[la]https://docs.mapbox.com/mapbox-gl-js/style-spec/#other-filter\[ra]) and only include those that match. Any features in layers that have no filter specified will be passed through. Filters for the layer \fB\fC"*"\fR apply to all layers.
Expand Down
29 changes: 19 additions & 10 deletions tile-join.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ void aprintf(std::string *buf, const char *format, ...) {
free(tmp);
}

void handle(std::string message, int z, unsigned x, unsigned y, std::map<std::string, layermap_entry> &layermap, std::vector<std::string> &header, std::map<std::string, std::vector<std::string>> &mapping, std::set<std::string> &exclude, std::set<std::string> &keep_layers, std::set<std::string> &remove_layers, int ifmatched, mvt_tile &outtile, json_object *filter) {
void handle(std::string message, int z, unsigned x, unsigned y, std::map<std::string, layermap_entry> &layermap, std::vector<std::string> &header, std::map<std::string, std::vector<std::string>> &mapping, std::set<std::string> &exclude, std::set<std::string> &include, std::set<std::string> &keep_layers, std::set<std::string> &remove_layers, int ifmatched, mvt_tile &outtile, json_object *filter) {
mvt_tile tile;
int features_added = 0;
bool was_compressed;
Expand Down Expand Up @@ -225,7 +225,7 @@ void handle(std::string message, int z, unsigned x, unsigned y, std::map<std::st
continue;
}

if (!exclude_all && exclude.count(std::string(key)) == 0 && exclude_attributes.count(std::string(key)) == 0) {
if (include.count(std::string(key)) || (!exclude_all && exclude.count(std::string(key)) == 0 && exclude_attributes.count(std::string(key)) == 0)) {
type_and_string tas;
tas.type = type;
tas.string = value;
Expand Down Expand Up @@ -258,7 +258,7 @@ void handle(std::string message, int z, unsigned x, unsigned y, std::map<std::st

const char *sjoinkey = joinkey.c_str();

if (!exclude_all && exclude.count(joinkey) == 0 && exclude_attributes.count(joinkey) == 0 && attr_type != mvt_null) {
if (include.count(joinkey) || (!exclude_all && exclude.count(joinkey) == 0 && exclude_attributes.count(joinkey) == 0 && attr_type != mvt_null)) {
mvt_value outval;
if (attr_type == mvt_string) {
outval.type = mvt_string;
Expand Down Expand Up @@ -512,6 +512,7 @@ struct arg {
std::vector<std::string> *header = NULL;
std::map<std::string, std::vector<std::string>> *mapping = NULL;
std::set<std::string> *exclude = NULL;
std::set<std::string> *include = NULL;
std::set<std::string> *keep_layers = NULL;
std::set<std::string> *remove_layers = NULL;
int ifmatched = 0;
Expand All @@ -525,7 +526,7 @@ void *join_worker(void *v) {
mvt_tile tile;

for (size_t i = 0; i < ai->second.size(); i++) {
handle(ai->second[i], ai->first.z, ai->first.x, ai->first.y, *(a->layermap), *(a->header), *(a->mapping), *(a->exclude), *(a->keep_layers), *(a->remove_layers), a->ifmatched, tile, a->filter);
handle(ai->second[i], ai->first.z, ai->first.x, ai->first.y, *(a->layermap), *(a->header), *(a->mapping), *(a->exclude), *(a->include), *(a->keep_layers), *(a->remove_layers), a->ifmatched, tile, a->filter);
}

ai->second.clear();
Expand Down Expand Up @@ -560,7 +561,7 @@ void *join_worker(void *v) {
return NULL;
}

void handle_tasks(std::map<zxy, std::vector<std::string>> &tasks, std::vector<std::map<std::string, layermap_entry>> &layermaps, sqlite3 *outdb, const char *outdir, std::vector<std::string> &header, std::map<std::string, std::vector<std::string>> &mapping, std::set<std::string> &exclude, int ifmatched, std::set<std::string> &keep_layers, std::set<std::string> &remove_layers, json_object *filter) {
void handle_tasks(std::map<zxy, std::vector<std::string>> &tasks, std::vector<std::map<std::string, layermap_entry>> &layermaps, sqlite3 *outdb, const char *outdir, std::vector<std::string> &header, std::map<std::string, std::vector<std::string>> &mapping, std::set<std::string> &exclude, std::set<std::string> &include, int ifmatched, std::set<std::string> &keep_layers, std::set<std::string> &remove_layers, json_object *filter) {
pthread_t pthreads[CPUS];
std::vector<arg> args;

Expand All @@ -571,6 +572,7 @@ void handle_tasks(std::map<zxy, std::vector<std::string>> &tasks, std::vector<st
args[i].header = &header;
args[i].mapping = &mapping;
args[i].exclude = &exclude;
args[i].include = &include;
args[i].keep_layers = &keep_layers;
args[i].remove_layers = &remove_layers;
args[i].ifmatched = ifmatched;
Expand Down Expand Up @@ -706,7 +708,7 @@ void handle_vector_layers(json_object *vector_layers, std::map<std::string, laye
}
}

void decode(struct reader *readers, std::map<std::string, layermap_entry> &layermap, sqlite3 *outdb, const char *outdir, struct stats *st, std::vector<std::string> &header, std::map<std::string, std::vector<std::string>> &mapping, std::set<std::string> &exclude, int ifmatched, std::string &attribution, std::string &description, std::set<std::string> &keep_layers, std::set<std::string> &remove_layers, std::string &name, json_object *filter, std::map<std::string, std::string> &attribute_descriptions, std::string &generator_options, std::vector<strategy> *strategies) {
void decode(struct reader *readers, std::map<std::string, layermap_entry> &layermap, sqlite3 *outdb, const char *outdir, struct stats *st, std::vector<std::string> &header, std::map<std::string, std::vector<std::string>> &mapping, std::set<std::string> &exclude, std::set<std::string> &include, int ifmatched, std::string &attribution, std::string &description, std::set<std::string> &keep_layers, std::set<std::string> &remove_layers, std::string &name, json_object *filter, std::map<std::string, std::string> &attribute_descriptions, std::string &generator_options, std::vector<strategy> *strategies) {
std::vector<std::map<std::string, layermap_entry>> layermaps;
for (size_t i = 0; i < CPUS; i++) {
layermaps.push_back(std::map<std::string, layermap_entry>());
Expand Down Expand Up @@ -761,7 +763,7 @@ void decode(struct reader *readers, std::map<std::string, layermap_entry> &layer

if (readers == NULL || readers->zoom != r->zoom || readers->x != r->x || readers->y != r->y) {
if (tasks.size() > 100 * CPUS) {
handle_tasks(tasks, layermaps, outdb, outdir, header, mapping, exclude, ifmatched, keep_layers, remove_layers, filter);
handle_tasks(tasks, layermaps, outdb, outdir, header, mapping, exclude, include, ifmatched, keep_layers, remove_layers, filter);
tasks.clear();
}
}
Expand Down Expand Up @@ -827,7 +829,7 @@ void decode(struct reader *readers, std::map<std::string, layermap_entry> &layer
st->minlat2 = min(minlat, st->minlat2);
st->maxlat2 = max(maxlat, st->maxlat2);

handle_tasks(tasks, layermaps, outdb, outdir, header, mapping, exclude, ifmatched, keep_layers, remove_layers, filter);
handle_tasks(tasks, layermaps, outdb, outdir, header, mapping, exclude, include, ifmatched, keep_layers, remove_layers, filter);
layermap = merge_layermaps(layermaps);

struct reader *next;
Expand Down Expand Up @@ -974,7 +976,7 @@ void decode(struct reader *readers, std::map<std::string, layermap_entry> &layer
}

void usage(char **argv) {
fprintf(stderr, "Usage: %s [-f] [-i] [-pk] [-pC] [-c joins.csv] [-X] [-x exclude ...] [-r inputfile.txt ] -o new.mbtiles source.mbtiles ...\n", argv[0]);
fprintf(stderr, "Usage: %s [-f] [-i] [-pk] [-pC] [-c joins.csv] [-X] [-x exclude ...] [-y include ...] [-r inputfile.txt ] -o new.mbtiles source.mbtiles ...\n", argv[0]);
exit(EXIT_ARGS);
}

Expand Down Expand Up @@ -1004,6 +1006,7 @@ int main(int argc, char **argv) {
std::map<std::string, std::vector<std::string>> mapping;

std::set<std::string> exclude;
std::set<std::string> include;
std::set<std::string> keep_layers;
std::set<std::string> remove_layers;

Expand All @@ -1021,6 +1024,7 @@ int main(int argc, char **argv) {
{"csv", required_argument, 0, 'c'},
{"exclude", required_argument, 0, 'x'},
{"exclude-all", no_argument, 0, 'X'},
{"include", required_argument, 0, 'y'},
{"layer", required_argument, 0, 'l'},
{"exclude-layer", required_argument, 0, 'L'},
{"quiet", no_argument, 0, 'q'},
Expand Down Expand Up @@ -1142,6 +1146,11 @@ int main(int argc, char **argv) {
exclude_all = true;
break;

case 'y':
exclude_all = true;
include.insert(std::string(optarg));
break;

case 'l':
keep_layers.insert(std::string(optarg));
break;
Expand Down Expand Up @@ -1274,7 +1283,7 @@ int main(int argc, char **argv) {
std::string generator_options;
std::vector<strategy> strategies;

decode(readers, layermap, outdb, out_dir, &st, header, mapping, exclude, ifmatched, attribution, description, keep_layers, remove_layers, name, filter, attribute_descriptions, generator_options, &strategies);
decode(readers, layermap, outdb, out_dir, &st, header, mapping, exclude, include, ifmatched, attribution, description, keep_layers, remove_layers, name, filter, attribute_descriptions, generator_options, &strategies);

if (set_attribution.size() != 0) {
attribution = set_attribution;
Expand Down
2 changes: 1 addition & 1 deletion version.hpp
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#ifndef VERSION_HPP
#define VERSION_HPP

#define VERSION "v2.24.0"
#define VERSION "v2.25.0"

#endif

0 comments on commit 605640e

Please sign in to comment.