Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Overview documentation, documentation guidelines and formating. #2

Open
wants to merge 14 commits into
base: master
Choose a base branch
from
Open

Conversation

prataprc
Copy link
Contributor

@prataprc prataprc commented Mar 7, 2014

No description provided.

Right now it contains a documentation section that developers can use
as guidelines to author secondary index documentation
Index Manager that resides in each index node
Index Coordinator -- only one in set of index nodes
Index Coordinator Replica -- backup nodes for the Index Manager
* Refactored index_manager documentation.
* created projector and pubsub documentation.
ns-codereview pushed a commit that referenced this pull request Jul 11, 2017
Change-Id: Ieaa95c5a0e4a9cf4ab6d9b1f440cde589a114491
ns-codereview pushed a commit that referenced this pull request Jan 5, 2022
Fixes a problem where metadataClient's cached slice of scheduled plus
existing indexes can be stale but it does not notice because the cache-
invalidating changes are not synced with cache invalidation itself. This
could lead to indexes showing as "building" forever in metadataClient
output because it missed a cache-invalidating event and kept returning
a stale cached list of scheduled plus existing indexes.

The replacement algorithm requires all changes to
  1. indexers.allIndexes[] -- all currently existing indexes
  2. schedTokenMon.scheduledIndexes[] -- all currently scheduled indexes
  3. comboIndexes[] -- cached combination of #1 and #2
to be done inside a new mutex, comboIndexesMut. Additionally, the cache
must be invalidated (by setting comboIndexes = nil) inside the same
instance of locking comboIndexesMut as any change to allIndexes or
scheduledIndexes. These behaviors maintain the cache coherency of
comboIndexes by ensuring that whenever it is computed, its content is
in sync with both of its inputs (#1 and #2), and whenever one of its
inputs is changed, comboIndexes gets set back to nil in the same
critical section so cache-invalidating changes can never be missed.

Change-Id: I64ab1a743fec76acb2e969089412039337382f33
ns-codereview pushed a commit that referenced this pull request Jan 12, 2022
1. cpu_throttle.go should use cgroup CPU usage stats when cgroups are
   supported. This is so the usage is correctly calculated against the
   number of CPUs allocated to the cgroup rather than the whole node.
   If not supported fall back to the standard algorithm of CPU ticks in
   different categories that is collected for the whole node.

2. Indexer set number of CPUs to min(node, cgroup, GSI) available CPUs,
   where cgroup is sigar num_cpu_prc when cgroups are supported and GSI
   is config value indexer.settings.max_cpu_percent (exposed on UI as
   Advanced Index Settings / Indexer Threads).

3. Log message to console if, in #2, GSI > min(node, cgroup) and use the
   latter instead.

4. planner/proxy.go RetrievePlanFromCluster was calling
   getIndexSettings() to set CpuQuota based on runtime.NumCPUs() and
   indexer.settings.max_cpu_percent, but the code in that method was
   incorrect. Replaced with a call to runtime.GOMAXPROCS(0) which will
   now get the logical min(node, cgroup, GSI) per #2 above. (Note that
   CpuQuota is not used in Planner anyway -- its enforcement is
   commented out in planner/planner.go Validate().) This changes
   eliminates REST calls to all remote indexer nodes and a use of
   ClusterInfoCache.

5. Replace direct lookup of indexer.settings.memory_quota with new
   Config.GetIndexerMemoryQuota() method that takes cgroup setting into
   account (sigar memory_max) if cgroups are supported.

Change-Id: I3c906070d78ba51888a6c07c7dd05959fba9b27b
ns-codereview pushed a commit that referenced this pull request Jun 24, 2022
* Adding the print in fillScans and not setExplodePositions to capture
  protoscans received from the query
* Index out of range panic was seen in setExplodedPositions function
  when accessing explodePositions array of size len(IndexInst.Defn.SecExprs)
* -- panic: runtime error: index out of range [4] with length 4
* Print the scans details and return error than panic
* Example print
  2022-06-17T02:37:02.185+05:30 [Error] SCAN##2 ReqID: ee9939aa-d17c-4705-9d60-91d773053bbd Defn: 9151033855720878966
	SecExprs: [`age`]
	Scans: <ud>([{ [150] 3 filterRange [{[{ 100 2}]  [100] 3 } {[{50 150 3}] [50] [150] 3 }] <nil>} {[200] [200] 3 range [{[{200 200 3}] [200] [200] 3 }] <nil>}])</ud>
	MaxCompositeFilters: 1
	ProtoScans:
		 protobuf scan 0 filters:<low:"null" high:"100" inclusion:2 >
		 protobuf scan 1 filters:<low:"50" high:"150" inclusion:3 >
		 protobuf scan 2 filters:<low:"200" high:"200" inclusion:3 >

Change-Id: Ibb017edb6e84af3d963e80f2549e28b69dc82d57
ns-codereview pushed a commit that referenced this pull request Jul 5, 2022
…eFilters > Num SecKeys

* Adding the print in fillScans and not setExplodePositions to capture
  protoscans received from the query
* Index out of range panic was seen in setExplodedPositions function
  when accessing explodePositions array of size len(IndexInst.Defn.SecExprs)
* -- panic: runtime error: index out of range [4] with length 4
* Print the scans details and return error than panic
* Example print
  2022-06-17T02:37:02.185+05:30 [Error] SCAN##2 ReqID: ee9939aa-d17c-4705-9d60-91d773053bbd Defn: 9151033855720878966
	SecExprs: [`age`]
	Scans: <ud>([{ [150] 3 filterRange [{[{ 100 2}]  [100] 3 } {[{50 150 3}] [50] [150] 3 }] <nil>} {[200] [200] 3 range [{[{200 200 3}] [200] [200] 3 }] <nil>}])</ud>
	MaxCompositeFilters: 1
	ProtoScans:
		 protobuf scan 0 filters:<low:"null" high:"100" inclusion:2 >
		 protobuf scan 1 filters:<low:"50" high:"150" inclusion:3 >
		 protobuf scan 2 filters:<low:"200" high:"200" inclusion:3 >


Change-Id: Ibb017edb6e84af3d963e80f2549e28b69dc82d57
(cherry picked from commit 20e09f1)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant