Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize sleep in test/systemtests #1037

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

Optimize sleep in test/systemtests #1037

wants to merge 10 commits into from

Conversation

vhosakot
Copy link
Member

@vhosakot vhosakot commented Oct 30, 2017

The Contiv system tests in netplugin/test/systemtests call time.Sleep() 151 times in code, many of them in loops. Due to this, the time needed to run these system tests is high. Netplugin CI needs 2.5 hours to run them.

This PR optimizes and reduces the sleeps in system tests' code in the following areas:

  1. Add polling code in contivModelClient.go to poll for the status of the objects AppProfile, EndpointGroup, Global, Network, Rule after creating (httpPost) and deleting (httpDelete) them.
  2. Remove sleep in the tests and poll for the status of objects AppProfile, EndpointGroup, Global, Network, Rule after creating and deleting them.
  3. Remove unwanted sleep in code.
  4. Reduce sleep in code wherever needed.
  5. Optimize polling interval in code when checking things like:
    • Node is reloaded
    • Etcd and consul are restarted
    • Interface is flapped
    • VTEPs are verified
    • ACI gateway is restarted
    • Checking for the status of Netplugin after starting and stopping it
    • k8s pods are deleted

Signed-off-by: Vikram Hosakote [email protected]

@vhosakot
Copy link
Member Author

build PR

1 similar comment
@unclejack
Copy link
Contributor

build PR

@contiv contiv deleted a comment from contivbot Oct 30, 2017
@unclejack
Copy link
Contributor

build PR

5 similar comments
@unclejack
Copy link
Contributor

build PR

@vhosakot
Copy link
Member Author

build PR

@vhosakot
Copy link
Member Author

build PR

@unclejack
Copy link
Contributor

build PR

@vhosakot
Copy link
Member Author

build PR

@dseevr
Copy link
Contributor

dseevr commented Oct 30, 2017

@vhosakot contivModelClient.go changes can't be made under vendor. You will need to make edits in the contivmodel repo or possibly back in the modelgen repo and vendor them in

@vhosakot
Copy link
Member Author

build PR

@@ -510,19 +508,15 @@ func (s *systemtestSuite) TestBgpTriggerNetpluginRestart(c *C) {
for _, node := range s.nodes {
var err error
c.Assert(node.stopNetplugin(), IsNil)
logrus.Info("Sleeping for a while to wait for netplugin's TTLs to expire")
time.Sleep(1 * time.Minute)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This requires some checks to ensure whatever is meant to expire has expired. We can't remove these two sleep calls without ensuring the relevant TTL has expired. The BGP tests run into some failures right now.

@@ -626,8 +620,7 @@ func (s *systemtestSuite) TestBgpMultiTrigger(c *C) {
)
iter++
c.Assert(nodeToStop.stopNetplugin(), IsNil)
logrus.Info("Sleeping for a while to wait for netplugin's TTLs to expire")
time.Sleep(2 * time.Minute)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This requires some checks to ensure the TTL has expired as well.

@@ -735,12 +730,10 @@ func (s systemtestSuite) testServiceTriggerNetpluginRestart(c *C, encap string)
}
}
c.Assert(node.stopNetplugin(), IsNil)
logrus.Info("Sleeping for a while to wait for netplugin's TTLs to expire")
time.Sleep(2 * time.Minute)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is similar - it needs some checks to ensure the TTLs have expired.

@vhosakot
Copy link
Member Author

build PR

3 similar comments
@vhosakot
Copy link
Member Author

vhosakot commented Nov 1, 2017

build PR

@vhosakot
Copy link
Member Author

vhosakot commented Nov 1, 2017

build PR

@vhosakot
Copy link
Member Author

vhosakot commented Nov 1, 2017

build PR

@vhosakot
Copy link
Member Author

vhosakot commented Nov 2, 2017

build PR

@vhosakot
Copy link
Member Author

vhosakot commented Nov 2, 2017

build PR

Signed-off-by: Vikram Hosakote <[email protected]>
@vhosakot
Copy link
Member Author

vhosakot commented Nov 2, 2017

build PR

Copy link
Contributor

@tiewei tiewei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the poller logic should be in testing codes instead of client codes, and also can be a function with a func pointer (of get method, if possible) passed as a parameter, this would save some efforts on adding such function for each object.
also the checkListener loops, inspect loops might be all benefits from such util func

@vhosakot
Copy link
Member Author

Since this has taken a long time, moving it to backlog, and I'm working on other high-priority Contiv tickets.

Please do not close this PR. I'll work on it later when this PR gets higher priority.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants