Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dualtor] Fix flakiness of route/test_static_route.py #14202

Merged
merged 4 commits into from
Sep 19, 2024

Conversation

vkjammala-arista
Copy link
Contributor

Description of PR

Summary:
Fix flakiness of route/test_static_route.py due to packet going to unexpected ToR
Fixes # https://github.com/aristanetworks/sonic-qual.msft/issues/230

Type of change

  • Bug fix
  • Testbed and Framework(new/improvement)
  • Test case(new/improvement)

Back port request

  • 202012
  • 202205
  • 202305
  • 202311
  • 202405

Approach

What is the motivation for this PR?

As part of traffic verification, sometimes packet is going out of wrong ToR and causing test failures with signature something like AssertionError: Received expected packet on port 30 for device 0, but it should have arrived on one of these ports: [52, 0, 36].

Reason being, in case of dualtor-aa, both the duts will be in active state and packet can go out of rand_unselected_dut.

How did you do it?

  1. Added setup_standby_ports_on_rand_unselected_tor fixture to setup ports in standby mode in case of active-active topology. This is needed for packets not to go out of unexpected tor and cause test failures.
  2. Test is performing "config_reload", this can cause switchover (active to standy and viceversa). But rand_selected_dut should be in active state for traffic verification to pass, so after config_reload we
    need to toggle ports to rand_selected_dut.

How did you verify/test it?

Stressed the test on Arista-7260CX3-D108C8 and test is passing.

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Fixes:
1) Adding "setup_standby_ports_on_rand_unselected_tor" fixture to setup
   ports in standby mode in case of active-active topology. This is
   needed for packets not to go out of unexpected tor and cause test
   failures.
2) Test is performing "config_reload", this can cause switchover (active
   to standy and viceversa). But rand_selected_dut should be in active
   state for traffic verification to pass, so after config_reload we
   need to toggle ports to rand_selected_dut.
tests/route/test_static_route.py Show resolved Hide resolved
tests/route/test_static_route.py Outdated Show resolved Hide resolved
@StormLiangMS
Copy link
Collaborator

/azp run Azure.sonic-mgmt

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@vkjammala-arista
Copy link
Contributor Author

Hi @lolyu @StormLiangMS Please review the changes. Thanks!

Copy link
Contributor

@lolyu lolyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@StormLiangMS
Copy link
Collaborator

hi @vkjammala-arista there is a conflict, could you help with?

@vkjammala-arista
Copy link
Contributor Author

hi @vkjammala-arista there is a conflict, could you help with?

Resolved the conflict, thanks!

@StormLiangMS StormLiangMS merged commit 5a631e5 into sonic-net:master Sep 19, 2024
16 checks passed
mssonicbld pushed a commit to mssonicbld/sonic-mgmt that referenced this pull request Sep 19, 2024
* [dualtor] Fix flakiness of route/test_static_route.py

Fixes:
1) Adding "setup_standby_ports_on_rand_unselected_tor" fixture to setup
   ports in standby mode in case of active-active topology. This is
   needed for packets not to go out of unexpected tor and cause test
   failures.
2) Test is performing "config_reload", this can cause switchover (active
   to standy and viceversa). But rand_selected_dut should be in active
   state for traffic verification to pass, so after config_reload we
   need to toggle ports to rand_selected_dut.

* Addressing review comments.

* Reverting minor unintended change.
mssonicbld pushed a commit to mssonicbld/sonic-mgmt that referenced this pull request Sep 19, 2024
* [dualtor] Fix flakiness of route/test_static_route.py

Fixes:
1) Adding "setup_standby_ports_on_rand_unselected_tor" fixture to setup
   ports in standby mode in case of active-active topology. This is
   needed for packets not to go out of unexpected tor and cause test
   failures.
2) Test is performing "config_reload", this can cause switchover (active
   to standy and viceversa). But rand_selected_dut should be in active
   state for traffic verification to pass, so after config_reload we
   need to toggle ports to rand_selected_dut.

* Addressing review comments.

* Reverting minor unintended change.
@mssonicbld
Copy link
Collaborator

Cherry-pick PR to 202405: #14645

@mssonicbld
Copy link
Collaborator

Cherry-pick PR to 202311: #14646

mssonicbld pushed a commit that referenced this pull request Sep 20, 2024
* [dualtor] Fix flakiness of route/test_static_route.py

Fixes:
1) Adding "setup_standby_ports_on_rand_unselected_tor" fixture to setup
   ports in standby mode in case of active-active topology. This is
   needed for packets not to go out of unexpected tor and cause test
   failures.
2) Test is performing "config_reload", this can cause switchover (active
   to standy and viceversa). But rand_selected_dut should be in active
   state for traffic verification to pass, so after config_reload we
   need to toggle ports to rand_selected_dut.

* Addressing review comments.

* Reverting minor unintended change.
hdwhdw pushed a commit to hdwhdw/sonic-mgmt that referenced this pull request Sep 20, 2024
* [dualtor] Fix flakiness of route/test_static_route.py

Fixes:
1) Adding "setup_standby_ports_on_rand_unselected_tor" fixture to setup
   ports in standby mode in case of active-active topology. This is
   needed for packets not to go out of unexpected tor and cause test
   failures.
2) Test is performing "config_reload", this can cause switchover (active
   to standy and viceversa). But rand_selected_dut should be in active
   state for traffic verification to pass, so after config_reload we
   need to toggle ports to rand_selected_dut.

* Addressing review comments.

* Reverting minor unintended change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants