Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: test_bgp_stress_link_flap case sometime hung #14226

Closed
lipxu opened this issue Aug 23, 2024 · 4 comments
Closed

[Bug]: test_bgp_stress_link_flap case sometime hung #14226

lipxu opened this issue Aug 23, 2024 · 4 comments
Assignees

Comments

@lipxu
Copy link
Contributor

lipxu commented Aug 23, 2024

Issue Description

This is a stress test case, would create sub-process per interface to shut/no shut to flap bgp session.
Sometime the case hung

Results you see

no response

Results you expected to see

case pass

Is it platform specific

generic

Relevant log output

No response

Output of show version

No response

Attach files (if any)

No response

@lipxu lipxu self-assigned this Aug 23, 2024
@lipxu
Copy link
Contributor Author

lipxu commented Aug 26, 2024

Add debug log, check the memory during the test, the available memory is less than 500m during the test after 36s.
The hung most like caused by memory exhausting.

https://elastictest.org/scheduler/testplan/66cbb7e7ad22601e54dacba1?testcase=bgp%2Ftest_bgp_stress_link_flap.py%7C%7C%7C0&type=console

bgp/test_bgp_stress_link_flap.py::test_bgp_stress_link_flap[dut]
-------------------------------- live log call ---------------------------------
23:40:15 test_bgp_stress_link_flap.test_bgp_stres L0316 ERROR | Available memory is less than 500 MB, stopping the test
PASSED [ 25%]
------------------------------ live log teardown -------------------------------
23:40:46 memory_utilization.handle_memory_thresho L0089 WARNING| [ALARM]: monit:memory_usage, Current memory usage 96.0 exceeds high threshold 90.0

Mem: 3846 3542 178 13 390 303
Swap: 0 0 0
25/08/2024 23:40:15 test_bgp_stress_link_flap.test_bgp_stres L0312 INFO | Available memory: 303
25/08/2024 23:40:15 test_bgp_stress_link_flap.test_bgp_stres L0316 ERROR | Available memory is less than 500 MB, stopping the test
25/08/2024 23:40:16 test_bgp_stress_link_flap.test_bgp_stres L0319 INFO | Test running for 35.91473174095154 seconds

@lipxu
Copy link
Contributor Author

lipxu commented Aug 26, 2024

on KVM device, config process took lots of memory
command, args=["sudo config interface startup Ethernet88"]

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND

9718 300 20 0 508808 124592 2536 S 0.0 3.2 0:50.66 bgpd
9345 root 20 0 636584 80004 3728 S 0.0 2.0 0:28.81 orchage+
22332 root 20 0 103564 72928 24632 R 7.5 1.9 0:01.13 config
22358 root 20 0 102540 72900 24604 S 5.0 1.9 0:01.00 config
22362 root 20 0 103568 72880 24584 R 5.0 1.9 0:01.06 config
22346 root 20 0 103564 72832 24532 R 15.0 1.8 0:01.16 config
22354 root 20 0 103568 72800 24504 R 15.0 1.8 0:01.00 config
22363 root 20 0 103552 72800 24488 R 5.0 1.8 0:01.09 config
22330 root 20 0 102544 72640 24340 R 5.0 1.8 0:01.09 config
22329 root 20 0 103564 72628 24340 R 5.0 1.8 0:01.08 config
22383 root 20 0 102396 72432 24652 R 7.5 1.8 0:00.93 config
676 root 20 0 1688480 61056 1344 S 0.0 1.6 0:16.79 dockerd
22404 root 20 0 79808 55556 19196 R 10.0 1.4 0:00.70 config
22474 root 20 0 76520 52828 18996 R 12.5 1.3 0:00.63 config

@lipxu
Copy link
Contributor Author

lipxu commented Aug 27, 2024

On physical device, BGP, swss docker memory usage increased.
But no obviously memory leak issue, will try run more times.

image

image

@lipxu
Copy link
Contributor Author

lipxu commented Sep 2, 2024

PR #14163 merged, close the issue.

@lipxu lipxu closed this as completed Sep 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant