Add support to Exploits model #1562

ziadhany · 2024-08-20T12:44:15Z

issue #95

Collect exploit pointers:

exploitdb issue Collect exploit from exploitdb #1529
metasploit issue Add support to metasploit #1539

I think it's best to handle these issues in a single pull request, as they're all closely related to our exploit model.

ziadhany · 2024-09-02T21:42:21Z

Here’s the updated view of the exploits tab. Let me know if any further changes are needed. @DennisClark

CVE-ID: CVE-2020-14871

keshav-space

@ziadhany thanks, see few nits for your considerations.

vulnerabilities/migrations/0064_rename_resources_and_notes_exploit_notes.py

vulnerabilities/improvers/exploitdb.py

vulnerabilities/improvers/metasploit.py

vulnerabilities/improvers/vulnerability_kev.py

keshav-space

@ziadhany some nits in metasploit.py for your consideration. I was wondering if we should rename this pipeline to something like enhance_with_metasploit.py? similar to how we name our scancode.io pipelines https://github.com/aboutcode-org/scancode.io/tree/main/scanpipe/pipelines.

vulnerabilities/pipelines/metasploit.py

keshav-space

@ziadhany few more suggestions for your consideration. How about using enhance_with_exploitdb.py and enhance_with_kev.py as pipeline names?
Since there are model changes, we need to make adjustments here too

vulnerablecode/vulnerabilities/api_extension.py

Lines 106 to 110 in ed17dbd

    
           class V2ExploitSerializer(ModelSerializer): 
        
               class Meta: 
        
                   model = Kev 
        
                   fields = ("description", "required_action", "date_added", "due_date", "resources_and_notes")

.

vulnerabilities/pipelines/exploitdb.py

vulnerabilities/pipelines/vulnerability_kev.py

Set data_source as the header for the exploit table. Squash the migration files into a single file. Add test for exploit-db , metasploit Add a missing migration file Rename resources_and_notes to notes Fix Api test Refactor metasploit , exploitdb , kev improver Rename Kev tab to exploit tab Add support for exploitdb , metasploit, kev Signed-off-by: ziadhany <[email protected]>

Refactor the error handling logic in the code. Signed-off-by: ziadhany <[email protected]>

Address the exploit in the API extension. Signed-off-by: ziadhany <[email protected]>

keshav-space · 2024-09-18T07:22:13Z

@ziadhany please mark the addressed review as Resolve Conversation. Also please avoid force-pushing after the review, as this removes the reviews from the Files Changed tab. Instead we should squash these commits while merging.

ziadhany · 2024-09-20T22:57:06Z

@ziadhany please mark the addressed review as Resolve Conversation. Also please avoid force-pushing after the review, as this removes the reviews from the Files Changed tab. Instead we should squash these commits while merging.

Got it, I try to avoid force-pushing

Remove unused logging module Signed-off-by: ziadhany <[email protected]>

# Conflicts: # vulnerabilities/improvers/__init__.py # vulnerabilities/improvers/vulnerability_kev.py

Set data_source as the header for the exploit table. Squash the migration files into a single file. Add test for exploit-db , metasploit Add a missing migration file Rename resources_and_notes to notes Fix Api test Refactor metasploit , exploitdb , kev improver Rename Kev tab to exploit tab Add support for exploitdb , metasploit, kev Signed-off-by: ziadhany <[email protected]>

Refactor the error handling logic in the code. Signed-off-by: ziadhany <[email protected]>

Address the exploit in the API extension. Signed-off-by: ziadhany <[email protected]>

Remove unused logging module Signed-off-by: ziadhany <[email protected]>

Add pipeline_id for ( kev, metasploit, exploit-db ) Signed-off-by: ziadhany <[email protected]>

# Conflicts: # vulnerabilities/pipelines/enhance_with_exploitdb.py # vulnerabilities/pipelines/enhance_with_kev.py # vulnerabilities/pipelines/enhance_with_metasploit.py

Signed-off-by: ziadhany <[email protected]>

ziadhany · 2024-09-21T16:12:23Z

@keshav-space I've merged all the required changes and hope I didn't overlook anything. Could you please review it once more?

keshav-space · 2024-09-23T15:17:41Z

@keshav-space I've merged all the required changes and hope I didn't overlook anything. Could you please review it once more?

@ziadhany Thanks ++, this looks good already.

There are still some unresolved conversation mostly related to csv, please use csv.DictReader insted of cv.reader that way code is much more readable Add support to Exploits model #1562 (comment), Add support to Exploits model #1562 (comment), Add support to Exploits model #1562 (comment), Add support to Exploits model #1562 (comment), and Add support to Exploits model #1562 (comment).
Also, whenever we catch an exception we should include the entire stack trace in the log to make debugging a bit easier, similar to what we do here

vulnerablecode/vulnerabilities/pipelines/__init__.py

Line 132 in 094c2bf

f"Failed to import advisory: {advisory!r} with error {e!r}:\n{traceback_format_exc()}",

Add missing logs Handle cases of one exploit for multiple vulnerabilities. Signed-off-by: ziadhany <[email protected]>

keshav-space

Thanks @ziadhany, LGTM!

Please run these pipelines locally and paste the logs here.

ziadhany · 2024-09-24T12:56:16Z

Thanks, @keshav-space

The logs are too large to paste directly, so I've compressed them.

logs.zip

keshav-space

@ziadhany Thanks for the logs.

Found these in the log you provided.

INFO 2024-09-24 12:40:10.640 No vulnerability found for aliases []
INFO 2024-09-24 12:40:10.640 No vulnerability found for aliases []
INFO 2024-09-24 12:40:10.640 No vulnerability found for aliases []
INFO 2024-09-24 12:40:10.640 No vulnerability found for aliases []

INFO 2024-09-24 12:40:28.783 No vulnerability found for aliases ['']
INFO 2024-09-24 12:40:28.783 No vulnerability found for aliases ['']
INFO 2024-09-24 12:40:28.783 No vulnerability found for aliases ['']
INFO 2024-09-24 12:40:28.784 No vulnerability found for aliases ['']

IMO we should skip the empty aliases altogether.

vulnerabilities/pipelines/enhance_with_exploitdb.py

vulnerabilities/pipelines/enhance_with_kev.py

vulnerabilities/pipelines/enhance_with_metasploit.py

Remove empty vulnerability_kev.py file Signed-off-by: ziadhany <[email protected]>

ziadhany · 2024-09-24T14:31:52Z

@keshav-space Here are the updated logs:
logs.zip

keshav-space

@ziadhany looking good. I'll run this locally on my side and then merge it.

keshav-space · 2024-09-24T15:02:36Z

vulnerabilities/pipelines/enhance_with_metasploit.py

+            continue
+
+    if not vulnerabilities:
+        logger(f"No vulnerability found for aliases {references}")


Suggested change

logger(f"No vulnerability found for aliases {references}")

logger(f"No vulnerability found for aliases {interesting_references}")

Signed-off-by: ziadhany <[email protected]>

This was referenced Aug 20, 2024

Add support to metasploit #1539

Closed

Collect exploit from exploitdb #1529

Closed

ziadhany force-pushed the vul-exploit branch from add381f to f78f89e Compare September 2, 2024 21:25

ziadhany marked this pull request as ready for review September 2, 2024 21:25

ziadhany requested review from keshav-space and TG1999 September 3, 2024 15:05

keshav-space requested changes Sep 10, 2024

View reviewed changes

ziadhany force-pushed the vul-exploit branch from 67ee99b to 0446fff Compare September 11, 2024 21:20

ziadhany requested a review from keshav-space September 11, 2024 21:22

keshav-space requested changes Sep 16, 2024

View reviewed changes

keshav-space requested changes Sep 17, 2024

View reviewed changes

ziadhany added 3 commits September 17, 2024 17:53

Implement the appropriate LoopProgress progress bar.

c513713

Refactor the error handling logic in the code. Signed-off-by: ziadhany <[email protected]>

Resolve migration conflicts.

fbde42f

Address the exploit in the API extension. Signed-off-by: ziadhany <[email protected]>

ziadhany force-pushed the vul-exploit branch from dfc40d2 to fbde42f Compare September 17, 2024 15:00

ziadhany added 9 commits September 21, 2024 02:28

Add any missing logs message

7165be6

Remove unused logging module Signed-off-by: ziadhany <[email protected]>

Merge remote-tracking branch 'origin/main' into vul-exploit

79973f0

# Conflicts: # vulnerabilities/improvers/__init__.py # vulnerabilities/improvers/vulnerability_kev.py

Implement the appropriate LoopProgress progress bar.

c83facd

Refactor the error handling logic in the code. Signed-off-by: ziadhany <[email protected]>

Resolve migration conflicts.

fe55cf2

Address the exploit in the API extension. Signed-off-by: ziadhany <[email protected]>

Add any missing logs message

b7a31cd

Remove unused logging module Signed-off-by: ziadhany <[email protected]>

Fix migration conflict

14fc920

Add pipeline_id for ( kev, metasploit, exploit-db ) Signed-off-by: ziadhany <[email protected]>

Merge remote-tracking branch 'origin/vul-exploit' into vul-exploit

46acf56

# Conflicts: # vulnerabilities/pipelines/enhance_with_exploitdb.py # vulnerabilities/pipelines/enhance_with_kev.py # vulnerabilities/pipelines/enhance_with_metasploit.py

Remove unwanted migration file

8136e9a

Signed-off-by: ziadhany <[email protected]>

ziadhany requested a review from keshav-space September 21, 2024 16:14

Add log traceback for all the errors.

3b96511

Add missing logs Handle cases of one exploit for multiple vulnerabilities. Signed-off-by: ziadhany <[email protected]>

keshav-space approved these changes Sep 24, 2024

View reviewed changes

keshav-space requested changes Sep 24, 2024

View reviewed changes

vulnerabilities/pipelines/enhance_with_exploitdb.py Show resolved Hide resolved

vulnerabilities/pipelines/enhance_with_kev.py Show resolved Hide resolved

vulnerabilities/pipelines/enhance_with_metasploit.py Outdated Show resolved Hide resolved

Skip empty aliases

5b35552

Remove empty vulnerability_kev.py file Signed-off-by: ziadhany <[email protected]>

ziadhany requested a review from keshav-space September 24, 2024 14:33

keshav-space reviewed Sep 24, 2024

View reviewed changes

Replace references log with interesting_references

ca853fa

Signed-off-by: ziadhany <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support to Exploits model #1562

Add support to Exploits model #1562

ziadhany commented Aug 20, 2024

ziadhany commented Sep 2, 2024 •

edited

Loading

keshav-space left a comment

keshav-space left a comment

keshav-space left a comment

keshav-space commented Sep 18, 2024

ziadhany commented Sep 20, 2024

ziadhany commented Sep 21, 2024

keshav-space commented Sep 23, 2024

keshav-space left a comment

ziadhany commented Sep 24, 2024

keshav-space left a comment

ziadhany commented Sep 24, 2024 •

edited

Loading

keshav-space left a comment

keshav-space Sep 24, 2024

ziadhany Sep 24, 2024

	class V2ExploitSerializer(ModelSerializer):
	class Meta:
	model = Kev
	fields = ("description", "required_action", "date_added", "due_date", "resources_and_notes")

	logger(f"No vulnerability found for aliases {references}")
	logger(f"No vulnerability found for aliases {interesting_references}")

Add support to Exploits model #1562

Are you sure you want to change the base?

Add support to Exploits model #1562

Conversation

ziadhany commented Aug 20, 2024

ziadhany commented Sep 2, 2024 • edited Loading

keshav-space left a comment

Choose a reason for hiding this comment

keshav-space left a comment

Choose a reason for hiding this comment

keshav-space left a comment

Choose a reason for hiding this comment

keshav-space commented Sep 18, 2024

ziadhany commented Sep 20, 2024

ziadhany commented Sep 21, 2024

keshav-space commented Sep 23, 2024

keshav-space left a comment

Choose a reason for hiding this comment

ziadhany commented Sep 24, 2024

keshav-space left a comment

Choose a reason for hiding this comment

ziadhany commented Sep 24, 2024 • edited Loading

keshav-space left a comment

Choose a reason for hiding this comment

keshav-space Sep 24, 2024

Choose a reason for hiding this comment

ziadhany Sep 24, 2024

Choose a reason for hiding this comment

ziadhany commented Sep 2, 2024 •

edited

Loading

ziadhany commented Sep 24, 2024 •

edited

Loading