The new version causing the build to fail

[alex-todorov]

Hi there,
since yesterday my scan builds are failing.
After a little debugging it became that the Makefile inside my root directory causing the issue and had to remove it.
I'm not sure what have been changed from 3.7.25 to 3.7.31, but before the it used to work regardless of the Makefile
Can you pls advaise how can we fix this?
Thanks

[newbishme]

hi @alex-todorov , we're working on fixing this.

As an interim solution, instead of removing your Makefile, it is also possible to specify --scan-collectors=maven, for example , so that it will only run those specific collectors. Or, specify --skip-collectors=makefile. See https://help.veracode.com/r/c_sc_scan_directives for more information.

Hope it helps temporarily.

[newbishme]

@alex-todorov Sorry for the late response. We've released a newer version of the agent that fixes this issue, so that a workaround is no longer required. Thanks!

[alex-todorov]

@newbishme thank you for the fast response, I'll test on our side and let you know!

[alex-todorov]

Hi there, just run the test and it's still failing.
this is the command I use:
curl -sSL https://download.sourceclear.com/ci.sh | DEBUG=1 NOCACHE=1 bash

And part of the output I get:
debug: check_and_set_latest_version: retrieved LATEST_VERSION: 3.7.34
debug: check_and_set_latest_version: latest version does not exist and will be downloaded.
debug: download: retrieving srcclr v3.7.34 for linux via https://download.sourceclear.com/srcclr-3.7.34-linux.tgz...
debug: download: retrieved in 1s.
debug: extract: version not extracted; continuing.
debug: extract: archive "/tmp/srcclr-3.7.34-linux.tgz" found
debug: extract: "/tmp/srcclr" created
debug: extract: extracting srcclr...
debug: extract: extraction complete in 1s.
debug: run_scan entry:
debug: run_scan: running "/tmp/srcclr/bin/srcclr" --debug scan --allow-dirty
2021-06-21/13:58:25.032 io.sentry.SentryClient DEBUG Adding 'com.sourceclear.agent.services.ErrorRecordServiceImpl$$Lambda$47/0x00000001001d4440@29df4d43' to the list of builder helpers.
2021-06-21/13:58:25.129 com.sourceclear.agent.util.TrustManagerFromEnv DEBUG Skipping additional SSL Trust Manager configuration because both $SRCCLR_SSL_CERT_DIR and $SRCCLR_SSL_CERT_FILE are blank
2021-06-21/13:58:25.133 com.sourceclear.agent.EntryPointImpl DEBUG Executing scan with com.sourceclear.agent.commands.ScanCommand
2021-06-21/13:58:25.134 com.sourceclear.agent.EntryPointImpl DEBUG Veracode SCA Agent/3.7.34 (Azul Systems, Inc./11.0.2+9-LTS; arch="amd64" name="Linux" version="4.15.0-1102-aws" language="en" timezone="Etc/UTC")
2021-06-21/13:58:25.138 com.sourceclear.agent.EntryPointImpl DEBUG Verb Options := verbOpts = {\n KEY:allow-dirty := []\n}\nleft-overs = []
2021-06-21/13:58:25.138 com.sourceclear.agent.services.LicenseServiceImpl INFO Checking Agent authentication against the Veracode SCA API...
2021-06-21/13:58:25.387 com.sourceclear.agent.commands.ScanCommand INFO Srcclr is scanning the folder: /home/circleci/app
2021-06-21/13:58:25.388 com.sourceclear.agent.commands.ScanCommand DEBUG Scan collectors:
2021-06-21/13:58:25.388 com.sourceclear.agent.commands.ScanCommand DEBUG Skip collectors:
2021-06-21/13:58:25.388 com.sourceclear.agent.commands.ScanCommand DEBUG Skip vulnerable methods: false
2021-06-21/13:58:25.390 com.sourceclear.agent.commands.ScanCommand DEBUG Setup completed in 0s
2021-06-21/13:58:25.393 com.sourceclear.agent.services.ScanServiceImpl DEBUG scanDir : /home/circleci/app
2021-06-21/13:58:25.393 com.sourceclear.agent.services.ScanServiceImpl DEBUG scanDir is directory: true
2021-06-21/13:58:25.414 com.sourceclear.util.io.GitUtils DEBUG SRCCLR_GIT_CEILING_DIR=Optional.empty
2021-06-21/13:58:28.530 com.sourceclear.agent.services.ScanServiceImpl DEBUG MetaGit: MetaGit[localPath=file:/home/circleci/app/, subPath='', remote=ssh://[email protected]/progyny/api.git, head='e4159789e9b39824c2c686d598d7f9ff23d12abd', refName='feature/veracode', refType=BRANCH]
2021-06-21/13:58:28.531 com.sourceclear.agent.services.ScanServiceImpl DEBUG Include Dockerfile scan: false
2021-06-21/13:58:28.531 com.sourceclear.agent.services.ScanServiceImpl DEBUG Allow scan to proceed without a build system: false
2021-06-21/13:58:28.542 com.sourceclear.engine.component.ComponentEngineBuilder DEBUG collectorsToRun: []
2021-06-21/13:58:28.542 com.sourceclear.engine.component.ComponentEngineBuilder DEBUG collectorsToSkip: []
2021-06-21/13:58:28.870 com.sourceclear.engine.component.collectors.MakefileNativeCollector DEBUG Found Makefile at: /home/circleci/app/Makefile
2021-06-21/13:58:29.198 com.sourceclear.engine.component.collectors.CollectorUtils DEBUG Found executable for make at /usr/bin/make
2021-06-21/13:58:29.199 com.sourceclear.engine.component.collectors.CollectorUtils DEBUG Found executable for yarn at /usr/local/bin/yarn
Veracode SCA agent scanning engine ready
2021-06-21/13:58:29.201 com.sourceclear.agent.services.ScanServiceImpl DEBUG Beginning dependency analysis
Running the PIP scanner
2021-06-21/13:58:29.203 com.sourceclear.engine.component.NativeLocalEngine DEBUG Collector "PIP" started
2021-06-21/13:58:29.487 com.sourceclear.engine.component.collectors.CollectorUtils DEBUG Found executable for virtualenv at /home/circleci/.pyenv/shims/virtualenv
2021-06-21/13:58:29.487 com.sourceclear.engine.component.collectors.PIPNativeCollector DEBUG Launching process with commands: /home/circleci/.pyenv/shims/virtualenv /tmp/dep-venv3881199375103982783 --quiet --no-download
2021-06-21/13:58:29.488 com.sourceclear.engine.component.collectors.PIPNativeCollector DEBUG Redirecting stdout to tmp file: /tmp/srcclr-12890014076409643644-output.txt
2021-06-21/13:58:30.137 com.sourceclear.engine.component.collectors.CollectorUtils DEBUG Found executable for pip at /home/circleci/app/venv/bin/pip
2021-06-21/13:58:30.138 com.sourceclear.engine.component.collectors.PIPNativeCollector DEBUG Launching process with commands: /tmp/dep-venv3881199375103982783/bin/pip install -r requirements.txt --disable-pip-version-check
2021-06-21/13:58:30.138 com.sourceclear.engine.component.collectors.PIPNativeCollector DEBUG Redirecting stdout to tmp file: /tmp/srcclr-15931370828297003418-output.txt
2021-06-21/13:58:49.013 com.sourceclear.engine.component.collectors.PIPNativeCollector DEBUG Launching process with commands: /tmp/dep-venv3881199375103982783/bin/pip install -r requirements.txt --disable-pip-version-check
2021-06-21/13:58:49.013 com.sourceclear.engine.component.collectors.PIPNativeCollector DEBUG Redirecting stdout to tmp file: /tmp/srcclr-876847076330764814-output.txt
2021-06-21/13:58:50.209 com.sourceclear.engine.component.NativeLocalEngine DEBUG Collector "PIP" completed in 21.0s
2021-06-21/13:58:50.209 com.sourceclear.engine.component.collectors.MakefileNativeCollector DEBUG Found Makefile at: /home/circleci/app/Makefile
Running the Makefile scanner
2021-06-21/13:58:50.210 com.sourceclear.engine.component.NativeLocalEngine DEBUG Collector "Makefile" started
2021-06-21/13:58:50.210 com.sourceclear.engine.component.collectors.MakefileNativeCollector DEBUG makeBin: /usr/bin/make, makeFile: Makefile
2021-06-21/13:58:50.210 com.sourceclear.engine.component.collectors.MakefileNativeCollector DEBUG buildTarget: , cleanTarget: clean
2021-06-21/13:58:50.210 com.sourceclear.engine.component.collectors.MakefileNativeCollector DEBUG Launching process with commands: /usr/bin/make clean
2021-06-21/13:58:50.210 com.sourceclear.engine.component.collectors.MakefileNativeCollector DEBUG Redirecting stdout to tmp file: /tmp/srcclr-4080138255963988755-output.txt
2021-06-21/13:58:50.214 com.sourceclear.engine.component.collectors.MakefileNativeCollector DEBUG Process ended with non-zero output. rc=2. error: make: *** No rule to make target 'clean'. Stop.
Processing results...
2021-06-21/13:58:50.239 com.sourceclear.engine.scan.PlatformScan DEBUG Evidence set is empty after collection.
2021-06-21/13:58:50.295 com.sourceclear.api.client.SourceClearClient DEBUG Skipping post-fact status code check because of retry-mode (but, code=200 status=null)
2021-06-21/13:58:50.361 com.sourceclear.engine.scan.PlatformScan DEBUG Matched in 0 seconds.
Processing results complete
2021-06-21/13:58:50.365 com.sourceclear.agent.EntryPointImpl DEBUG Exiting with 1

com.sourceclear.agent.FatalException: This project does not seem to build.\nBecause of this, Veracode SCA agent cannot scan it. Please ensure that the project compiles prior to scanning.
at com.sourceclear.agent.commands.ScanCommand.handleScanErrors(ScanCommand.java:1059)
at com.sourceclear.agent.commands.ScanCommand.executeRepoScan(ScanCommand.java:834)
at com.sourceclear.agent.commands.ScanCommand.execute(ScanCommand.java:529)
at com.sourceclear.agent.EntryPointImpl.runVerb(EntryPointImpl.java:372)
at com.sourceclear.agent.EntryPointImpl.dispatchVerbOptions(EntryPointImpl.java:321)
at com.sourceclear.agent.EntryPointImpl.apply(EntryPointImpl.java:157)
at com.sourceclear.agent.Main.start(Main.java:116)
at com.sourceclear.agent.Main.main(Main.java:121)
Caused by:
com.sourceclear.engine.common.CollectionException: This project does not seem to build.\nBecause of this, Veracode SCA agent cannot scan it. Please ensure that the project compiles prior to scanning.
at com.sourceclear.engine.component.collectors.CollectorUtils.launchProcessInternal(CollectorUtils.java:353)
at com.sourceclear.engine.component.collectors.CollectorUtils.launchProcess(CollectorUtils.java:266)
at com.sourceclear.engine.component.collectors.MakefileNativeCollector.buildProject(MakefileNativeCollector.java:443)
at com.sourceclear.engine.component.collectors.MakefileNativeCollector.collect(MakefileNativeCollector.java:184)
at com.sourceclear.engine.component.NativeLocalEngine.collect(NativeLocalEngine.java:98)
at com.sourceclear.agent.services.ScanServiceImpl.performScanEx(ScanServiceImpl.java:476)
at com.sourceclear.agent.services.ScanServiceImpl.performScan(ScanServiceImpl.java:320)
at com.sourceclear.agent.commands.ScanCommand.lambda$executeRepoScan$5(ScanCommand.java:842)
at com.sourceclear.agent.commands.ScanCommand.handleScanErrors(ScanCommand.java:1039)
at com.sourceclear.agent.commands.ScanCommand.executeRepoScan(ScanCommand.java:834)
at com.sourceclear.agent.commands.ScanCommand.execute(ScanCommand.java:529)
at com.sourceclear.agent.EntryPointImpl.runVerb(EntryPointImpl.java:372)
at com.sourceclear.agent.EntryPointImpl.dispatchVerbOptions(EntryPointImpl.java:321)
at com.sourceclear.agent.EntryPointImpl.apply(EntryPointImpl.java:157)
at com.sourceclear.agent.Main.start(Main.java:116)
at com.sourceclear.agent.Main.main(Main.java:121)

This project does not seem to build.
Because of this, Veracode SCA agent cannot scan it. Please ensure that the project compiles prior to scanning.

[newbishme]

Ok, this is slightly different from what we have faced. Will take a look and come back to you with an update in a couple of days. Sorry, and thanks for the debug log!

[alex-todorov]

Just one more note, when I run the command with a set version it runs as charm:
curl -sSL https://download.sourceclear.com/ci.sh | DEBUG=1 SRCCLR_VERSION=3.7.25 NOCACHE=1 bash

[newbishme]

For now, please feel free to use curl -sSL https://download.sourceclear.com/ci.sh | DEBUG=1 SRCCLR_SKIP_COLLECTORS=makefile NOCACHE=1 bash to get around this.

[newbishme]

@alex-todorov Would you mind describing a little more about the Makefile in the app root?

Specifically, if the Makefile is cmake-generated, and whether it has any CMAKE string in the header comment.

The issue still persisted even in 3.7.34 because it found the executable Make in the system. It should be possible for us to deduce if this Makefile is to be run to build the application, and to automatically skip it.

[alex-todorov]

@newbishme The SCA run inside CircleCI, which create a docker container and runs the actual scan. We scan Python files though. Regarding the Makefile, we don't use any of the commands inside it for this specific execution/scan. We need the file for diff purposes. The file is manually built.

[newbishme]

cool, thanks!

[newbishme]

@alex-todorov Please try a new verison of the Agent released targeted for this issue: 3.7.38. This would check the makefile for any mentions of compilers such as Cmake before initiating a makefile scan.