Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Zephyr trace dma #4866

Merged
merged 2 commits into from
Oct 26, 2021
Merged

Zephyr trace dma #4866

merged 2 commits into from
Oct 26, 2021

Conversation

lyakh
Copy link
Collaborator

@lyakh lyakh commented Oct 13, 2021

I'd really like to get these patches from @marc-hb re-applied, let's see if they're still causing problems on CML

From deep down trace.c:va_tracelog() up to the _log_message() level.

Also rename va_tracelog() to the more specific dma_tracelog()

Preparation to support the DMA trace in Zephyr.

The only functional change in this commit is that DMA messages copied to
the shared memory are not de-duplicated any more (a.k.a "adaptive rate
limiting" or CONFIG_TRACE_FILTERING_ADAPTIVE). These are generally
supposed to be high level hence rare enough; otherwise there is probably
a "bigger problem".

Signed-off-by: Marc Herbert <[email protected]>
As of July 2021 we support (too) many tracing options and this
duplication is unfortunately the only way I found to support them all
while giving the compiler the opportunity to optimize away as many
strings as possible.

Supported configurations:

- Systems with limited memory and zero space for full strings, must use
  SOF dictionary only.
- Systems with enough space for all strings to be in memory.
- Anything in between

- Support to duplicate only important message to both the DMA and the
  mailbox (the default)
- CONFIG_TRACEM: supports duplicating ALL messages to both the DMA and
  the mailbox
- CONFIG_TRACEV: supports deleting verbose statements at compile time to
  save space
- CONFIG_TRACE: support turning off all traces at compile-time

- SOF dict trace de-duplication a.k.a. "adaptive filtering"

- Dynamic log levels per component

- Redirection to Zephyr's shared memory tracing that requires full strings.

Signed-off-by: Marc Herbert <[email protected]>
@marc-hb marc-hb requested a review from ranj063 October 13, 2021 16:57
@marc-hb
Copy link
Collaborator

marc-hb commented Oct 13, 2021

Could you add the SHA1 of original commit in the commit message? Like git cherry-pick -x does.

Everything passed in https://sof-ci.01.org/sofpr/PR4866/build10693/devicetest/ even before #4844 was merged. However the reproduction rate of #4558 was variable but never 100% and something extremely strange happened: https://sof-ci.01.org/sofpr/PR4866/build10693/devicetest/?model=CML_HEL_RT5682&testcase=check-suspend-resume-with-capture ran for only TWO suspend/resume cycles whereas most other platforms had THREE suspend/resume cycles... looking into it.

EDIT, mystery solved: in the current PR test plan, this test is configured to run once PER (capture) device.

@ranj063
Copy link
Collaborator

ranj063 commented Oct 13, 2021

@lyakh @marc-hb it will cause problems. You need to take my PR for HD Audio DMA and trace free and rename thi afterwards

@marc-hb
Copy link
Collaborator

marc-hb commented Oct 15, 2021

SOFCI TEST

@marc-hb
Copy link
Collaborator

marc-hb commented Oct 15, 2021

Thanks @fredoh9 for extending the suspend-resume-capture test from 1 cycle per device to 5 cycles per device. After re-running this, the usual hda_dsp_stream_trigger: cmd 5 on dai_link "HDA Analog" (Capture, stream_tag: 2): timeout on STREAM_SD_OFFSET read reported in #4558 appeared again as expected:

https://sof-ci.01.org/sofpr/PR4866/build10718/devicetest/?model=CML_SKU0955_HDA&testcase=check-suspend-resume-with-capture-5

@marc-hb
Copy link
Collaborator

marc-hb commented Oct 15, 2021

These patches exceed the memory available on APL with Zephyr and XCC, I was afraid of that: https://sof-ci.01.org/sofpr/PR4866/build10718/build/

@lyakh lyakh marked this pull request as ready for review October 20, 2021 08:11
@kv2019i kv2019i requested a review from marc-hb October 25, 2021 11:15
@lyakh
Copy link
Collaborator Author

lyakh commented Oct 25, 2021

These patches exceed the memory available on APL with Zephyr and XCC, I was afraid of that: https://sof-ci.01.org/sofpr/PR4866/build10718/build/

@marc-hb #4908 should fix that

@lyakh
Copy link
Collaborator Author

lyakh commented Oct 25, 2021

Extending the suspend resume cycles a little bit also resurrected some old friends:

@marc-hb are any of those bugs caused by this PR or do they exist with upstream / Zephyr + gcc too?

@lgirdwood
Copy link
Member

@lyakh I think we still need the HDA DMA fixes applied to the kernel too before these fails will turn green. I'll restart CI as I suspect these could be merged now.

@lgirdwood
Copy link
Member

SOFCI TEST

@marc-hb
Copy link
Collaborator

marc-hb commented Oct 25, 2021

Nit: can you add one cherry-pick -x references in each commit messages?

#4908 should fix that

#4908 has not been merged but somehow there is enough room right now: https://sof-ci.01.org/sofpr/PR4866/build10825/build/apl_zph.txt

@marc-hb are any of those bugs caused by this PR or do they exist with upstream / Zephyr + gcc too?

No, these bugs been here forever (with XTOS and xcc) whereas these patches you're restoring in this PR were in the main branch for less than a few days.

@marc-hb
Copy link
Collaborator

marc-hb commented Oct 25, 2021

In https://sof-ci.01.org/sofpr/PR4866/build10825/devicetest/?model=CML_RVP_SDW&testcase=check-suspend-resume-with-playback-5 the whole system crashed short after Oct 25 15:46:23. The journalctl ends suddenly (no clean shutdown) and on the next boot:

Oct 25 15:50:10 jf-cml-rvp-sdw-3 systemd-fsck[450]: fsck.fat 4.1 (2017-01-24)
Oct 25 15:50:10 jf-cml-rvp-sdw-3 systemd-fsck[450]: 0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.

@lgirdwood lgirdwood merged commit a03e538 into thesofproject:main Oct 26, 2021
@lyakh lyakh deleted the z-trace-dma branch October 27, 2021 06:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants