Merge pull request #155 from firesim/dev

Release 1.9 dev->master

.gitignore

@@ -1,5 +1,3 @@
-*.img
-*-bin
 *.swp
 test.log
 wlutil/_command.sh

CHANGELOG.md

@@ -3,6 +3,39 @@
 This changelog follows the format defined here: https://keepachangelog.com/en/1.0.0/
 Versioning follows semantic versioning as described here: https://semver.org/spec/v2.0.0.html
 
+## [1.9.0] - 2021-05-21
+This is largely a maintenance release with a few minor features and a bunch of
+bug fixes. The most significant change is a bump to Linux 5.7rc3. The new
+'firesim-dir' configuration option is also signficant because it enables more
+flexible deployment of FireMarshal in Chipyard and FireSim and is required for
+Chipyard 1.3.0.
+
+### Added
+* A number of new global configuration options were added. This includes
+  specifying where the 'install' command finds firesim (PR #127) and allowing
+  for custom workload searchpaths (PR #140)
+* PR #129 adds a number of new features to help support profiling and debug. In
+  particular, Marshal now saves the raw kernel ELF file including debug symbols
+  (BBL strips DWARF info). It also handles a few handy scripts for running
+  FirePerf tools.
+* PR #141 adds the 'post-bin' script option to workloads. This is primarily
+  useful for building custom drivers but may have other uses.
+
+### Changed
+* PR #139 enables RVC in the kernel by default
+* PR #143 changes the behavior of kernel fragments. Previously, workload kfrags
+  were applied directly on top of the distro's default config. Now kfrags are
+  inherited through the chain of parents.
+* PR #151 bumps us to Linux 5.7rc3
+* PR #148 switches to the SBI serial driver from the sifive uart due to a
+  baud-rate configuration issue. This will be reversed eventually.
+
+### Fixed
+* PR #126 adds detection of changes in overlay directories that were missed before
+* PR #128 fixes an issue when calling FireMarshal inside a makefile
+* PR #133 fixes the --workdir option that had broken (and adds tests so it won't happen again)
+* PR #135 jobs were not including device drivers
+
 ## [1.8.0] - 2020-01-24
 This release introduces user-configurable options for FireMarshal through a
 config file or environment variables. It also relaxes constraints on the

boards/firechip/base-workloads/br-base.json

@@ -2,5 +2,10 @@
   "name" : "br-base",
   "base" : "br",
   "overlay" : "overlay",
-  "linux-config" : "linux-config"
+  "linux-config" : "linux-config",
+  "host-init" : "host-init.sh",
+  "files" : [
+      [ "trigger/start", "/usr/bin/firesim-start-trigger"],
+      [ "trigger/end", "/usr/bin/firesim-end-trigger"]
+  ]
 }

boards/firechip/base-workloads/br-base/host-init.sh

@@ -0,0 +1,3 @@
+#!/bin/sh
+
+exec make -C trigger

boards/firechip/base-workloads/br-base/linux-config

@@ -1,6 +1,12 @@
 
 CONFIG_DEFAULT_HOSTNAME="ucbvax"
 
+#
+# SoC selection
+#
+
+# CONFIG_SOC_SIFIVE is not set
+
 #
 # Kernel Performance Events And Counters
 #
@@ -11,7 +17,7 @@ CONFIG_PERF_EVENTS=y
 #
 CONFIG_RISCV_BASE_PMU=y
 
-CONFIG_RISCV_ISA_C=n
+CONFIG_RISCV_ISA_C=y
 
 #
 # Boot options
@@ -64,6 +70,12 @@ CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y
 
 CONFIG_MAILBOX=y
 
+#
+# Non-8250 serial port support
+#
+
+# CONFIG_SERIAL_SIFIVE is not set
+
 #
 # File systems
 #

boards/firechip/base-workloads/br-base/trigger/.gitignore

@@ -0,0 +1,2 @@
+start
+end

boards/firechip/base-workloads/br-base/trigger/Makefile

@@ -0,0 +1,14 @@
+
+CC := riscv64-unknown-linux-gnu-gcc
+CFLAGS := -nostartfiles
+LDFLAGS := -nostdlib
+
+bin := start end
+
+.PHONY: all
+all: $(bin)
+
+$(bin): %: %.S
+	$(CC) $(CFLAGS) $(LDFLAGS) -o $@ $<
+
+.SUFFIXES:

boards/firechip/base-workloads/br-base/trigger/end.S

@@ -0,0 +1,11 @@
+	.text
+	.global _start
+_start:
+	/* end trigger */
+	addi x0, x2, 0
+	/* exit syscall */
+	li a5, 0
+	li a7, 94
+	ecall
+	/* trap */
+	ebreak

boards/firechip/base-workloads/br-base/trigger/start.S

@@ -0,0 +1,11 @@
+	.text
+	.global _start
+_start:
+	/* start trigger */
+	addi x0, x1, 0
+	/* exit syscall */
+	li a5, 0
+	li a7, 94
+	ecall
+	/* trap */
+	ebreak

boards/firechip/base-workloads/fedora-base/linux-config

@@ -267,12 +267,14 @@ CONFIG_RISCV_PLIC=y
 # CONFIG_SCSI_NETLINK is not set
 # CONFIG_SENSORS_LIS3LV02D is not set
 # CONFIG_SERIAL_8250_FSL is not set
+# CONFIG_SERIAL_SIFIVE is not set
 # CONFIG_SGI_PARTITION is not set
 # CONFIG_SG_SPLIT is not set
 # CONFIG_SH_TIMER_CMT is not set
 # CONFIG_SH_TIMER_MTU2 is not set
 # CONFIG_SH_TIMER_TMU is not set
 # CONFIG_SIMPLE_PM_BUS is not set
+# CONFIG_SOC_SIFIVE is not set
 # CONFIG_SOLARIS_X86_PARTITION is not set
 CONFIG_STP=y
 # CONFIG_STREAM_PARSER is not set

docs/source/Tutorials/custom.rst

@@ -10,12 +10,8 @@ A custom workload takes the form of a workload configuration file, and a
 directory containing any files needed by the workload. Additionally, workloads
 can be based on a parent workload to avoid duplicate work. These files can
 be anywhere you like, but FireMarshal must be able to find the workload
-descriptions. The default search paths for FireMarshal are as follows (in
-order):
-
-    #. Built-in workloads (defined in ``boards/firechip/base-workloads``)
-    #. The directory specified by the ``--workdir`` command-line option
-    #. The current working directory
+descriptions. See :ref:`workload-search-paths` for details on how FireMarshal
+searches for workloads.
 
 We now walk through two examples of building a custom workload. For full
 documentation of every option, see :ref:`workload-config`.
@@ -38,10 +34,7 @@ specifies an existing workload to base off of. FireMarshal will first build
 ``fedora-base.json``, and use a copy of its rootfs for example-fed before
 applying the remaining options. Additionally, if fedora-base.json specifies any
 configuration options that we do not include, we will inherit those (e.g. we
-will use the ``linux-config`` option specified by fedora-base). Notice that we
-do not specify a workload source directory. FireMarshal will look in
-``example-workloads/example-fed/`` for any sources specified in the remaining options
-(you can change this behavior with the :ref:`config-workdir` configuration option).
+will use the ``linux-config`` option specified by fedora-base).
 
 Next come a few options that specify common setup options used by all jobs in
 this workload. The ``overlay`` option specifies a filesystem overlay to copy

docs/source/Tutorials/quickstart.rst

@@ -25,8 +25,8 @@ by building the workload:
 
 .. Note:: The ``base-workloads`` directory is on the default search path for
    FireMarshal workloads. This means that we do not need to provide the full-path
-   to the br-base.json configuration file. For custom workloads, you will need to
-   provide a path to the workload configuration file. 
+   to the br-base.json configuration file. See :ref:`workload-search-paths` for
+   details on how workloads are located.
 
 The first time you build a workload may take a long time (buildroot must
 download and cross-compile a large number of packages), but subsequent builds
@@ -64,7 +64,7 @@ caches intermediate build steps whenever possible.
 Finally, FireMarshal supports installing workloads to the FireSim cycle-exact
 simulator. To do this, you will need to use the FireMarshal that comes with
 `FireSim <https://www.fires.im>`_ or `Chipyard
-<https://chipyard.readthedocs.io/en/latest/>`_. To run a workload in FireSim,
+<https://chipyard.readthedocs.io/en/latest/>`_ (or manually :ref:`configure firesim <config-firesim>`). To run a workload in FireSim,
 you must first install it from FireMarshal:
 
 ::

docs/source/commands.rst

@@ -13,11 +13,14 @@ The base ``marshal`` command provides a number of options that apply to most
 sub-commands. You can also run ``marshal -h`` for the most up-to-date
 documentation.
 
+.. _command-opt-workdir:
+
 ``--workdir``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 By default, FireMarshal will search the same directory as the provided
 configuration file for ``base`` references and the workload source directory.
-This option instructs FireMarshal to look elsewhere for these references.
+This option instructs FireMarshal to look elsewhere for these references. See
+:ref:`workload-search-paths` for details of how workloads are located.
 
 ``-d --nodisk``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -88,7 +91,7 @@ test
 --------------------------------------
 The test command will build and run the workload, and compare its output
 against the ``testing`` specification provided in its configuration. See
-:ref:`firemarshal-config` for details of the testing specification. If jobs
+:ref:`config-testing` for details of the testing specification. If jobs
 are specified, all jobs will be run independently and their outputs will be
 included in the output directory.
 
@@ -104,9 +107,10 @@ specification against a pre-existing output. This allows you to check the
 output of firesim runs against a workload. It is also useful when developing a
 workload test.
 
+.. _command-install:
+
 install
 --------------------------------------
-.. _firemarshal-install:
 
 Creates a firesim workload definition file in ``firesim/deploy/workloads`` with
 all appropriate links to the generated workload. This allows you to launch the

docs/source/internal/Build.rst

@@ -70,14 +70,14 @@ archive by staging several filesystems at ``wlutil/initramfs{disk, nodisk,
 drivers}``:
 
 * ``disk/``: contains a fully-functioning root filesystem with a busybox-based
-environment and an init script that knows to load drivers and look for a disk
-to boot from (either ``/dev/vda`` for qemu or ``/dev/iceblk`` for firesim).
+  environment and an init script that knows to load drivers and look for a disk
+  to boot from (either ``/dev/vda`` for qemu or ``/dev/iceblk`` for firesim).
 * ``nodisk/``: contains just the init script to load drivers (it must be
-combined with a working root filesystem).
+  combined with a working root filesystem).
 * ``drivers/``: contains the platform drivers built earlier.
 * ``devNodes.cpio``: A pre-built archive containing the ``/dev/console`` and
-``/dev/tty`` special files. These require a special procedure to create so we
-only do it once and commit the result.
+  ``/dev/tty`` special files. These require a special procedure to create so we
+  only do it once and commit the result.
 
 Marshal combines the needed initramfs sources in a temporary directory into a
 single cpio archive and configures the kernel to include this archive at boot

docs/source/marshalConfig.rst

@@ -41,6 +41,15 @@ All path-like options are interpreted as relative to the location of the
 configuration file. Size options can be written in human readable form (e.g.
 '4KiB') or simply as the number of bytes (e.g. '4096').
 
+.. _config-workload-dirs:
+
+``workload-dirs``
+^^^^^^^^^^^^^^^^^^^^^
+List of paths to search when looking up workloads (either the target workload,
+or parent workloads). This list is ordered with later entries taking precedence
+of earlier entries. See :ref:`workload-search-paths` for details of how
+workloads are located.
+
 ``board-dir``
 ^^^^^^^^^^^^^^^^^
 Root for default board (platform-specific resources).
@@ -55,6 +64,12 @@ Default linux source. This is intended to override the global default linux
 version, if you need a special kernel for your workload, you should set that in
 your workload configuration directly (see :ref:`workload-linux-src`).
 
+.. _config-firesim:
+
+``firesim-dir``
+^^^^^^^^^^^^^^^^^^^
+Location of the firesim repository to use for the :ref:`command-install` command.
+
 ``pk-dir``
 ^^^^^^^^^^^^^^^^^
 Default proxy-kernel source directory. The RISC-V proxy kernel repository