Update docs (#20)

* Update docs

* Update usercases

* Update .gitignore

.gitignore

@@ -573,3 +573,5 @@ cython_debug/
 duetector-dbcollector.sqlite3*
 dev-tools/duetector-dbcollector.sqlite3*
 dev-tools/config.toml
+docs/usercases/tracking-mljob-in-kata-containers/cifar-10-batches-py/*
+docs/usercases/tracking-mljob-in-kata-containers/cifar-10-python.tar.gz

README.md

@@ -1,4 +1,4 @@
-<h2 align="center">duetector🔍: 基于eBPF的数据使用探测器</h2>
+<h2 align="center">duetector🔍: 支持eBPF的可扩展数据使用探测器</h2>
 <p align="center">
 <a href="https://github.com/hitsz-ids/duetector/actions"><img alt="Actions Status" src="https://github.com/hitsz-ids/duetector/actions/workflows/python-package.yml/badge.svg"></a>
 <a href="https://results.pre-commit.ci/latest/github/hitsz-ids/duetector/main"><img alt="pre-commit.ci status" src="https://results.pre-commit.ci/badge/github/hitsz-ids/duetector/main.svg"></a>
@@ -21,11 +21,11 @@
 >
 > [深入了解并部署DataUCON](https://github.com/hitsz-ids/dataucon)
 
-duetector🔍是一个基于eBPF的数据使用探测器，它可以在Linux内核中对数据使用行为进行探测，从而为数据使用控制提供支持。
+duetector🔍是一个基于可扩展的的数据使用探测器，它可以在Linux内核中对数据使用行为进行探测（基于eBPF），从而为数据使用控制提供支持。
 
 **🐛🐞🧪 项目正在大力开发中，期待任何Bug报告、功能请求、合并请求**
 
-在[ABAUC控制模型](https://github.com/hitsz-ids/dataucon)当中，duetector可作为PIP（Policy Information Point）来获取数据使用行为，从而为PDP（Policy Decision Point）提供数据使用行为的信息。
+在[ABAUC控制模型](https://github.com/hitsz-ids/dataucon)当中，duetector可作为PIP（Policy Information Point）来获取数据使用行为，从而为PDP（Policy Decision Point）提供数据使用行为的信息。[快速了解用户案例](./docs/usercases/)
 
 ## 目录
 
@@ -51,11 +51,14 @@ duetector🔍是一个基于eBPF的数据使用探测器，它可以在Linux内
 - [ ] 基于eBPF的数据使用探测器
   - [X] 文件打开操作
   - [ ] ……
+- [ ] 基于Shell命令的探测器
+  - [X] 内核信息探测
+  - [ ] ……
 - [X] 支持SQL数据库的数据收集器
 - [X] CLI工具
 - [ ] PIP服务
 
-eBPF探测器需要内核支持，详见[内核支持](./docs/kernel_config.md)
+eBPF程序需要内核支持，详见[内核支持](./docs/kernel_config.md)
 
 ## 安装
 
@@ -101,12 +104,27 @@ sudo duectl start
 sudo DUETECTOR_LOG_LEVEL=DEBUG duectl start
 ```
 
-启动时，配置文件将自动生成，对应路径为 `~/.config/duetector` ，可以使用 `--config`指定使用的配置文件
+启动时，配置文件将自动生成，对应路径为 `~/.config/duetector` ，修改这个配置文件可以修改数据库地址等内容，可以使用 `--config`指定使用的配置文件
 
 ```bash
 sudo duectl start --config <config-file-path>
 ```
 
+也支持使用环境变量进行配置:
+
+```bash
+Usage: duectl start [OPTIONS]
+
+  Start A bcc monitor and wait for KeyboardInterrupt
+
+Options:
+  ...
+  --load_env BOOLEAN            Weather load env variables,Prefix: DUETECTOR_,
+                                Separator:__, e.g. DUETECTOR_config__a means
+                                config.a, default: True
+  ...
+```
+
 当使用插件时，默认的配置文件不会包含插件的配置内容，使用动态生成配置的指令生成带有插件配置的配置文件，这个指令也支持合并当前已有的配置文件和环境变量
 
 ```bash
@@ -121,17 +139,18 @@ duectl generate-config
 
 更进一步的，后台运行可以使用 `duectl-daemon start`命令，这将会在后台运行一个守护进程，你可以使用 `duectl-daemon stop`来停止它
 
+使用 `duectl-daemon --help` 获取更多细节：
+
 ```bash
-duectl-daemon --help
 Usage: duectl-daemon [OPTIONS] COMMAND [ARGS]...
 
 Options:
   --help  Show this message and exit.
 
 Commands:
-  start   Start a daemon of command `duectl start`, All arguments after...
-  status  Status of daemon Determined by the existence of pid file in...
-  stop    Stop daemon Determined by the existence of pid file in `workdir`
+  start   Start a background process of command `duectl start`.
+  status  Show status of process.
+  stop    Stop the process.
 ```
 
 更多文档和例子可以在[这里](./docs/)找到。
@@ -146,11 +165,18 @@ WIP 这一部分内容是PIP相关的，目前还没有完成，完成后将包
 
 ## 如何贡献
 
-非常欢迎你的加入！[提一个 Issue](https://github.com/hitsz-ids/duetector/issues/new) 或者提交一个 Pull Request。
+非常欢迎您的加入！[我们欢迎任何类型的Issue](https://github.com/hitsz-ids/duetector/issues/new)，同时也期待您的PR
+
+我们提供了以下资料让您更快了解项目
+
+- 开发环境配置和其他注意事项请参考：[开发者文档](./CONTRIBUTING.md)
+- 在这里了解本项目的设计思路和架构：[设计文档](./docs/design/README.md)
+
+# 如何开发插件
 
-开发环境配置和其他注意事项请参考[开发者文档](./CONTRIBUTING.md)。
+目前，tracer、filter、collector都支持自定义插件开发，以Python包作为单个插件或多个插件，可以查看[自定义插件示例](./examples/)了解开发步骤
 
-在这里了解本项目的设计思路和架构：[设计文档](./docs/design/README.md)
+TODO: 提供一个插件的cookiecutter模板
 
 ## 许可证
 

README_en.md

@@ -1,4 +1,4 @@
-<h2 align="center">duetector🔍: Data Usage eBPF detector </h2>
+<h2 align="center">duetector🔍: Data Usage Extensible detector </h2>
 <p align="center">
 <a href="https://github.com/hitsz-ids/duetector/actions"><img alt="Actions Status" src="https://github.com/hitsz-ids/duetector/actions/workflows/python-package.yml/badge.svg"></a>
 <a href="https://github.com/hitsz-ids/duetector/blob/main/LICENSE"><img alt="LICENSE" src="https://img.shields.io/github/license/hitsz-ids/duetector"></a>
@@ -15,7 +15,7 @@
 
 > duetector is one of the components in the DataUCON project, which is designed to provide support for data usage control. [Intro DataUCON](https://dataucon.idslab.io/).
 
-duetector🔍 is an eBPF-based data usage control probe that provides support for data usage control by probing for data usage behavior in the Linux kernel.
+duetector🔍 is an extensible data usage control detector that provides support for data usage control by probing for data usage behavior in the Linux kernel(based on eBPF).
 
 **🐛🐞🧪 The project is under heavy development, looking forward to any bug reports, feature requests, pull requests!**
 
@@ -45,11 +45,14 @@ In the [ABAUC control model](https://github.com/hitsz-ids/dataucon), duetector c
 - [ ] eBPF-based data usage probes
   - [X] File Open Operation
   - [ ] ......
+- [ ] Shell command probes
+  - [X] Kernel Information Probe
+  - [ ] ......
 - [X] Data collector with SQL database support
 - [X] CLI Tools
 - [ ] PIP Service
 
-The eBPF probe requires kernel support, see [Kernel Support](./docs/kernel_config.md)
+The eBPF program requires kernel support, see [Kernel Support](./docs/kernel_config.md)
 
 ## Installation
 
@@ -101,6 +104,22 @@ At startup, the configuration file will be automatically generated at `~/.config
 sudo duectl start --config <config-file-path>
 ```
 
+Configuration using environment variables is also supported:
+
+```bash
+Usage: duectl start [OPTIONS]
+
+  Start A bcc monitor and wait for KeyboardInterrupt
+
+Options:
+  ...
+  --load_env BOOLEAN            Weather load env variables,Prefix: DUETECTOR_,
+                                Separator:__, e.g. DUETECTOR_config__a means
+                                config.a, default: True
+  ...
+```
+
+
 When using a plugin, the default configuration file will not contain the plugin's configuration, use the dynamically-generated configuration directive to generate a configuration file with the plugin's configuration, this directive also supports merging existing configuration files and environment variables.
 
 ```bash
@@ -115,18 +134,18 @@ duectl generate-config
 
 Going a step further, running in the background you can use the `duectl-daemon start` command, which will run a daemon in the background, which you can stop using `duectl-daemon stop`
 
-```bash
-$ duectl-daemon --help
+Use `duectl-daemon --help` for more details:
 
+```bash
 Usage: duectl-daemon [OPTIONS] COMMAND [ARGS]...
 
 Options:
   --help  Show this message and exit.
 
 Commands:
-  start   Start a daemon of command `duectl start`, All arguments after...
-  status  Status of daemon Determined by the existence of pid file in...
-  stop    Stop daemon Determined by the existence of pid file in `workdir`
+  start   Start a background process of command `duectl start`.
+  status  Show status of process.
+  stop    Stop the process.
 ```
 
 More documentation and examples can be found [here](. /docs/).

docker/start.sh

@@ -1,6 +1,8 @@
 #!/usr/bin/env bash
+mount -t debugfs debugfs /sys/kernel/debug # enable debugfs
 duectl-daemon start --loglevel=DEBUG
 
 cd /home/application
+# Config user's local path for pip install some scripts
 export PATH=/home/application/.local/bin:$PATH
 exec su "application" -c "jupyter-lab --ip=0.0.0.0 --port=8888 --no-browser"

docs/design/README.md

@@ -6,18 +6,18 @@ Key Components and Features:
 
 - [ ] **HTTP / RPC Server**: PIP Server, providing API for PDP to get data usage information.
 - [ ] **Analyzer**: Analyze data usage information and generate data usage behavior.
-- [x] **CLI**: CLI for administrator to manage duetector.
-- [x] **BccMonitor**: Monitor data usage behavior in kernel space. Use BCC to implement.
-- [ ] **PollingMonitor**: A general monitor for custom command. Polling the output of command.
-- [x] **TracerManager**: Manage tracers, support plugin.
-    - [x] **OpenTracer**: A `bcc` tracer, trace `open` syscall.
-    - [ ] ...
-- [x] **FilterManager**: Manage filters, support plugin.
-    - [x] **DefaultFilter**: Filtering some meaningless information
-    - [ ] ...
-- [x] **CollectorManager**: Manage collectors, support plugin.
-    - [x] **DBCollector**: Collect filted trackings and store them into database.
-    - [ ] ...
+- [X] **CLI**: CLI for administrator to manage duetector.
+- [X] **BccMonitor**: Monitor data usage behavior in kernel space. Use BCC to implement.
+- [X] **ShMonitor**: A general monitor for custom command. Polling the output of command.
+- [X] **TracerManager**: Manage tracers, support plugin.
+  - [X] **OpenTracer**: A `bcc` tracer, trace `open` syscall.
+  - [ ] ...
+- [X] **FilterManager**: Manage filters, support plugin.
+  - [X] **DefaultFilter**: Filtering some meaningless information
+  - [ ] ...
+- [X] **CollectorManager**: Manage collectors, support plugin.
+  - [X] **DBCollector**: Collect filted trackings and store them into database.
+  - [ ] ...
 
 # Data Flow
 

docs/design/src/architecture.drawio

@@ -1,6 +1,6 @@
 <mxfile host="65bd71144e">
     <diagram id="71-AI3Yj6Bn5oummquvE" name="第 1 页">
-        <mxGraphModel dx="658" dy="1562" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="827" pageHeight="1169" math="0" shadow="0">
+        <mxGraphModel dx="575" dy="1417" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="827" pageHeight="1169" math="0" shadow="0">
             <root>
                 <mxCell id="0"/>
                 <mxCell id="1" parent="0"/>
@@ -19,7 +19,7 @@
                 <mxCell id="20" value="TracerManager" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;" parent="1" vertex="1">
                     <mxGeometry x="280" y="390" width="110" height="60" as="geometry"/>
                 </mxCell>
-                <mxCell id="25" value="PollingMonitor" style="rounded=0;whiteSpace=wrap;html=1;dashed=1;fillColor=#fff2cc;strokeColor=#d6b656;" parent="1" vertex="1">
+                <mxCell id="25" value="ShMonitor" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;" parent="1" vertex="1">
                     <mxGeometry x="460" y="310" width="170" height="60" as="geometry"/>
                 </mxCell>
                 <mxCell id="26" value="FilterManager" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;" parent="1" vertex="1">
@@ -76,7 +76,7 @@
                 <mxCell id="56" value="bcc" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#f5f5f5;strokeColor=#666666;fontColor=#333333;" parent="1" vertex="1">
                     <mxGeometry x="285" y="340" width="45" height="25" as="geometry"/>
                 </mxCell>
-                <mxCell id="58" value="Analyzer" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;dashed=1;" vertex="1" parent="1">
+                <mxCell id="58" value="Analyzer" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;dashed=1;" parent="1" vertex="1">
                     <mxGeometry x="640" y="310" width="60" height="140" as="geometry"/>
                 </mxCell>
             </root>