Skip to content

Commit

Permalink
add docs
Browse files Browse the repository at this point in the history
  • Loading branch information
haobibo committed Feb 13, 2024
1 parent ebfe05c commit b12fc01
Show file tree
Hide file tree
Showing 13 changed files with 316 additions and 369 deletions.
17 changes: 15 additions & 2 deletions .github/workflows/deploy-docusaurus.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ on:
- main # default branch

jobs:
build-and-deploy:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
Expand All @@ -30,6 +30,19 @@ jobs:
with: # 👇 Specify build output path
path: QPod/build

deploy:
needs: build

permissions: # Grant GITHUB_TOKEN the permissions required to make a Pages deployment
pages: write # to deploy to Pages
id-token: write # to verify the deployment originates from an appropriate source

environment: # Deploy to the github-pages environment
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}

runs-on: ubuntu-latest
steps:
- name: Deploy to GitHub Pages
uses: actions/deploy-pages@v2
id: deployment
uses: actions/deploy-pages@v4
121 changes: 121 additions & 0 deletions QPod/docs/intro-cn.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
---
sidebar_position: 02
---

# QPod使用指引

AI/数据科学的瑞士军刀——QPod提供了一站式、开箱即用、可自由定制的,基于容器的、开源AI/数据科学开发、分析工具。

## AI/数据科学的瑞士军刀

QPod将常见的、最新的开放数据科学环境和工具封装成为了docker镜像,你无需进行繁琐的环境配置即可快速开始AI/数据科学的工程,同时能够方便地复现与分享您的研究工作。您可以在QPod中使用Jupyter Notebook/JupyterLab中运行Python, R, OpenJDK, NodeJS, Go, Julia, Octave等语言,QPod也包含了VS Code, R-Studio等工具。使用QPod,您可以:

- 📦 避免繁琐的环境配置、安装过程,QPod已经把常用的、最新的环境和工具封装在容器镜像中,您可以开箱即用;
- 🌍 让您的工作更容易被自己/他人`复现`——QPod让科学研究和数据分析项目成为[可复现的工作流(reproducible pipelines)](https://doi.org/10.1038/d41586-018-07196-1),这能让你更好地和同行[分享你的工作](https://doi.org/10.1038/515151a).
- 🆙 能让你的算法和关键创新更容易横向、纵向扩展,QPod让你把自己在开发环境所作的工作无缝、极其简单地部署到生产环境,如提供RESTful API、进行map/reduce操作等。

![Screenshot of QPod](https://raw.githubusercontent.com/wiki/QPod/qpod-hub/img/QPod-screenshot.webp "Screenshot of QPod")

## QPod包含了什么

QPod封装、整理、维护了一些列的容器镜像,这些镜像包含了常见的开放AI/数据科学语言环境和安装包:Ptyhon, R, OpenJDK, NodeJS, Go, Julia, Octave等,同时封装了Jupyter Notebook / JupyterLab / VS Code / RStudio等IDE让用户便捷地进行开发、交互计算。QPod适用于下面的应用场景:

- 单机使用:在笔记本/台式机/工作站上使用,作为AI/数据科学开发环境;
- 多租户使用:在服务器/集群上供多用户使用,以共享服务器计算资源(如GPU);
- 部署于生产服务:使用这些镜像在生产环境中提供RESTful API或者作为map/reduce等操作的执行单元等。

## 安装使用 `1-2-3-GO`🎉

### 0.安装Docker

Linux (e.g.: Ubuntu LTS) / Windows (>=10) / macOS.

- CPU用户:请安装 docker-ce ( [Linux](https://hub.docker.com/search/?offering=community&type=edition&operating_system=linux) | [macOS](https://download.docker.com/mac/stable/Docker.dmg) | [Windows](https://download.docker.com/win/stable/Docker%20for%20Windows%20Installer.exe) ) 或 [docker-ee](https://hub.docker.com/search/?offering=enterprise&type=edition)
- GPU用户:需使用Linux,在安装Docker之后还需安装[NVIDIA driver](https://github.com/NVIDIA/nvidia-docker/wiki/Frequently-Asked-Questions#how-do-i-install-the-nvidia-driver)与最新版的[nvidia-container-toolkit](https://github.com/NVIDIA/nvidia-docker#quickstart)

### 1.选择你需要的功能包和你的工作目录

- 从项目主页底部的表格(QPod镜像功能列表)中,选择你需要的功能包,如果你的磁盘空间和网速满足要求则推荐CPU用户选择`full`,GPU用户选择`full-cuda`
- 选择你的工作目录,请使用绝对路径(如`/root`,`/User/me`,`D:/work`

### 2.准备下载和启动容器服务

根据你的操作系统运行下方脚本,将其中的`IMG``WORKDIR`改为自己的安装配置。运行前关闭Jupyter以及其他占用8888或9999端口的程序。

#### 大陆地区网络环境特殊提示

由于下载过程需要从DockerHub下载较大的镜像文件,如果您使用的网络环境需要加速,
请尝试使用阿里云提供的镜像:我们已经将镜像同步至了阿里云的北京、杭州镜像仓库(阿里云用户也可以使用VPS网络进一步加速)。

方法为,在下面的`IMG`变量中,加上镜像前缀:

- 北京镜像:`registry.cn-beijing.aliyuncs.com`
- 杭州镜像:`registry.cn-hangzhou.aliyuncs.com`

例如:`IMG="registry.cn-beijing.aliyuncs.com/qpod/base-dev:full"`

#### Linux/macOS用户,在bash或终端中运行

```shell
IMG="qpod/base-dev:latest" # <- 你选择的功能包
WORKDIR="/root" # <- macOS用户建议改为用户目录,如:/Users/your_user_name

docker run -d --restart=always \
--name=QPod \
--hostname=QPod \
-p 8888:8888 -p 9999:9999 \
-v $WORKDIR:/root \
$IMG
sleep 10s && docker logs QPod 2>&1|grep token=
```

⚠️ 使用带有`nvidia-docker`的NVIDIA GPU时请注意:

- 👉 确保Docker版本在19.03以上并在主机上能正常够运行`nvidia-smi`命令
- 👉 在`docker run`指令中增加一项: `--gpus all` (位于`--restart=always`之后,旧nvidia-container版本请使用`--runtime nvidia`)
- 👉 使用`IMG="qpod/qpod:full-cuda"`或其他支持cuda的镜像

#### Windows用户,在[Terminal](https://github.com/microsoft/terminal)或CMD中运行

```cmd
SET IMG="qpod/qpod:full"
SET WORKDIR="D:/work"
docker run -d --restart=always ^
--name=QPod ^
--hostname=QPod ^
-p 8888:8888 9999:9999 ^
-v %WORKDIR%:/root ^
%IMG%
timeout 10 && docker logs QPod 2>&1|findstr token=
```

### 3.等待下载和启动

以上指令会下载docker镜像,建立一个名为QPod的docker容器,并打印出一条含有48位十六进制密钥的URL地址。

### Go! 🎉

复制`?token=`之后的密钥(步骤3中最后打印出的),从你的浏览器打开[http://localhost:8888](http://localhost:8888)[http://ip-address:8888](http://ip-address:8888)并粘贴密钥,即可开始使用。

## 附加信息

### FAQ

关于FAQ系列问题请参见[此处](https://github.com/QPod/docker-images/wiki)

### 硬件

镜像需建立在`ubuntu:latest`且只能在×86平台测试,arm64/ppc64le平台预计进行少量修改之后可以适配。

### 安装包管理

强烈不建议使用`conda`来安装lib或package,鉴于conda无法复用已有的系统库且无法提供稳定、最新的Liunx系统安装包库(例如某些包在`debian:jessie`下构建,在`debian:stretch`下就无法正常使用)。

### 定制化

若发现有系统库、Python模块或R包的缺失,可在`work`文件夹的`install_XX.list`中进行补充。

### 致谢

感谢[Logan](https://github.com/liuhaoge)同学的翻译和建议。
124 changes: 124 additions & 0 deletions QPod/docs/intro-en.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
---
sidebar_position: 01
---

# QPod Tutorial - Quick Start

## Your Swiss Army Knife for AI & Data Science

In a nutshell, `QPod` ( [DockerHub](https://hub.docker.com/r/qpod/qpod/) | [GitHub](https://github.com/QPod/docker-images) ) is an **out-of-box Data Science / AI environment and platform** at your fingertip which you would love 💕.

With Docker and `QPod`, you

- 📦 can start your data science / AI projects with nearly `zero configuration` - QPod puts everything about installing (latest) packages and configuring environment into standard docker images and set you free from these tedious work.
- 🌍 will find your work more `easy-to-reproduce` - QPod standard images make scientific research or data analysis project as [reproducible pipelines](https://doi.org/10.1038/d41586-018-07196-1) and help you [share your work with others](https://doi.org/10.1038/515151a).
- 🆙 can easily `scale-up and scale-out` your algorithms and key innovations - QPod help you move forward smoothly from the development stage to deployment stage by re-using these images to either to provide RESTful APIs or orchestrate map/reduce operations on big data.

![Screenshot of QPod](https://raw.githubusercontent.com/wiki/QPod/qpod-hub/img/QPod-screenshot.webp "Screenshot of QPod")

## What's actually there

`QPod` curates and maintains a series of Docker images including interactive computing environment to run a Jupyter Notebook (or JupyterLab) with Python, R, OpenJDK, NodeJS, Go, Julia, etc. Other IDE-like tools (e.g VS Code, R-Studio) are also included.

`QPod` supports use cases of both research and production:

- (Stand-alone) Use it on your laptop as default data science / develop environment.
- (Multi-tenant) Use it on a server/cluster to host multiple users to exploit hardware resources like GPU.
- (Deployment/Production) Use it as the base image to host RESTful APIs or work as executors or map/reduce operations.

![QPod-tech-arch](https://raw.githubusercontent.com/wiki/QPod/docker-images/img/QPod-arch.svg)

## How to use? `1-2-3-GO`🎉

### 0. Have docker installed on your laptop/server - Linux (e.g.: Ubuntu LTS) / Windows (>=10) / macOS

- Install **Docker >= 19.03**: `docker-ce` ( community version & free: [Linux](https://hub.docker.com/search/?offering=community&type=edition&operating_system=linux) | [macOS](https://hub.docker.com/editions/community/docker-ce-desktop-mac) | [Windows](https://desktop.docker.com/win/stable/amd64/Docker%20Desktop%20Installer.exe) ) on your laptop/server. **Docker installed from default Ubuntu/CentOS repository probably won't work for GPU!**
- If you want to use *NVIDIA GPUs* with `QPod`, Linux server or latest Windows WSL2 is **required**. After installing **Docker >= 19.03**, also install both the [`NVIDIA driver`](https://github.com/NVIDIA/nvidia-docker/wiki/Frequently-Asked-Questions#how-do-i-install-the-nvidia-driver) and the latest version of [`nvidia-container-toolkit`](https://github.com/NVIDIA/nvidia-docker#quickstart) to use the GPUs in containers.

### 1. Choose the features and choose a folder on your disk

- Choose a folder on your laptop/server to server as the base directory (e.g.: `/root`, `/User/me`, or `D:/work`). Use an absolute path instead of relative path -- files in this folder are visible in the environment (files outside this folder are not).

- Choose an tag from the table below (e.g `full` for your laptop, or `full-cuda` for a Linux server with NVIDIA GPU), depends on what features/moduels do you want.
Typically, you can choose `full` / `full-cuda` if you have enough disk space and no worry about your network speed.

### 2. Start the container

Change the value of `IMG` and `WORKDIR` to your choices in the script below, and run the script. Shutdown Jupyter or other service/program which are using port 8888 or 9999.

#### For Linux/macOS/Windows WSL, run this in bash/terminal

```shell
IMG="qpod/base-dev:latest"
WORKDIR="/root" # <- macOS change this to /Users/your_user_name

docker run -d --restart=always \
--name=QPod \
--hostname=QPod \
-p 8888:8888 -p 9999:9999 \
-v $WORKDIR:/root \
$IMG
sleep 10s && docker logs QPod 2>&1|grep token=

```

⚠️ To use `QPod` with NVIDIA GPU machines with `nvidia-docker`, be sure to:

- 👉 Use **Docker >= 19.03** and the command `nvidia-smi` works well on host machine
- 👉 Add option (after `--restart=always`) in the `docker run` command to enable GPU access: `--gpus all` (for older version of nvidia-container, use `--runtime nvidia`)
- 👉 Use `IMG="qpod/full-cuda"` or other images with cuda support

#### For Windows, run this in [Terminal](https://github.com/microsoft/terminal) or CMD

Docker on Windows doesn't support GPU yet (cuda WSL support is coming soon).

```cmd
SET IMG="qpod/full:latest"
SET WORKDIR="D:/work"
docker run -d --restart=always ^
--name=QPod ^
--hostname=QPod ^
-p 8888:8888 9999:9999 ^
-v %WORKDIR%:/root ^
%IMG%
timeout 10 && docker logs QPod 2>&1|findstr token=
```

### 3. Sit back for minutes and get the first-time login token

The commands in the last step will:

- trigger a docker image download process which may take minutes
- start a docker container named `QPod`
- print a string contains a URL, which includes a 48-digit hexadecimal number

Copy the printed hexadecimal string *after* `?token=` as the first-time login token.

### Go! 🎉

Access `http://localhost:8888` (or `http://ip-address:8888`) in your browser and paste the token you just copied to start the journey.

## Additional Information

### FAQ

For a list of FAQ or other information, please refer to the [wiki page](https://github.com/QPod/docker-images/wiki) of this repo.

### Hardware

The images are built based on `ubuntu:latest` and only tested on the `x86` platform.
Minor modifications are expected to port to `arm64`, `ppc64le` platform.

### Package Management

Although `conda` is installed, we do not recommend to use conda to install a lib/package, because:

- `conda` repo mirrors are generally not avaliable in restricted enterprise LAN, especially in fincial/medical related companies.
- `conda` does not reuse the existing system library yet if a system lib is already installed -- `conda` installs it again.
- An open-source alternative, `mamba` is installed.

### Customization

These images are highly customizable. If you find a system lib / Python module / R packages is missing, you can easily add one in the `install_XX.list` in the `work` folder. Utilites scripts and functions in `/opt/utils` folder will be helpful for custimize images.
47 changes: 0 additions & 47 deletions QPod/docs/intro.md

This file was deleted.

21 changes: 0 additions & 21 deletions QPod/docs/tutorial-basics/congratulations.md

This file was deleted.

34 changes: 0 additions & 34 deletions QPod/docs/tutorial-basics/create-a-blog-post.md

This file was deleted.

Loading

0 comments on commit b12fc01

Please sign in to comment.