Runtime environment (image)overview

Common dependencies in runtime environment

HyperAI By default, the runtime environment contains a large number of basic dependencies. To reduce the consumption of runtime resources and dependency downloads and installations during each container startup.

Pre installed dependencies can be divided into the following aspects according to their purpose:

1. General Machine Learning Library

• scikit-learn General Machine Learning Library. Contains a large number of machine learning models. Data analysis. Data mining algorithms and visualization tools
• XGBoost A high-performance GBDT Model implementation, of large number kaggle The award-winning algorithms are all built on top of this algorithm
• ONNX Deep Learning Model Conversion Library
• spaCy Industrial grade natural language processing library
• LightGBM Launched by Microsoft boosting frame

2. Image processing tools

Commonly used graphics processing libraries

• OpenCV Powerful image processing tools
• Pillow Python Commonly used image processing tools below

3. Data Analysis Library

• pandas
• SciPy
• Matplotlib
• NumPy
• h5py

How to add dependencies that are not in the list

default HyperAI The runtime environment has already installed a large number of dependencies for machine learning scenarios. If you still need additional dependencies. You can install it through the following methods.

note

Run time environment Conda Manage it. Can support Conda Install additional dependencies.

danger

Installed in each runtime environment CUDA The version is different. Be sure to pay attention to the runtime environment when installing additional dependencies CUDA Matching the environment.

Install Python The Class Library

If you are uploading code and "Python Script execution" When running code in a certain way, it requires some additional dependencies. You can define a name in the root directory of the uploaded code HyperAI_requirements.txt perhaps requirements.txt Add the necessary dependencies and other files to the file and upload them together. Before running the code. The system will first install these dependencies before executing them "python script" .

The format and content of the file Python of requirements.txt The format is consistent. A typical example HyperAI_requirements.txt The file content is as follows:

requirements.txt

jieba
tqdm==4.11.2

among jieba tqdm It's two that can be passed through pip Installed libraries. The above format can be used to execute "python script" Before, install these libraries first.tqdm==4.11.2 of == Afterwards, the specific version to be installed was specified.

danger

Some dependencies in the system, such as tensorflow pytorch The version is not recommended to be modified arbitrarily. Because of different versions tensorflow perhaps pytorch There are also differences in the dependencies on its underlying layers. May cause damage to the current environment.

Adopt Conda Perform dependency management

about "working space" See this section use Conda Manage dependencies.

about "Python Script execution" You can provide a code named in the root directory of the uploaded code conda-packages.txt The file. Its file format follows

[channel::]package[=version[=buildid]]

Here is an example:

conda-packages.txt

conda-forge::rdkit
conda-forge::pygpu

If requirements.txt, HyperAI_requirements.txt and conda-packages.txt Simultaneously existing. So we will install it first conda-packages.txt Dependence within. Then install it again HyperAI_requirements.txt as well as requirements.txt Dependence within.

Install other dependencies

If it is in "Jupyter working space" You can refer to the next section. If it is "Python Script execution" You can install additional non scenarios in the following ways Python The dependence:

1. Include the dependency installation command in "Execute command" in

   For example, if you want to download necessary files before running the program git Warehouse. You can use the following "Execute command" :

   $ git clone https://github.com/tensorflow/models.git && cd models && python ...

2. Prepare one dependencies.sh script

   For non Conda and PyPI The dependence. You can provide a name in the root directory called dependencies.sh The file, stay "Python Script execution" It will be activated upon startup bash implement. And its execution will be earlier than HyperAI_requirements.txt requirements.txt and conda-packages.txt Dependent installation.

   For example. I use commands here dependencies.sh The script. The content is as follows:

   dependencies.sh

   git clone https://github.com/tensorflow/models.git
   cd models
   pip install -r requirements.txt

info

Your runtime environment is Linux Ubuntu environment. If you want to install additional package dependencies. Can be used apt-get perhaps apt command. Of course, commands usually need to be executed before execution apt-get update perhaps apt update

Stay Jupyter Installing dependencies in the workspace

stay "Jupyter working space" You can install any dependencies you need in it. Regardless of whether it is Python Or through apt Other dependencies installed.

For example, the following is installing an additional one in the editor Python rely on:

In the input ! Follow me afterwards pip The installation command is sufficient, for example !pip install jieba.

And through !apt install xxx You can install it in it apt Package dependencies:

Retain pip rely on

at present HyperAI Additional installations below pip Dependency will be saved by default on the system disk. When the container restarts, dependencies need to be reinstalled. But if used pip Add additional parameters when installing dependencies --user So the installed dependencies will be saved in the workspace of the container (that is /hyperai/home)lower. The specific directory location is /hyperai/home/.pylibs, When "working space" When restarting after shutting down, .pylibs The directory will also be copied back to the specified directory of the container, adopt pip list Viewing dependencies still allows you to see the list of already installed dependencies.

1. Remove unnecessary dependencies

If you don't want to keep these dependencies. Can be in "Execute command" Add in rm -rf /hyperai/home/.pylibs command.

2. Need different Python edition

If you need a brand new one Python edition (at present HyperAI lower Python The version is 3.6 perhaps 3.8)Can be based on [stay /hyperai/home Create a new environment below](/docs/gear/conda#1-stay-HyperAIhome-Create a new environment below) Create a brand new environment in /hyperai/home Under the directory. And install complete dependencies within it.

info

adopt conda Do not use after creating a brand new environment pip --user Use the parameters to install dependencies; Adding --user Post dependency will be arranged in /hyperai/home/.pylibs Below rather than newly created conda Under the environment. Easy to trigger dependency conflicts.

Install Jupyter Workspace expansion

"Jupyter working space" There are many extensions available. We can "Terminal" Add the extensions we need in the middle. We demonstrate here how to install an extension that automatically increases the editor directory jupyterlab-toc.

Open one "Terminal" , And enter the following command:

jupyter labextension install @jupyterlab/toc

Open another one .ipynb file. You can see it appearing on the left side "catalogue" The tab. After clicking, you can see the directory of the current file.