Skip to main content

Jupyter working space

The latest version Gear Will be used "Jupyter working space" (It will also be directly referred to as "working space" )Replace the original "JupyterLab" , "Jupyter working space" It's about the original JupyterLab Improvement of:

  • What has been activated is still one JupyterLab Interactive environment for entrance. Its experience is different from the original JupyterLab There is no difference.
  • Provided "restart" The process. It's no longer like before JupyterLab equally. Every time it starts (Continue to execute)A new execution will be created later. Avoiding it "Continue to execute" The problem of doubling the storage scale caused by. Also easy for novice users to understand
  • Every time it starts, it will bind its previous one "working directory" content. No longer allow changing bindings at startup; But it can still be activated during startup "Computing power" , "Runtime environment" (image), Data binding (input binding)Waiting for configuration adjustments.

Basic interface​

Click on a single execution page "open Jupyter working space" Enter the following interface:

The sidebar on the right displays some basic information about the current execution, include [System indicators](/docs/gear/metrics/#Show the key-metrics), Basic information for execution (Computing power type. Operating environment. Running time)And the setting interface. Among them, system indicators can be used to understand whether the current execution is effectively utilizing the computing resources provided by the system. After opening, it will also pop up by default HyperAI-intro.ipynb The file contains HyperAI Jupyter Some basic introductions to workspaces.

Although JupyterLab The directory displayed at the top of the file navigation bar on the left side of the page is /home But it doesn't mean in the system /home catalogue. But it refers to relative to JupyterLab The location of the set root directory. HyperAI take JupyterLab The working directory is set to /hyperai , So when JupyterLab The displayed directory is /home When. In fact, the directory in the corresponding system is /hyperai/home catalogue.

The left file navigation bar cannot display /hyperai The content of the assumed directory. If you need to save files from other directories. Please proceed through JupyterLab of Terminal Copy the contents of other directories to the command line /hyperai/home catalogue.

Jupyter Creation of workspace​

Compared to the previous creation JupyterLab The process is the same. Just choosing "Access method" When. There are no longer any existing ones JupyterLab But it's a new workspace:

note

When creating execution "Data binding" stage. The workspace can still perform data binding, and "Python Script execution" And the previous ones JupyterLab The behavior is consistent. See document ["Data binding"](/docs/gear/#Data binding).

After execution startup. Its interface and JupyterLab It's consistent:

Perform long-term training​

The training tasks of machine learning, especially deep learning, usually require a relatively long running time. But as an interactive programming environment. JupyterLab Not very suitable for this scenario.

Although JupyterLab Even if the executed program is closed JupyterLab The webpage. The program will still run, But in JupyterLab Running tasks for a long time will have the following two issues:

  1. Long term tasks that experience network fluctuations during runtime can cause interruptions in the printing of their running logs. Causing inability to view the current training progress
  2. Long term tasks Jupyter When executing in the workspace, even if the task is completed or an error occurs for some reason. Jupyter The workspace will not immediately close. But billing continues to lead to wastage of computing resources

Therefore, if it is necessary to perform long-term training tasks. Suggest viewing [Task With Jupyter Integration of workspace](/docs/gear/#task-With-jupyter-Combining two modes of workspace)Partially adopted task Execute long-term tasks. However, two corresponding solutions are still provided here:

Use Log Console Find the lost log​

Here, a simple code snippet is used to simulate a long-running task:

import time

for i in range(100000):
    print(i)
    time.sleep(1)

Put this code in one .ipynb I saw the following log information during the execution of the file:

This code block will continuously update the log. But if we refresh the page and come back, we will find that the logs are no longer updated. Right click on the code block at this time "Show Log Console" Open the current one Log Console Interface.

Then choose Debug You can see the progress of previous logs by logging at a certain level:

Use log redirection to save logs to a file and coordinate with it tail Command to view the latest log content​

For the passage Terminal Long tasks to be executed. You can redirect the log to a specific file. And through tail -f The command to view logs:

# Take python train.py The output is redirected to train.log In the document
python train.py > train.log 2>&1 &

# Adopt tail -f Command View train.log The content of the file
tail -f train.log

Jupyter Automatic closure of workspace​

because "working space" After activation, billing will begin. To reduce the waste of computing resources caused by forgetting to close. HyperAI By default, it has been added to the container "working space" Setting for automatic shutdown upon timeout.

stay HyperAI in, "working space" Two conditions are required to be considered free:

  1. JupyterLab The page has been closed
  2. CPU The usage rate continues to be lower than 2%

stay "container" of "set up" The tab can be viewed "JupyterLab Idle automatic shutdown setting" , You can set how long the container will be automatically closed when it is idle. The default duration is 30 minute. When a container is identified as idle, the creator of the container will receive a reminder (Send via SMS or email), If the container is still in an idle state. It will be closed due to reaching the maximum idle time set, under these circumstances HyperAI We will also send a notification to the creator of the container that the container has been closed. The notification method for container idle closure can be found in [gear#set notification](/docs/gear#set notification) Configure it.

be careful very important!

judge JupyterLab Is the page closed or not JupyterLab Provided by api Implemented. We will not record users' editing content. If JupyterLab The page is not closed. So the backend will query it JupyterLab In an active state. In this situation, we would consider "working space" Still being used. It will not trigger the idle shutdown mechanism.

JupyterLab Closing the page will not cause interruptions to programs that are already running. Therefore, if it is indeed necessary to JupyterLab Mid term Executive Task. I also hope that the program will trigger idle shutdown after running. You can directly close the page.

sometimes JupyterLab Although it is not turned off, the computer's own locking or sleep can also cause issues HyperAI I think the container is inactive. But this situation is related to everyone's computer settings. If you want to trigger idle shutdown without wasting computing resources, it is still recommended to close the page.

When the container is automatically closed due to idle time. HyperAI It will be displayed on the status of the container "Timeout closure" .

Jupyter Kernel Kernel restart​

Every time Jupyter Open a workspace .ipynb After the file, the system will create a corresponding file named kernel The backend is responsible for executing the code submitted on the page. If you discover your own .ipynb There was an issue with executing the command for the file. The simplest solution is to restart the corresponding system kernel:

Jupyter Restarting the workspace​

Can be restarted after the workspace is closed. Currently, the workspace supports two startup modes: "quick start" and "Modify configuration and start" .

Quick start​

Click on the page "start-up" Perform a quick start. Default retention of the last execution "Data binding" "Computing power" "Runtime environment" (image), And automatically bind to the previous runtime "working directory" The data.

Modify configuration and start​

If you need to modify the configuration, you can click "start-up" The arrow on the right:

Afterwards, the computing power can be modified according to the process. Runtime environment (image), Data binding and other information.

Jupyter Cache mechanism for workspace​

  • There is no limit to the number of workspaces that each user can create. But the number of cached workspaces we provide for each user is limited
  • When creating a workspace. By default, everything will be cached. The cached workspace will display an additional lightning icon on the interface
  • After reaching the maximum cache capacity that the current user can save. Will downgrade the old cached workspace to a regular workspace. When downgraded by cached workspace. Always prioritize deleting the one with the farthest closure time from now (The earliest one)
  • The cached workspace has the maximum caching time after being closed. After exceeding this time. Even if no new workspace is created. It will also be downgraded to a regular workspace
  • The cached workspace is functionally no different from a regular workspace. There is only a difference in startup speed

Can we still create old versions JupyterLab?​

The workspace is automatically opened on all platforms. After the update. Users will no longer be able to create older versions JupyterLab I got it. If you want to migrate to a new workspace, you only need to bind the original working directory when creating the workspace.

Create a snapshot for the workspace​

establish "snapshot" You can transfer the current workspace "working directory" Save it. As a fixed version. For future use:

After snapshot creation, based on "working directory" The size of the scale requires a certain amount of copying time, working space "working directory" The larger. The longer it takes to copy, the longer it will take.

A snapshot will record the usage of the workspace "Computing power type" "Data binding" "Mirror Category" , adopt "Continue to execute" You can start a new one from this snapshot "working space" , By default, the above configuration recorded in the snapshot will be used and the snapshot will be updated "working directory" Copy to a new workspace /hyperai/home Under the directory:

Several points to note in snapshot mode:

  • Users are only present in the workspace "Running" and "Closed" Snapshot can be created in two states
  • The space occupied by the snapshot is the standard storage space cost. No additional fees. Deleting snapshots that are no longer in use will free up the corresponding storage space
  • Try to avoid writing files to the running workspace when creating a snapshot / Modify operation. To prevent the generated snapshot from being inconsistent with the expected content
Last updated on by sparanoid