Spot ML
Search…
Getting started(5 mins)
Step 0: Setup AWS credentials (one time)
SpotML uses your AWS credentials to manage the runs for you. You can verify that you have the credentials setup if you see the contents of
~/.aws/credentials to be something like below.1
[my-aws-profile]
2
aws_access_key_id = your_aws_access_key_id
3
aws_secret_access_key = your_aws_secret_access_key
Copied!
Secondly, the above
access key IAM user needs to have the permissions to create all the aws resources.Step 1: Install spotml cli (one time)
1
pip install spotml --upgrade
Copied!
Step 2: Go through simple workflow:
- How to
startan aws instance to train MNIST code. - How to
sshinto instance to check progress. - How to
downloadresults to local machine
1. Clone the repo
1
git clone https://github.com/SpotML/spotml-examples.git
2
cd spotml-examples/mnist
Copied!
2. Start the instance
1
spotml start
Copied!
Wait for the instance to start, you will see an output like below once complete.
By default, SpotML will track the instance for idle time. If the instance is idle for more than 30 mins, it's automatically terminated.
3. SSH into instance
1
spotml sh
Copied!
SpotML uses tmux sessions. So to exit the ssh session type
Ctrl + b, then type d to disconnect from the session.4. Download the generated model file.
Make sure you have disconnected from the above ssh session. Once you have, from your local terminal type below command to download the generated model file.
1
spotml download -i 'my_model.h5'
Copied!
Step 3: Go through managed run workflow (optional)
- How to let SpotML automatically manage a long-running training on
spotinstances. - SpotML automatically restarts the interrupted instances and resumes training.
- SpotML automatically turns off idle instance.
1. Update config file
Open the spotml.yaml file and find the line that says
1
spotStrategy: on-demand
Copied!
change it to
1
spotStrategy: spot
Copied!
Also notice the scripts section of config file like below.
This allows you to configure keywords like train to run your custom training command.
2. Run the script
1
spotml run train
Copied!
This should produce output like below. If the instance is not already running, spotML tries to spawn a new spot instance and runs the above script once the instance is ready.
Note that if a spot instance is not available, spotML backend service keeps trying every 15 mins, until it can spawn the instance. So you can turn off your laptop and do other things, while SpotML tries to schedule the run.
If you intend to cancel the scheduled run, type
1
spotml run stop
Copied!
3. Check Status
1
spotml status
Copied!
You can check the status of the
instance, and the run with the above command. Once the instance is running you should see an output like below.To also check any logs generated when starting the instance type
1
spotml status --logs
Copied!
Once you see the run status as
RUNNING from the status command you can ssh into the actual run session by typing1
spotml sh run
Copied!
This opens the tmux session where spotML ran the
train command. You can also just ssh into a separate normal ssh session by typing below command as before for an interactive session.
1
spotml sh
Copied!
Last modified 6mo ago