So if the spotml.yaml file has a script like below configured
Then the below command basically queues a managed run in the spotML cloud
spotml run train
The scheduled run It tries to do the following:
If the instance is not running - Try spawning the spot instance.
If a spot instance is not available, retry launching every 15 mins, until spot capacity is available.
If the instance is running - SSH into the session and run the above script.
spotml run stop
stop is a special named argument. If you intend to cancel the scheduled run, use above command.
Note that if there is an active running script, you need to manually kill the process by spotml sh run and terminate the process. Otherwise the Idle time checker will continue to think there is an active command running.