Are you sure you want to delete this access key?
Legend |
---|
DVC Managed File |
Git Managed File |
Metric |
Stage File |
External File |
Legend |
---|
DVC Managed File |
Git Managed File |
Metric |
Stage File |
External File |
This repository contains the code for setting up DVC to use a remote computer server using dask.
The use case have the following prerequisites:
/scratch/dvc_data_cache/
./scratch/dvc_users/[REMOTE_USERNAME]/
.ssh-keygen
), which have been copied to the remote server: ssh-copy-id [REMOTE_USERNAME]@[REMOTE_IP]
.ssh -L 8786:[REMOTE_USERNAME]@[REMOTE_IP]:8786 [REMOTE_USERNAME]@[REMOTE_IP]
.git clone git@github.com:<GITHUB_USERNAME>/dvc.git
cd dvc
conda create -n py36_open_source_dvc python=3.6
conda activate py36_open_source_dvc
pip install -r requirements.txt
pip install -r tests/requirements.txt
pip install -e .
pip install pre-commit
pre-commit install
conda install -n py36_open_source_dvc dask
which dvc
should say [HOME]/anaconda3/envs/py36_open_source_dvc/bin/dvc
and dvc --version
should say the exact version available in you local DVC development repository.conda activate py36_open_source_dvc
dvc config core.analytics false --global
dvc remote add ahsoka ssh://[REMOTE_IP]/scratch/dvc_users/[REMOTE_USERNAME]/ --global
dvc remote modify ahsoka user [REMOTE_USERNAME] --global
dvc remote modify ahsoka port 22 --global
dvc remote modify ahsoka keyfile [PATH_TO_YOUR_PUBLIC_SSH_KEY] --global
dvc remote add ahsoka_cache ssh://[REMOTE_IP]/scratch/dvc_data_cache --global
dvc remote modify ahsoka_cache user [REMOTE_USERNAME] --global
dvc remote modify ahsoka_cache port 22 --global
dvc remote modify ahsoka_cache keyfile [PATH_TO_YOUR_PUBLIC_SSH_KEY] --global
dvc config cache.ssh ahsoka_cache --global
This use case of DVC and Dask has been set up as follow:
cd scratch/dvc_users/[REMOTE_USERNAME]
mkdir dvc_dask_use_case
mkdir data
wget -P data/ https://s3-us-west-2.amazonaws.com/dvc-share/so/100K/Posts.xml.tgz
tar zxf data/Posts.xml.tgz -C data/
git clone git@github.com:PeterFogh/dvc_dask_use_case.git
Press p or to see the previous file or, n or to see the next file
Are you sure you want to delete this access key?
Are you sure you want to delete this access key?
Are you sure you want to delete this access key?
Are you sure you want to delete this access key?