Register
Login
Resources
Docs Blog Datasets Glossary Case Studies Tutorials & Webinars
Product
Data Engine LLMs Platform Enterprise
Pricing Explore
Connect to our Discord channel
Integration:  dvc git github
Justin Hsi 9527fb7e6e
update bleach to 3.1.1 that github alerted
4 years ago
f073d4d660
trying circleci
4 years ago
a25be91be4
added reflink to dvc cache.type, although current format of filesystem is ext4
4 years ago
9397e751ff
some work on dash app
4 years ago
e930dc7772
add timings and storing into database
4 years ago
e930dc7772
add timings and storing into database
4 years ago
9527fb7e6e
update bleach to 3.1.1 that github alerted
4 years ago
7226dbb0ae
reran on master, full run of catboost_regr
4 years ago
37ce73f3b4
subset run of catboost_both with 29% clf wt until 10_evaluate.dvc
4 years ago
run
9397e751ff
some work on dash app
4 years ago
bacc16da58
altered pipeline so base_loan_info matches api_loans
4 years ago
17be561263
alter dockerfile to clone all branches since should be super light
4 years ago
0630ca2100
commit all before full run for logistic regression
4 years ago
bacc16da58
altered pipeline so base_loan_info matches api_loans
4 years ago
59392f2e09
changed permissions recurisvely
4 years ago
32ba57afd4
alter dockerfile and requirement.txt
4 years ago
e930dc7772
add timings and storing into database
4 years ago
0630ca2100
commit all before full run for logistic regression
4 years ago
32ba57afd4
alter dockerfile and requirement.txt
4 years ago
0ba9c974a7
save before reinstall ubuntu
4 years ago
685fbd4b61
try push again
4 years ago
Storage Buckets
Data Pipeline
Legend
DVC Managed File
Git Managed File
Metric
Stage File
External File

README.md

You have to be logged in to leave a comment. Sign In

Lendingclub

For data driven loan selection on lendingclub. Important packages are sklearn, pandas, numpy, pytorch, fastai.

  1. Current model is RF (sklearn) + NN (pytorch). Performance was compared against picking entirely at random and picking at random within the best performing loan grade historically.
  2. Investigative models are trained on old done loans and validated on newest of old done loans.
  3. Models used in invest scripts are trained on all available training data.

TODOS:

  1. Add timings to invest script 1a) speed up as much as possible
  2. Compare speed when retraining single model on prediction of ensembled model
  3. Compare loans captured through API (in database) to information coming on csvs and ensure that all data matches (grade and subgrade in particular)
  4. Continue to build out Dash dashboard

Notes:

about the csvs/data

  1. Even though LC only issues loasn A1-D5, they still internally have A1 - G3/5 in the loan info. I checked the interest rates and grades with the information at https://www.lendingclub.com/foliofn/rateDetail.action

Strange loans are separated out after all cleaning steps

Git Tags

Various git tags for navigating between datasets and dev/full datasets datav0.0.0 <- model/scorer.dataprocessingtype.raw_data_csvs? Each redownload of new data, increment rightmost #?

DVC Stuff

  1. when want new raw_csvs: python lendingclub/csv_dl_archiving/01_download_LC_csvs.python but beware: https://dvc.org/doc/user-guide/update-tracked-file

Usage:

Advisable to set up an environment After cloning: in root dir (lendingclub) with setup.py, run pip install -e . properly setup account_info.py in user_creds (see example)

Run order (all scripts in lendingclub subdir):

  1. python lendingclub/csv_dl_archiving/01_download_and_check_csvs.py
  2. python lendingclub/csv_prepartion/02_unzip_csvs.py
  3. python Before running clean_loan_info, have to cd to lendingclub/csv_preparation python setup.py build_ext. Will make a build dir in cd, copy the .so (unix) or .pyd(windows) to cd

other notes to self:

j_utils is imported and use in several scripts. See repo https://github.com/jmhsi/j_utils

To fix permissions troubles, I ended up adding jenkins and justin to each others groups (sudo usermod -a -G groupName userName) and doing chmod 664(774) on .fth and dataframes or other files as necessary.

Current jenkins setup runs in conda environment (based off https://mdyzma.github.io/2017/10/14/python-app-and-jenkins/) Considering moving to docker containers once I build the Dockerfiles?

Made symlink: ln -s /home/justin/projects to /var/lib/jenkins/projects so jenkins can run scripts like the actual projects directory

Tip!

Press p or to see the previous file or, n or to see the next file

About

mirror lendingclub repo from github

Collaborators 1

Comments

Loading...