nl2ml

Alexander Levin 24b4e5df40 upd: SVM v5 validaten on the golden set; 11 hours ago
.dvc 7c97b2a9a3 dvc data fix 1 month ago
.ipynb_checkpoints 82b810b974 last checkpoints added 1 month ago
code2vec 97c5b91e60 code2vec folder added 1 month ago
data
graph 6c73f0a9c9 Update graph_v6.txt 13 hours ago
model_interpretation_results ee690b0621 SVM conf matrix graph 3 and 3.1, validation chunk_size == 10 3 weeks ago
models
open_data
.gitattributes bdb584c714 Get rid of csv in Git LFS 2 months ago
.gitignore 08a39d0906 open_data folder (with Python150k) added 1 month ago
Comments vs commented code.ipynb a4b1957697 in-code comments classification added 1 month ago
Makefile 3e3cfc582f Makefile 2 weeks ago
README.md 7cc55383b5 Update README.md 2 weeks ago
RNN.ipynb ed844fb89f Add files via upload 2 weeks ago
bert_classifier.ipynb 45a0bcfcdb changed names 2 months ago
bert_distances.ipynb 45a0bcfcdb changed names 2 months ago
check_results.ipynb 91d79ae683 upd: clear outputs; 6 days ago
data.dvc 999d24c49f added: golden 884 set added; 1 day ago
github_dataset.ipynb 3f08f4b1c5 exploring.. 1 month ago
kaggle.sh fc9e9f9c28 ramazyant files added 1 month ago
kaggle_parser.ipynb fc9e9f9c28 ramazyant files added 1 month ago
logreg_classifier.ipynb db490fb05e rm: deprecated weights analysis; clear outputs; 1 day ago
logreg_weights_analysis.ipynb 38ffcf58fa upd: logreg weights analysis moved to separate notebook 2 weeks ago
metrics.csv 24b4e5df40 upd: SVM v5 validaten on the golden set; 11 hours ago
models.dvc 0b9b0d2347 upd: graph_v4, graph_v5 models and data 1 week ago
nl2ml.pptx de41f6d52b init: presentation 1 day ago
nl2ml_notebook_parser.py 8ba024a8c5 no message 5 months ago
open_data.dvc 08a39d0906 open_data folder (with Python150k) added 1 month ago
params.yml 24b4e5df40 upd: SVM v5 validaten on the golden set; 11 hours ago
predict_tag.ipynb 78b621fb69 upd: logreg v5 validated on golden set; 11 hours ago
regex.ipynb 1ce2f0fa4a added: regex evaluation 12 hours ago
svm_classifier.ipynb d703f657c1 getting inference for logreg v3 and v3.1; 3 weeks ago
svm_train.py 9cf6418f00 upd: DAGSHUB source -> script_dir; 12 hours ago

Data Pipeline

Legend
DVC Managed File
Git Managed File
Metric
Stage File
External File

README.md

Source Code Classification

This is a repo of the Natural Language to Machine Learning (NL2ML) project of the Laboratory of Methods for Big Data Analysis at Higher School of Economics (HSE LAMBDA).

The project's official repo is stored on GitLab (HSE LAMBDA repository) - https://gitlab.com/lambda-hse/nl2ml \ The project's full description is stored on Notion - https://www.notion.so/NL2ML-Corpus-1ed964c08eb049b383c73b9728c3a231 \ The project's experiments are stored on DAGsHub - https://dagshub.com/levin/source_code_classification

Project Goals

Short-Term Goal

To build a model classifying a source code chunk and to specify where the detected class is exactly in the chunk (tag segmentation).

Long-Term Goal

To build a model generating code by getting a short raw english task in as an input.

Repository Description

This repository contains instruments which the project's team has been using to label source code chunks with Knowledge Graph vertices and to train models to recognize these vertices in future. By the Knowledge Graph vertices we mean an elementary part of ML-pipeline. The current latest version of the Knowledge Graph contains the following high-level vertices: ['import', 'data_import', 'data_export', 'preprocessing', 'visualization', 'model', 'deep_learning_model', 'train' 'predict'].

Data Download

To download the project data and models:

  1. Clone this repository
  2. Install DVC from https://dvc.org/doc/install
  3. Do dvc pull data or dvc pull data. Note: if you are failing on dvc pull [folder_to_pull], try dvc pull [folder_to_pull] --jobs 1

Contents:

The instruments which we have been using to reach the project goals are: notebooks parsing from Kaggle API and Github API, data preparation, regex-labellig, training models, validation models, model weights/coefficients analysis, errors analysis, synonyms analysis.

nl2ml_notebook_parser.py - a script for parsing Kaggle notebooks and process them to JSON/CSV/Pandas.

bert_distances.ipynb - a notebook with BERT expiremints concerning sense of distance between BERT embeddings where input tokens were tokenized source code chunks.

bert_classifier.ipynb - a notebook with preprocessing and training BERT-pipeline.

regex.ipynb - a notebook with creating labels for code chunks with regex

logreg_classifier.ipynb - a notebook with training logistic regression model on the regex labels with tf-idf and analyzing the outputs

Comments vs commented code.ipynb - a notebook with a model distinguishing NL-comments from commented source code

github_dataset.ipynb - a notebook with opening github_dataset

predict_tag.ipynb - a notebook with predicting class label (tag) with any model

svm_classifier.ipynb - a notebook with training SVM (replaced by _svmtrain.py) and analyzing SVM outputs

svm_train.py - a script for training SVM model