Are you sure you want to delete this access key?
All public Kaggle kernels are distributed under Apache 2.0 license.
Kaggle has a lot of references for its objects. Let's clarify the terminology.
https://www.kaggle.com/data-scientist/cool-solution
https://www.kaggle.com/
. The slug of the kernel in previous example is data-scientist/cool-solution
clean_data.py goes through every code block in the labeled dataset, removes comments, formats code to conform to the PEP8 standard, removes service labels. This script has 2 CLI arguments:
collect_kernels_from_competitions.py traverses popular competitions that are available through Kaggle API and collects the slugs of the connected kernels.
The script does the following: - Collects the links to the Kaggle competitions to ‘competitions_ref.csv’ - Runs competition_kernels.sh, which collect kernels of every competition - Collects .csv files with the Kaggle kernels links to ./kernel_lists directory
competition_kernels.sh collects the kernel slugs for a single competition. This script is used in collect_kernels_from_competitions.py
. This script has 1 CLI argument which is competition slug.
competition_collector.py collects the competition slugs from the Kaggle search engine. This script has 3 CLI arguments:
--competition_count
is the maximum amount of collected competition slugs--process_id
is the id of the worker--total_processes
is the total amount of workerscompetition_tags.py extracts competition tags. The tags include metric as well as metadata like data type or subject.
kernel_collector.py collects kernel slugs from the Kaggle search engine. This script has 2 CLI arguments:
--kernel_count
is the maximum amount of collected kernel slugs--process_id
is the id of the worker. The script assumes that there are 3 workers in total.kernel_parser.py takes the kernel slugs from csv file in KERNEL_FILENAME
variable, then it loads each kernel and splits it into code blocks. This script has 1 CLI argument:
--process_id
is the id of the worker. The script assumes that there are 3 workers in total.Input: table with kernel links Output: table with each kernel code blocks, code blocks ids and kernels metadata:"kaggle_score", "kaggle_comments", "kaggle_upvotes", "kaggle_link", "kaggle_id", "kaggle_dataset”.
unite_kernel_lists.py takes all csv files from a folder and appends them to the main csv file which is defined in MERGED_LIST
variable.
mkdir kernel_lists
python collect_kernels_from_competitions.py
The script does the following:
python unite_kernel_lists_w_year.py
python unite_kernel_lists_w_year.py
Input: kernel_lists directory Output: .csv table with link to the kernel and submission year
python kernel_parser.py --process_id $process_id 1>./parser.err 2>./parser.log
process_id = {0,1,2}
The parser uses the BeautifulSoup4 library, as well as tqdm for a beautiful and convenient progress bar. to collect references to the data used, Selenium was required. Selenium uses the Google Chrome webdriver.
Press p or to see the previous file or, n or to see the next file
Are you sure you want to delete this access key?
Are you sure you want to delete this access key?
Are you sure you want to delete this access key?
Are you sure you want to delete this access key?