Connect to our Discord channel
mraxilus
I mainly use DAGsHub for mirroring purposes. My personal projects are hosted on GitLab (mraxilus).
mraxilus
I mainly use DAGsHub for mirroring purposes. My personal projects are hosted on GitLab (mraxilus).
mraxilus synced commits to master at mraxilus/thesis-doctorate from mirror
4199ea8438
feat: extract lines plot into module
2 years ago
mraxilus synced commits to master at mraxilus/thesis-doctorate from mirror
67dba604a3
feat: finish running entire pipeline
I.e. the air quality prophet stages.
2 years ago
mraxilus
synced and deleted reference refs/pipelines/533685439 at mraxilus/thesis-doctorate from mirror
2 years ago
mraxilus synced new reference refs/pipelines/533685439 to mraxilus/thesis-doctorate from mirror
2 years ago
mraxilus synced commits to master at mraxilus/thesis-doctorate from mirror
ba164f790c
feat: run Prophet pipeline for water data
2 years ago
mraxilus synced commits to master at mraxilus/thesis-doctorate from mirror
3ee837d693
feat: rerun entire pipeline less Prophet air/water
2 years ago
mraxilus synced commits to master at mraxilus/thesis-doctorate from mirror
e5be80d65e
fix: ensure Prophet predictions don't include timestep 0
2d50202703
feat: refactor entire pipeline to remove data redundency across cores
Drastically simplified code in all stages of the pipeline, which shuold
also give some minor performance improvements. I swapped from
`multiprocessing` to `ray` to reduce the memory duplication. Also,
simplified the logging so that it would be more clear what progress has
been made within a stage (more specifically modelling).
2 years ago
mraxilus synced commits to master at mraxilus/thesis-doctorate from mirror
a9925a13fb
ci: fix dag command misconfiguration to ensure pipeline passes
0631010c02
feat: run pipeline for naive prediction and prophet (synthetic)
2 years ago
mraxilus synced commits to master at mraxilus/thesis-doctorate from mirror
24250a95bd
feat: refactor entire pipeline and improve performance/readability
a2e000feb8
feat: make logging more consistent
2 years ago
mraxilus synced commits to master at mraxilus/thesis-doctorate from mirror
ef9a3dc81e
refactor: run black on notebook
c8a8319a13
feat: add more validation checks for pipeline data
Ideally I would actually create tets to verify things such as whether or
not the caches adhere to the desired schema, but these spot checks will
have to do for now.
5938d5f3dc
feat: add root relative squared error metric
2 years ago
mraxilus synced commits to master at mraxilus/thesis-doctorate from mirror
ea4ad43204
refactor: apply black to notebooks
72b92e09c6
test: start validating individual samples througout pipeline
f0e2e099e2
fix: run new pipeline for all naive and synthetic prophet.
Also add required moments as parameters.
2 years ago
mraxilus synced commits to master at mraxilus/thesis-doctorate from mirror
67dba9e2e9
ci: fix pipeline from failing on dag generation
Stage names have changed which broke it.
4bed5d2d9e
feat: add addional moments for error surface generation
Added standard deviation and skew in addition to mean.
084b8e5cee
revert: remove numexpr as it doesnt work well with new nullable dtypes
3049cf531e
feat: use numexpr for error calucations for performance improvements
222162e920
feat: run stages up to estalbish and all of naive
2 years ago
mraxilus
synced and deleted reference refs/pipelines/523127577 at mraxilus/thesis-doctorate from mirror
2 years ago
mraxilus synced new reference refs/pipelines/523127577 to mraxilus/thesis-doctorate from mirror
2 years ago
mraxilus synced commits to master at mraxilus/thesis-doctorate from mirror
037e1a8d5d
feat: start investigating data issues
Seems as though the prophet model is only storing out of sample
predictions for some reason.
66db53d665
feat: refactor dvc pipeline significantly
2 years ago
mraxilus synced commits to master at mraxilus/thesis-doctorate from mirror
838939fa55
feat: produce complete results for air quality dataset
2 years ago
mraxilus synced commits to master at mraxilus/thesis-doctorate from mirror
a6ab245f85
fix: remove unnecesary file checks from pre-commit
2 years ago
mraxilus synced commits to master at mraxilus/thesis-doctorate from mirror
6fb8f28bbe
ci: ensure dag plot produces entire graphs
It broke last commit as the stage names were misspecified.
a7a42aee55
feat: run pipeline for naive air
67618ca5e2
ci: move over to aws from gcs
GCS and GitLab runners don't seem to work well together. I can run the
pipeline successfully, locally, but it always fails with `Forbidden` on
the shared runners. Just using s3 instead, but still keeping the main
backup in a GCS bucket.
2 years ago
mraxilus synced commits to master at mraxilus/thesis-doctorate from mirror
6b356e342b
feat: add amazon as backup dvc remote
d94e7fb5df
feat: produce results for synthetic and water data
6cc3be582b
ci: attempt to resolve pipeline failure
Remove verbosity flags. Added visualization of the pipelines.
I got it working on a local GitLab runner instance, so it should work
when I push. I'm unsure if the `status` command is the fix culprit, or
if I had to change the cache type to copy and relink (or both).
f2c5c676fa
ci: add dvc status command for troubleshooting
e58578f144
ci: try using copy as cache type to resolve dvc errors
2 years ago
mraxilus synced commits to master at mraxilus/thesis-doctorate from mirror
3b26d8fa26
ci: ensure stages are properly defined
e7b3736d91
ci: attempt to get more debugging info from dvc
9ece4ee368
ci: fix pipeline by ingoring notebooks
Notebooks will still be checked locally. Once I have more time I can
refactor all the existing notebooks and start checking them within the
CI pipeline.
43a34e33c0
ci: fix prettier command line option
4e6fcf5ef0
ci: add pre-commit validation stage
Ensure that pre-commit was run locally. It required that I ignore some
things that I don't want to deal with at the moment. I added TODOs so
that I would remember to unignore them in the future.
2 years ago
