Register
Login
Resources
Docs Blog Datasets Glossary Case Studies Tutorials & Webinars
Product
Data Engine LLMs Platform Enterprise
Pricing Explore
Connect to our Discord channel
Resources  
Docs Blog Datasets Glossary Case Studies Tutorials & Webinars
Explore Pricing
Product  
Data Engine LLMs Platform Enterprise
Connect to Dagshub Discord channel
Register
Login

cwerner
/
deadtrees
connected to https://github.com/cwerner/deadtrees.git

Watch 2
Issues 14 Pull Requests 2 Discussions 0 Invite collaborators

dvc.yaml 3.0 KB
Permalink History Raw

You have to be logged in to leave a comment. Sign In
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112 
  1. # dvc project pipeline
    2. 

  2. 

  3. # NOTE: see scripts/README.MD for preprocessing from raw data up to the first
    4. 

  4. # tracked stage `createtiles`
    5. 

  5. 

  6. stages:
    7. 

  7.   createtiles:
    8. 

  8.     foreach:
    9. 

  9.       - 2017
    10. 

  10.       - 2018
    11. 

  11.       - 2019
    12. 

  12.       - 2020
    13. 

  13.     do:
    14. 

  14.       cmd: >-
    15. 

  15.         mkdir -p data/processed.images.${item};
    16. 

  16.         gdal_retile.py 
    17. 

  17.         -csv locations.csv 
    18. 

  18.         -v -ps ${source_dim} ${source_dim}
    19. 

  19.         -co "TILED=YES" -co "COMPRESS=LZW" -co "PREDICTOR=2" -co "ALPHA=NO" -co "NUM_THREADS=ALL_CPUS"
    20. 

  20.         -targetDir data/processed.images.${item} 
    21. 

  21.         data/raw/ortho_ms_${item}_EPSG3044.tif
    22. 

  22.       deps:
    23. 

  23.       - data/raw/ortho_ms_${item}_EPSG3044.tif
    24. 

  24.       params:
    25. 

  25.       - source_dim
    26. 

  26.       outs:
    27. 

  27.       - data/processed.images.${item}
    28. 

  28. 

  29.   computestats:
    30. 

  30.     cmd: >-
    31. 

  31.       python scripts/computestats.py --frac 0.1
    32. 

  32.       data/processed.images.2017  
    33. 

  33.       data/processed.images.2018  
    34. 

  34.       data/processed.images.2019 
    35. 

  35.       data/processed.images.2020 
    36. 

  36.     deps:
    37. 

  37.       - data/processed.images.2017
    38. 

  38.       - data/processed.images.2018
    39. 

  39.       - data/processed.images.2019
    40. 

  40.       - data/processed.images.2020 
    41. 

  41.     outs:
    42. 

  42.       - data/processed.images.stats.json
    43. 

  43.   
    44. 

  44.   createmasks:
    45. 

  45.     cmd: >-
    46. 

  46.       python scripts/createmasks.py 
    47. 

  47.       data/processed.images.2019 
    48. 

  48.       data/processed.masks.2019 
    49. 

  49.       data/raw/shapefiles/deadtrees_2019/deadtrees_2019.shp
    50. 

  50.       --negativesample data/raw/shapefiles/deadtrees_notdead/deadtrees_notdead.shp
    51. 

  51.     deps:
    52. 

  52.     - data/processed.images.2019
    53. 

  53.     - data/raw/shapefiles/deadtrees_2019
    54. 

  54.     - data/raw/shapefiles/deadtrees_notdead
    55. 

  55.     outs:
    56. 

  56.     - data/processed.masks.2019
    57. 

  57.     - data/processed.masks.2019.neg_sample
    58. 

  58. 

  59.   createdataset:
    60. 

  60.     cmd: >-
    61. 

  61.       python scripts/createdataset.py 
    62. 

  62.       data/processed.images.2019 
    63. 

  63.       data/processed.masks.2019 
    64. 

  64.       data/dataset 
    65. 

  65.       --source_dim ${source_dim}
    66. 

  66.       --tile_size ${createdataset.tile_size}
    67. 

  67.       --format ${file_type}
    68. 

  68.     deps:
    69. 

  69.     - data/processed.images.2019
    70. 

  70.     - data/processed.masks.2019 
    71. 

  71.     - data/processed.masks.2019.neg_sample 
    72. 

  72.     params:
    73. 

  73.     - source_dim
    74. 

  74.     - createdataset.tile_size
    75. 

  75.     - file_type
    76. 

  76.     outs:
    77. 

  77.     - data/dataset/train
    78. 

  78.     - data/dataset/stats.csv 
    79. 

  79. 

  80.   # train: do this manually
    81. 

  81. 

  82.   # inference
    83. 

  83.   inference:
    84. 

  84.     foreach:
    85. 

  85.       - 2017
    86. 

  86.       - 2019
    87. 

  87.     do:
    88. 

  88.       cmd: >-
    89. 

  89.         mkdir -p data/predicted.${item};
    90. 

  90.         stdbuf -i0 -o0 -e0 python scripts/inference.py --all --nopreview -o data/predicted.${item} data/processed.images.${item};
    91. 

  91.         gdal_merge.py 
    92. 

  92.         -co "TILED=YES" -co "COMPRESS=LZW" -co "PREDICTOR=2" -co "NUM_THREADS=ALL_CPUS"
    93. 

  93.         -o data/predicted_mosaic_${item}.tif 
    94. 

  94.         data/predicted.${item}/ortho_ms_${item}_EPSG3044_*
    95. 

  95.       deps:
    96. 

  96.       - data/processed.images.${item}
    97. 

  97.       - checkpoints/bestmodel.ckpt
    98. 

  98.       outs:
    99. 

  99.       - data/predicted.${item}
    100. 

  100.       - data/predicted_mosaic_${item}.tif
    101. 

  101. 

  102.       
    103. 

  103.   computestatsinference:
    104. 

  104.     cmd: >-
    105. 

  105.       python scripts/computestats_inference.py
    106. 

  106.       data/predicted.2017  
    107. 

  107.       data/predicted.2019 
    108. 

  108.     deps:
    109. 

  109.       - data/predicted.2017
    110. 

  110.       - data/predicted.2019
    111. 

  111.     outs:
    112. 

  112.       - data/predicted.stats.csv
    113.
Prev
Next
Tip!

Press p or to see the previous file or, n or to see the next file

Comments

Write Preview
Loading...
Integrate Google Cloud Storage
Use Google Storage
Select bucket
Upload key
Finish

Use Google Cloud Storage!

Browsing data directories saved to Google Cloud Storage is possible with DAGsHub. Let's configure your repository to easily display your data in the context of any commit!

Specify your Google Storage bucket

Service Account Key

Congratulations!

deadtrees is now integrated with Google Cloud Storage!

Delete Storage Key

Are you sure you want to delete this access key?

No
Yes
Integrate AWS S3
Use S3 remote
Select bucket
Access key
Finish

Use AWS S3 as storage!

Browsing data directories saved to S3 is possible with DAGsHub. Let's configure your repository to easily display your data in the context of any commit!

Specify your S3 bucket

Access key (If needed)

Congratulations!

deadtrees is now integrated with AWS S3!

Delete Storage Key

Are you sure you want to delete this access key?

No
Yes
Integrate S3 compatible storage
Use S3 like remote
Select bucket
Access key
Finish

Use any S3 compatible storage!

Browsing data directories saved to S3 compatible storage is possible with DAGsHub. Let's configure your repository to easily display your data in the context of any commit!

Existing keys:
s3://deadtrees/dvc
Apr 28, 2022

Specify your S3 bucket

Access key (If needed)

Congratulations!

deadtrees is now integrated with your S3 compatible storage!

Delete Storage Key

Are you sure you want to delete this access key?

No
Yes
Integrate Azure Cloud Storage
Use Azure Storage
Select bucket
Set key
Finish

Use Azure Cloud Storage!

Browsing data directories saved to Azure Cloud Storage is possible with DAGsHub. Let's configure your repository to easily display your data in the context of any commit!

Specify your Azure Storage bucket

Access key (If needed)

Congratulations!

deadtrees is now integrated with Azure Cloud Storage!

Delete Storage Key

Are you sure you want to delete this access key?

No
Yes