Register
Login
Resources
Docs Blog Datasets Glossary Case Studies Tutorials & Webinars
Product
Data Engine LLMs Platform Enterprise
Pricing Explore
Connect to our Discord channel
Resources  
Docs Blog Datasets Glossary Case Studies Tutorials & Webinars
Explore Pricing
Product  
Data Engine LLMs Platform Enterprise
Connect to Dagshub Discord channel
Register
Login

avatar Dean
/
Bookdata-tools
mirror of https://github.com/BoiseState/bookdata-tools

Watch 2
Issues   Discussions 0 Invite collaborators

dvc.lock 8.1 KB
History Raw

You have to be logged in to leave a comment. Sign In
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260 
  1. schema: '2.0'
    2. 

  2. stages:
    3. 

  3.   scan-authors:
    4. 

  4.     cmd: python ../run.py --rust openlib scan-authors ../data/openlib/ol_dump_authors.txt.gz
    5. 

  5.     deps:
    6. 

  6.     - path: ../data/openlib/ol_dump_authors.txt.gz
    7. 

  7.       md5: d29e6e8811a215e43e95ee92b9b08e6b
    8. 

  8.       size: 430740681
    9. 

  9.     - path: ../src/cli/openlib.rs
    10. 

  10.       md5: ac2b0d52d93e6865e0d9b322b25f7127
    11. 

  11.       size: 1860
    12. 

  12.     - path: ../src/openlib/
    13. 

  13.       md5: ac94aee1e1d7e70ef36cb41acf22e3de.dir
    14. 

  14.       size: 12843
    15. 

  15.       nfiles: 5
    16. 

  16.     outs:
    17. 

  17.     - path: author-names.parquet
    18. 

  18.       md5: 1af0e26605c0f6b58b9c881d53633622
    19. 

  19.       size: 149833187
    20. 

  20.     - path: authors.parquet
    21. 

  21.       md5: da91945ac0ec61cd87ced891218ab676
    22. 

  22.       size: 179288999
    23. 

  23.   scan-editions:
    24. 

  24.     cmd: python ../run.py --rust openlib scan-editions ../data/openlib/ol_dump_editions.txt.gz
    25. 

  25.     deps:
    26. 

  26.     - path: ../data/openlib/ol_dump_editions.txt.gz
    27. 

  27.       md5: a1bec0711d017789c45f068430debbee
    28. 

  28.       size: 7771713448
    29. 

  29.     - path: ../src/cli/openlib.rs
    30. 

  30.       md5: ac2b0d52d93e6865e0d9b322b25f7127
    31. 

  31.       size: 1860
    32. 

  32.     - path: ../src/openlib/
    33. 

  33.       md5: ac94aee1e1d7e70ef36cb41acf22e3de.dir
    34. 

  34.       size: 12843
    35. 

  35.       nfiles: 5
    36. 

  36.     - path: author-ids-after-works.parquet
    37. 

  37.       md5: 455d061be0bfdd5d2d300085965c969c
    38. 

  38.       size: 69139925
    39. 

  39.     - path: authors.parquet
    40. 

  40.       md5: da91945ac0ec61cd87ced891218ab676
    41. 

  41.       size: 179288999
    42. 

  42.     - path: works.parquet
    43. 

  43.       md5: 88ab3700b15672381c0e5f7b029891d8
    44. 

  44.       size: 662682117
    45. 

  45.     outs:
    46. 

  46.     - path: all-authors.parquet
    47. 

  47.       md5: 5f024bb345e3049d1c9426784692928a
    48. 

  48.       size: 69155209
    49. 

  49.     - path: all-works.parquet
    50. 

  50.       md5: 1b9a3d75dd8a0e4c25176daf864d1ee2
    51. 

  51.       size: 181910106
    52. 

  52.     - path: edition-authors.parquet
    53. 

  53.       md5: 887d3b1709b880338a9e93ecbeb213f6
    54. 

  54.       size: 193094588
    55. 

  55.     - path: edition-isbns.parquet
    56. 

  56.       md5: 01512ce2cc14b72d23b039d132f194f5
    57. 

  57.       size: 211695542
    58. 

  58.     - path: edition-subjects.parquet
    59. 

  59.       md5: 9fef04d9efe7f640462e9a7b454bdda7
    60. 

  60.       size: 500132074
    61. 

  61.     - path: edition-works.parquet
    62. 

  62.       md5: 53632b1fc28f4bc8f0775f5518ec865b
    63. 

  63.       size: 203821436
    64. 

  64.     - path: editions.parquet
    65. 

  65.       md5: bc080214dcb896f99624cb14d13c9706
    66. 

  66.       size: 887407054
    67. 

  67.   scan-works:
    68. 

  68.     cmd: python ../run.py --rust openlib scan-works ../data/openlib/ol_dump_works.txt.gz
    69. 

  69.     deps:
    70. 

  70.     - path: ../data/openlib/ol_dump_works.txt.gz
    71. 

  71.       md5: 6ce3e136b51768cffd0bcb7e0b8c8b11
    72. 

  72.       size: 2397066368
    73. 

  73.     - path: ../src/cli/openlib.rs
    74. 

  74.       md5: ac2b0d52d93e6865e0d9b322b25f7127
    75. 

  75.       size: 1860
    76. 

  76.     - path: ../src/openlib/
    77. 

  77.       md5: ac94aee1e1d7e70ef36cb41acf22e3de.dir
    78. 

  78.       size: 12843
    79. 

  79.       nfiles: 5
    80. 

  80.     - path: authors.parquet
    81. 

  81.       md5: da91945ac0ec61cd87ced891218ab676
    82. 

  82.       size: 179288999
    83. 

  83.     outs:
    84. 

  84.     - path: author-ids-after-works.parquet
    85. 

  85.       md5: 455d061be0bfdd5d2d300085965c969c
    86. 

  86.       size: 69139925
    87. 

  87.     - path: work-authors.parquet
    88. 

  88.       md5: 25387d62102aceb6b7e1e7174f403649
    89. 

  89.       size: 153751612
    90. 

  90.     - path: work-subjects.parquet
    91. 

  91.       md5: 2e0ace66d8661fc7af09c9cfb8300c33
    92. 

  92.       size: 289616554
    93. 

  93.     - path: works.parquet
    94. 

  94.       md5: 88ab3700b15672381c0e5f7b029891d8
    95. 

  95.       size: 662682117
    96. 

  96.   edition-isbn-ids:
    97. 

  97.     cmd: python run.py --rust link-isbn-ids -R edition -o openlibrary/edition-isbn-ids.parquet
    98. 

  98.       openlibrary/edition-isbns.parquet
    99. 

  99.     deps:
    100. 

  100.     - path: book-links/all-isbns.parquet
    101. 

  101.       md5: f1de86d41234d3b0ea975be3a188a0cc
    102. 

  102.       size: 275147630
    103. 

  103.     - path: openlibrary/edition-isbns.parquet
    104. 

  104.       md5: 01512ce2cc14b72d23b039d132f194f5
    105. 

  105.       size: 211695542
    106. 

  106.     outs:
    107. 

  107.     - path: openlibrary/edition-isbn-ids.parquet
    108. 

  108.       md5: 41b3c9f8c0daef62b26c4cc6049f69df
    109. 

  109.       size: 145369776
    110. 

  110.   schema@edition-isbn-ids:
    111. 

  111.     cmd: python ../run.py --rust pq-info -o edition-isbn-ids.json edition-isbn-ids.parquet
    112. 

  112.     deps:
    113. 

  113.     - path: edition-isbn-ids.parquet
    114. 

  114.       md5: 41b3c9f8c0daef62b26c4cc6049f69df
    115. 

  115.       size: 145369776
    116. 

  116.     outs:
    117. 

  117.     - path: edition-isbn-ids.json
    118. 

  118.       md5: 827dca74b99c048363d44079676043b9
    119. 

  119.       size: 251
    120. 

  120.   schema@works:
    121. 

  121.     cmd: python ../run.py --rust pq-info -o works.json works.parquet
    122. 

  122.     deps:
    123. 

  123.     - path: works.parquet
    124. 

  124.       md5: 88ab3700b15672381c0e5f7b029891d8
    125. 

  125.       size: 662682117
    126. 

  126.     outs:
    127. 

  127.     - path: works.json
    128. 

  128.       md5: 706aca7030101d951bcb0b66b963c86b
    129. 

  129.       size: 327
    130. 

  130.   schema@all-works:
    131. 

  131.     cmd: python ../run.py --rust pq-info -o all-works.json all-works.parquet
    132. 

  132.     deps:
    133. 

  133.     - path: all-works.parquet
    134. 

  134.       md5: 1b9a3d75dd8a0e4c25176daf864d1ee2
    135. 

  135.       size: 181910106
    136. 

  136.     outs:
    137. 

  137.     - path: all-works.json
    138. 

  138.       md5: c45c3ee516ca84d35a7cd015153007e2
    139. 

  139.       size: 239
    140. 

  140.   schema@editions:
    141. 

  141.     cmd: python ../run.py --rust pq-info -o editions.json editions.parquet
    142. 

  142.     deps:
    143. 

  143.     - path: editions.parquet
    144. 

  144.       md5: bc080214dcb896f99624cb14d13c9706
    145. 

  145.       size: 887407054
    146. 

  146.     outs:
    147. 

  147.     - path: editions.json
    148. 

  148.       md5: 4dcd48ec4508c167afbc14765ca3eee5
    149. 

  149.       size: 327
    150. 

  150.   schema@work-authors:
    151. 

  151.     cmd: python ../run.py --rust pq-info -o work-authors.json work-authors.parquet
    152. 

  152.     deps:
    153. 

  153.     - path: work-authors.parquet
    154. 

  154.       md5: 25387d62102aceb6b7e1e7174f403649
    155. 

  155.       size: 153751612
    156. 

  156.     outs:
    157. 

  157.     - path: work-authors.json
    158. 

  158.       md5: 2fdda7889bd40c794eed8fff81e6252f
    159. 

  159.       size: 331
    160. 

  160.   schema@author-names:
    161. 

  161.     cmd: python ../run.py --rust pq-info -o author-names.json author-names.parquet
    162. 

  162.     deps:
    163. 

  163.     - path: author-names.parquet
    164. 

  164.       md5: 1af0e26605c0f6b58b9c881d53633622
    165. 

  165.       size: 149833187
    166. 

  166.     outs:
    167. 

  167.     - path: author-names.json
    168. 

  168.       md5: 4cce0edb96a3cba9bbaf3fb80cce80b8
    169. 

  169.       size: 331
    170. 

  170.   schema@edition-authors:
    171. 

  171.     cmd: python ../run.py --rust pq-info -o edition-authors.json edition-authors.parquet
    172. 

  172.     deps:
    173. 

  173.     - path: edition-authors.parquet
    174. 

  174.       md5: 887d3b1709b880338a9e93ecbeb213f6
    175. 

  175.       size: 193094588
    176. 

  176.     outs:
    177. 

  177.     - path: edition-authors.json
    178. 

  178.       md5: 543090377abfe966c7d979b8d39584bb
    179. 

  179.       size: 336
    180. 

  180.   schema@edition-isbns:
    181. 

  181.     cmd: python ../run.py --rust pq-info -o edition-isbns.json edition-isbns.parquet
    182. 

  182.     deps:
    183. 

  183.     - path: edition-isbns.parquet
    184. 

  184.       md5: 01512ce2cc14b72d23b039d132f194f5
    185. 

  185.       size: 211695542
    186. 

  186.     outs:
    187. 

  187.     - path: edition-isbns.json
    188. 

  188.       md5: cdf9af7ea7b32093db56aa712b0f3524
    189. 

  189.       size: 247
    190. 

  190.   schema@edition-works:
    191. 

  191.     cmd: python ../run.py --rust pq-info -o edition-works.json edition-works.parquet
    192. 

  192.     deps:
    193. 

  193.     - path: edition-works.parquet
    194. 

  194.       md5: 53632b1fc28f4bc8f0775f5518ec865b
    195. 

  195.       size: 203821436
    196. 

  196.     outs:
    197. 

  197.     - path: edition-works.json
    198. 

  198.       md5: 13393bc08b510c76dad29a2c4eddb086
    199. 

  199.       size: 248
    200. 

  200.   schema@all-authors:
    201. 

  201.     cmd: python ../run.py --rust pq-info -o all-authors.json all-authors.parquet
    202. 

  202.     deps:
    203. 

  203.     - path: all-authors.parquet
    204. 

  204.       md5: 5f024bb345e3049d1c9426784692928a
    205. 

  205.       size: 69155209
    206. 

  206.     outs:
    207. 

  207.     - path: all-authors.json
    208. 

  208.       md5: a2da52b17a972e97c7f5ac00113114cc
    209. 

  209.       size: 237
    210. 

  210.   schema@authors:
    211. 

  211.     cmd: python ../run.py --rust pq-info -o authors.json authors.parquet
    212. 

  212.     deps:
    213. 

  213.     - path: authors.parquet
    214. 

  214.       md5: da91945ac0ec61cd87ced891218ab676
    215. 

  215.       size: 179288999
    216. 

  216.     outs:
    217. 

  217.     - path: authors.json
    218. 

  218.       md5: f9031457b89faff9c2b47d288cd92ad2
    219. 

  219.       size: 325
    220. 

  220.   schema@work-subjects:
    221. 

  221.     cmd: python ../run.py --rust pq-info -o work-subjects.json work-subjects.parquet
    222. 

  222.     deps:
    223. 

  223.     - path: work-subjects.parquet
    224. 

  224.       md5: 2e0ace66d8661fc7af09c9cfb8300c33
    225. 

  225.       size: 289616554
    226. 

  226.     outs:
    227. 

  227.     - path: work-subjects.json
    228. 

  228.       md5: de90e9aa7cbba45569aa8d43d9a93a15
    229. 

  229.       size: 245
    230. 

  230.   schema@edition-subjects:
    231. 

  231.     cmd: python ../run.py --rust pq-info -o edition-subjects.json edition-subjects.parquet
    232. 

  232.     deps:
    233. 

  233.     - path: edition-subjects.parquet
    234. 

  234.       md5: 9fef04d9efe7f640462e9a7b454bdda7
    235. 

  235.       size: 500132074
    236. 

  236.     outs:
    237. 

  237.     - path: edition-subjects.json
    238. 

  238.       md5: 453a9e5291756a40d9347a1d2371af58
    239. 

  239.       size: 245
    240. 

  240.   work-clusters:
    241. 

  241.     cmd: python run.py --rust cluster extract-books -n work_id -o openlibrary/work-clusters.parquet
    242. 

  242.       OL-W
    243. 

  243.     deps:
    244. 

  244.     - path: book-links/cluster-graph-nodes.parquet
    245. 

  245.       md5: 3151fdb3f0c19c69a7b4c9f82670cf65
    246. 

  246.       size: 832829368
    247. 

  247.     outs:
    248. 

  248.     - path: openlibrary/work-clusters.parquet
    249. 

  249.       md5: ff37e4b7fb36f9d4f292f65ad4307d00
    250. 

  250.       size: 172719910
    251. 

  251.   schema@work-clusters:
    252. 

  252.     cmd: python ../run.py --rust pq-info -o work-clusters.json work-clusters.parquet
    253. 

  253.     deps:
    254. 

  254.     - path: work-clusters.parquet
    255. 

  255.       md5: ff37e4b7fb36f9d4f292f65ad4307d00
    256. 

  256.       size: 172719910
    257. 

  257.     outs:
    258. 

  258.     - path: work-clusters.json
    259. 

  259.       md5: e99e4eaf4d61244087755ee9f897371e
    260. 

  260.       size: 251
    261.
Prev
Next
Tip!

Press p or to see the previous file or, n or to see the next file

Comments

Write Preview
Loading...
Integrate Google Cloud Storage
Use Google Storage
Select bucket
Upload key
Finish

Use Google Cloud Storage!

Browsing data directories saved to Google Cloud Storage is possible with DAGsHub. Let's configure your repository to easily display your data in the context of any commit!

Specify your Google Storage bucket

Service Account Key

Congratulations!

Bookdata-tools is now integrated with Google Cloud Storage!

Delete Storage Key

Are you sure you want to delete this access key?

No
Yes
Integrate AWS S3
Use S3 remote
Select bucket
Access key
Finish

Use AWS S3 as storage!

Browsing data directories saved to S3 is possible with DAGsHub. Let's configure your repository to easily display your data in the context of any commit!

Specify your S3 bucket

Access key (If needed)

Congratulations!

Bookdata-tools is now integrated with AWS S3!

Delete Storage Key

Are you sure you want to delete this access key?

No
Yes
Integrate S3 compatible storage
Use S3 like remote
Select bucket
Access key
Finish

Use any S3 compatible storage!

Browsing data directories saved to S3 compatible storage is possible with DAGsHub. Let's configure your repository to easily display your data in the context of any commit!

Specify your S3 bucket

Access key (If needed)

Congratulations!

Bookdata-tools is now integrated with your S3 compatible storage!

Delete Storage Key

Are you sure you want to delete this access key?

No
Yes
Integrate Azure Cloud Storage
Use Azure Storage
Select bucket
Set key
Finish

Use Azure Cloud Storage!

Browsing data directories saved to Azure Cloud Storage is possible with DAGsHub. Let's configure your repository to easily display your data in the context of any commit!

Specify your Azure Storage bucket

Access key (If needed)

Congratulations!

Bookdata-tools is now integrated with Azure Cloud Storage!

Delete Storage Key

Are you sure you want to delete this access key?

No
Yes