Register
Login
Resources
Docs Blog Datasets Glossary Case Studies Tutorials & Webinars
Product
Data Engine LLMs Platform Enterprise
Pricing Explore
Connect to our Discord channel
Resources  
Docs Blog Datasets Glossary Case Studies Tutorials & Webinars
Explore Pricing
Product  
Data Engine LLMs Platform Enterprise
Connect to Dagshub Discord channel
Register
Login

avatar Dean
/
Bookdata-tools
mirror of https://github.com/BoiseState/bookdata-tools

Watch 2
Issues   Discussions 0 Invite collaborators

dvc.lock 5.5 KB
Permalink History Raw

You have to be logged in to leave a comment. Sign In
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184 
  1. schema: '2.0'
    2. 

  2. stages:
    3. 

  3.   scan-books:
    4. 

  4.     cmd: python ../run.py --rust scan-marc --book-mode --glob "../data/loc-books/BooksAll.2016*.xml.gz"
    5. 

  5.     deps:
    6. 

  6.     - path: ../data/loc-books
    7. 

  7.       md5: 1b1e7ab1d98cc81e373dfc53345d4bb7.dir
    8. 

  8.       size: 3129774145
    9. 

  9.       nfiles: 43
    10. 

  10.     - path: ../src/cli/scan_marc.rs
    11. 

  11.       md5: bcbc2c6cfebc08fbd4072a6b01fba27d
    12. 

  12.       size: 3706
    13. 

  13.     - path: ../src/marc
    14. 

  14.       md5: 066418c6e3db231224a89a97aa94da9f.dir
    15. 

  15.       size: 19381
    16. 

  16.       nfiles: 5
    17. 

  17.     outs:
    18. 

  18.     - path: book-authors.parquet
    19. 

  19.       md5: 3585eceab422d960e403a2a146bb0b5f
    20. 

  20.       size: 84128212
    21. 

  21.     - path: book-fields.parquet
    22. 

  22.       md5: e90234eba3c42a15c0514a5178d222d7
    23. 

  23.       size: 2703843200
    24. 

  24.     - path: book-ids.parquet
    25. 

  25.       md5: 38346ca168b32be26216dba43e5a6ef4
    26. 

  26.       size: 66927842
    27. 

  27.     - path: book-isbns.parquet
    28. 

  28.       md5: 0775712f467248239f0d8f6e6daa3302
    29. 

  29.       size: 56227773
    30. 

  30.   book-authors:
    31. 

  31.     cmd: python ../run.py --rust fusion book-authors.tcl
    32. 

  32.     deps:
    33. 

  33.     - path: book-authors.tcl
    34. 

  34.       md5: 56396f59e4ea77683b2af8e74d205a4f
    35. 

  35.       size: 250
    36. 

  36.     - path: book-fields.parquet
    37. 

  37.       md5: b305bde2a13f77c0cb62ed586574480c
    38. 

  38.       size: 3094731732
    39. 

  39.     outs:
    40. 

  40.     - path: book-authors.parquet
    41. 

  41.       md5: c46c8998dfdfd25b91c88b225a436466
    42. 

  42.       size: 100853758
    43. 

  43.   scan-names:
    44. 

  44.     cmd: python ../run.py --rust scan-marc --glob "../data/loc-names/Names.2016*.xml.gz"
    45. 

  45.       -o name-fields.parquet
    46. 

  46.     deps:
    47. 

  47.     - path: ../data/loc-names
    48. 

  48.       md5: fc488a8775561070cced774803fe0d72.dir
    49. 

  49.       size: 1410755359
    50. 

  50.       nfiles: 40
    51. 

  51.     - path: ../src/cli/scan_marc.rs
    52. 

  52.       md5: bcbc2c6cfebc08fbd4072a6b01fba27d
    53. 

  53.       size: 3706
    54. 

  54.     - path: ../src/marc
    55. 

  55.       md5: 066418c6e3db231224a89a97aa94da9f.dir
    56. 

  56.       size: 19381
    57. 

  57.       nfiles: 5
    58. 

  58.     outs:
    59. 

  59.     - path: name-fields.parquet
    60. 

  60.       md5: 9643ffeda173dba838da8975c88bd958
    61. 

  61.       size: 1398366309
    62. 

  62.   author-names:
    63. 

  63.     cmd: python ../run.py --rust fusion author-names.tcl
    64. 

  64.     deps:
    65. 

  65.     - path: author-names.tcl
    66. 

  66.       md5: 29c68172aa2a10d6054b73656c21567c
    67. 

  67.       size: 193
    68. 

  68.     - path: name-fields.parquet
    69. 

  69.       md5: 800b068931bb1a2cba59b85b3ebf471d
    70. 

  70.       size: 1543059992
    71. 

  71.     outs:
    72. 

  72.     - path: author-names.csv.gz
    73. 

  73.       md5: 9e68c20496a2f62922334701f9e1492f
    74. 

  74.       size: 87125358
    75. 

  75.   author-genders:
    76. 

  76.     cmd: python ../run.py --rust fusion author-genders.tcl
    77. 

  77.     deps:
    78. 

  78.     - path: author-genders.tcl
    79. 

  79.       md5: 58abbe4a110eb782450d9db0e461e087
    80. 

  80.       size: 197
    81. 

  81.     - path: name-fields.parquet
    82. 

  82.       md5: 800b068931bb1a2cba59b85b3ebf471d
    83. 

  83.       size: 1543059992
    84. 

  84.     outs:
    85. 

  85.     - path: author-genders.csv.gz
    86. 

  86.       md5: 71a6ef56b674c032c5a572d478e66762
    87. 

  87.       size: 1469682
    88. 

  88.   book-isbn-ids:
    89. 

  89.     cmd: python run.py --rust link-isbn-ids -R rec_id -o loc-mds/book-isbn-ids.parquet
    90. 

  90.       loc-mds/book-isbns.parquet
    91. 

  91.     deps:
    92. 

  92.     - path: book-links/all-isbns.parquet
    93. 

  93.       md5: f1de86d41234d3b0ea975be3a188a0cc
    94. 

  94.       size: 275147630
    95. 

  95.     - path: loc-mds/book-isbns.parquet
    96. 

  96.       md5: 0775712f467248239f0d8f6e6daa3302
    97. 

  97.       size: 56227773
    98. 

  98.     outs:
    99. 

  99.     - path: loc-mds/book-isbn-ids.parquet
    100. 

  100.       md5: d318bbfa7f023370f780aa74c0bd60b5
    101. 

  101.       size: 33782689
    102. 

  102.   loc-clusters:
    103. 

  103.     cmd: python ../run.py --rust fusion loc-clusters.tcl
    104. 

  104.     deps:
    105. 

  105.     - path: ../book-links/cluster-graph-nodes.parquet
    106. 

  106.       md5: 6d0ebe8ef88733428ac88a222f959b96
    107. 

  107.       size: 798139218
    108. 

  108.     - path: loc-clusters.tcl
    109. 

  109.       md5: 31716e72f6d58f047ea391109be5d806
    110. 

  110.       size: 221
    111. 

  111.     outs:
    112. 

  112.     - path: loc-clusters.parquet
    113. 

  113.       md5: 0748a8001c63142ebda3d099b6ebd84c
    114. 

  114.       size: 35754914
    115. 

  115.   schema@book-fields:
    116. 

  116.     cmd: python ../run.py --rust pq-info -o book-fields.json book-fields.parquet
    117. 

  117.     deps:
    118. 

  118.     - path: book-fields.parquet
    119. 

  119.       md5: e90234eba3c42a15c0514a5178d222d7
    120. 

  120.       size: 2703843200
    121. 

  121.     outs:
    122. 

  122.     - path: book-fields.json
    123. 

  123.       md5: caafeec111379cec95011e60ed31ec8d
    124. 

  124.       size: 693
    125. 

  125.   schema@book-isbn-ids:
    126. 

  126.     cmd: python ../run.py --rust pq-info -o book-isbn-ids.json book-isbn-ids.parquet
    127. 

  127.     deps:
    128. 

  128.     - path: book-isbn-ids.parquet
    129. 

  129.       md5: d318bbfa7f023370f780aa74c0bd60b5
    130. 

  130.       size: 33782689
    131. 

  131.     outs:
    132. 

  132.     - path: book-isbn-ids.json
    133. 

  133.       md5: 12cc90bd30dce870a246014e288fc7fa
    134. 

  134.       size: 249
    135. 

  135.   schema@book-ids:
    136. 

  136.     cmd: python ../run.py --rust pq-info -o book-ids.json book-ids.parquet
    137. 

  137.     deps:
    138. 

  138.     - path: book-ids.parquet
    139. 

  139.       md5: 38346ca168b32be26216dba43e5a6ef4
    140. 

  140.       size: 66927842
    141. 

  141.     outs:
    142. 

  142.     - path: book-ids.json
    143. 

  143.       md5: 9f886551928eceb2b60ffbc0c7ce9687
    144. 

  144.       size: 605
    145. 

  145.   schema@book-authors:
    146. 

  146.     cmd: python ../run.py --rust pq-info -o book-authors.json book-authors.parquet
    147. 

  147.     deps:
    148. 

  148.     - path: book-authors.parquet
    149. 

  149.       md5: 3585eceab422d960e403a2a146bb0b5f
    150. 

  150.       size: 84128212
    151. 

  151.     outs:
    152. 

  152.     - path: book-authors.json
    153. 

  153.       md5: 3f2bc8e71f9a4d8c5623f966780b40bf
    154. 

  154.       size: 252
    155. 

  155.   schema@book-isbns:
    156. 

  156.     cmd: python ../run.py --rust pq-info -o book-isbns.json book-isbns.parquet
    157. 

  157.     deps:
    158. 

  158.     - path: book-isbns.parquet
    159. 

  159.       md5: 0775712f467248239f0d8f6e6daa3302
    160. 

  160.       size: 56227773
    161. 

  161.     outs:
    162. 

  162.     - path: book-isbns.json
    163. 

  163.       md5: 92446742844cfef47d5baa62d5586bdb
    164. 

  164.       size: 329
    165. 

  165.   schema@loc-clusters:
    166. 

  166.     cmd: python ../run.py --rust pq-info -o loc-clusters.json loc-clusters.parquet
    167. 

  167.     deps:
    168. 

  168.     - path: loc-clusters.parquet
    169. 

  169.       md5: 0748a8001c63142ebda3d099b6ebd84c
    170. 

  170.       size: 35754914
    171. 

  171.     outs:
    172. 

  172.     - path: loc-clusters.json
    173. 

  173.       md5: a9217be376ffffe877ee4f3a2f0dcbb9
    174. 

  174.       size: 352
    175. 

  175.   schema@name-fields:
    176. 

  176.     cmd: python ../run.py --rust pq-info -o name-fields.json name-fields.parquet
    177. 

  177.     deps:
    178. 

  178.     - path: name-fields.parquet
    179. 

  179.       md5: 9643ffeda173dba838da8975c88bd958
    180. 

  180.       size: 1398366309
    181. 

  181.     outs:
    182. 

  182.     - path: name-fields.json
    183. 

  183.       md5: 670f2e18ea5c27e7d6c9e60d36241d08
    184. 

  184.       size: 693
    185.
Prev
Next
Tip!

Press p or to see the previous file or, n or to see the next file

Comments

Write Preview
Loading...
Integrate Google Cloud Storage
Use Google Storage
Select bucket
Upload key
Finish

Use Google Cloud Storage!

Browsing data directories saved to Google Cloud Storage is possible with DAGsHub. Let's configure your repository to easily display your data in the context of any commit!

Specify your Google Storage bucket

Service Account Key

Congratulations!

Bookdata-tools is now integrated with Google Cloud Storage!

Delete Storage Key

Are you sure you want to delete this access key?

No
Yes
Integrate AWS S3
Use S3 remote
Select bucket
Access key
Finish

Use AWS S3 as storage!

Browsing data directories saved to S3 is possible with DAGsHub. Let's configure your repository to easily display your data in the context of any commit!

Specify your S3 bucket

Access key (If needed)

Congratulations!

Bookdata-tools is now integrated with AWS S3!

Delete Storage Key

Are you sure you want to delete this access key?

No
Yes
Integrate S3 compatible storage
Use S3 like remote
Select bucket
Access key
Finish

Use any S3 compatible storage!

Browsing data directories saved to S3 compatible storage is possible with DAGsHub. Let's configure your repository to easily display your data in the context of any commit!

Specify your S3 bucket

Access key (If needed)

Congratulations!

Bookdata-tools is now integrated with your S3 compatible storage!

Delete Storage Key

Are you sure you want to delete this access key?

No
Yes
Integrate Azure Cloud Storage
Use Azure Storage
Select bucket
Set key
Finish

Use Azure Cloud Storage!

Browsing data directories saved to Azure Cloud Storage is possible with DAGsHub. Let's configure your repository to easily display your data in the context of any commit!

Specify your Azure Storage bucket

Access key (If needed)

Congratulations!

Bookdata-tools is now integrated with Azure Cloud Storage!

Delete Storage Key

Are you sure you want to delete this access key?

No
Yes