Skip to content
GitLab
Projects
Groups
Snippets
/
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
kihoon.lee
dataset
Commits
c9bfd263
Commit
c9bfd263
authored
Aug 19, 2024
by
kihoon.lee
Browse files
add korean vocab file
parent
047b7bfe
Changes
1
Hide whitespace changes
Inline
Side-by-side
korean-vocab/readme.md
0 → 100644
View file @
c9bfd263
# korean vocab
토크나이저에 추가하는 용으로 토큰화된 한글을 모아놓음
출처는
[
여기
](
https://huggingface.co/beomi/llama-2-ko-7b
)
입니다.
16488개 존재합니다.
토크나이저의
`add_new_vocab`
을 통해 추가하면 중복없이 추가된다고 합니다.
\ No newline at end of file
Write
Preview
Supports
Markdown
0%
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment