tm_map删除包含我的停用词的词吗？

Question

我申请removeWords过滤这样的文集：

corpus <- Corpus(vs, readerControl = list(language="en")) 
corpus <- tm_map(corpus, content_transformer(tolower)) 
corpus <- tm_map(corpus, stripWhitespace) 
corpus <- tm_map(corpus, removeWords, c(stopwords("english"))) 
corpus <- tm_map(corpus, removeWords, bannedWords$V1) 

然而，这只是匹配工作正是，所以：

• 他妈的被删除
• F * cking不会被删除

如何删除co ndain我的停用词？

Answer 1

我找到了答案通过查看tm库source code为removeWords功能和扩展的正则表达式：

gsub(sprintf("(*UCP)\\b(%s)\\b", 

到

gsub(sprintf("(*UCP)\\b[a-zA-Z]*(%s)[a-zA-Z]*\\b", 

完整的功能

removeWordsContaining <- 
function(x, words) 
    UseMethod("removeWordsContaining", x) 
removeWordsContaining.character <- 
function(x, words) 
    gsub(sprintf("(*UCP)\\b[a-zA-Z]*(%s)[a-zA-Z]*\\b", 
       paste(sort(words, decreasing = TRUE), collapse = "|")), 
     "", x, perl = TRUE) 
removeWordsContaining.PlainTextDocument <- 
    content_transformer(removeWordsContaining.character) 

blog_corpus <- Corpus(vs, readerControl = list(language="en")) 
blog_corpus <- tm_map(blog_corpus, content_transformer(tolower)) 
blog_corpus <- tm_map(blog_corpus, stripWhitespace) 
blog_corpus <- tm_map(blog_corpus, removePunctuation) 
blog_corpus <- tm_map(blog_corpus, removeNumbers) 
blog_corpus <- tm_map(blog_corpus, removeWords, c(stopwords("english"))) 
blog_corpus <- tm_map(blog_corpus, removeWordsContaining, bannedWords$V1) 

Answer 2

你可以使用词干来带来被禁止的单词ba CK到他们的基本形式。看下面的例子。

library(tm) 

banned <- c("buck") 
text <- c("He is bucking the trend", "A buck is not worth a dollar anymore!") 

corpus <- Corpus(VectorSource(text), readerControl = list(language="en")) 
corpus <- tm_map(corpus, content_transformer(tolower)) 
corpus <- tm_map(corpus, stripWhitespace) 
corpus <- tm_map(corpus, stemDocument) 
corpus <- tm_map(corpus, removeWords, c(stopwords("english"), banned)) 

writeLines(as.character(corpus[[1]])) 
    trend 

，如果你不阻止文件，你会得到：

corpus <- Corpus(VectorSource(text), readerControl = list(language="en")) 
corpus <- tm_map(corpus, content_transformer(tolower)) 
corpus <- tm_map(corpus, stripWhitespace) 
corpus <- tm_map(corpus, removeWords, c(stopwords("english"), banned)) 

writeLines(as.character(corpus[[1]])) 
    bucking trend