lucene中的高亮

论坛 › 期权论坛 › 脚本     
匿名技术用户   2021-1-7 13:41   14   0

这里我们搜索的内容是“一步一步跟我学习lucene”，搜索引擎展示的结果中对用户的输入信息进行了配色方面的处理，这种区分正常文本和输入内容的效果即是高亮显示；

这样做的好处：

  • 视觉上让人便于查找有搜索对应的文本块；
  • 界面展示更友好；

lucene提供了highlighter插件来体现类似的效果；

highlighter对查询关键字高亮处理；

highlighter包包含了用于处理结果页查询内容高亮显示的功能，其中Highlighter类highlighter包的核心组件，借助Fragmenter, fragment Scorer, 和Formatter等类来支持用户自定义高亮展示的功能；

示例程序

这里边我利用了之前的做的目录文件索引

[java] view plain copy
  1. package com.lucene.search.util;
  3. import java.io.IOException;
  4. import java.io.StringReader;
  5. import java.util.concurrent.ExecutorService;
  6. import java.util.concurrent.Executors;
  8. import org.apache.lucene.analysis.Analyzer;
  9. import org.apache.lucene.analysis.TokenStream;
  10. import org.apache.lucene.analysis.standard.StandardAnalyzer;
  11. import org.apache.lucene.document.Document;
  12. import org.apache.lucene.index.Term;
  13. import org.apache.lucene.search.IndexSearcher;
  14. import org.apache.lucene.search.ScoreDoc;
  15. import org.apache.lucene.search.TermQuery;
  16. import org.apache.lucene.search.TopDocs;
  17. import org.apache.lucene.search.highlight.Highlighter;
  18. import org.apache.lucene.search.highlight.InvalidTokenOffsetsException;
  19. import org.apache.lucene.search.highlight.QueryScorer;
  20. import org.apache.lucene.search.highlight.SimpleFragmenter;
  21. import org.apache.lucene.search.highlight.SimpleHTMLFormatter;
  22. import org.apache.lucene.util.BytesRef;
  24. public class HighlighterTest {
  25. public static void main(String[] args) {
  26. IndexSearcher searcher;
  27. TopDocs docs;
  28. ExecutorService service = Executors.newCachedThreadPool();
  29. try {
  30. searcher = SearchUtil.getMultiSearcher("index", service);
  31. Term term = new Term("content",new BytesRef("lucene"));
  32. TermQuery termQuery = new TermQuery(term);
  33. docs = SearchUtil.getScoreDocsByPerPage(1, 30, searcher, termQuery);
  34. ScoreDoc[] hits = docs.scoreDocs;
  35. QueryScorer scorer = new QueryScorer(termQuery);
  36. SimpleHTMLFormatter simpleHtmlFormatter = new SimpleHTMLFormatter("<B>","</B>");//设定高亮显示的格式<B>keyword</B>,此为默认的格式
  37. Highlighter highlighter = new Highlighter(simpleHtmlFormatter,scorer);
  38. highlighter.setTextFragmenter(new SimpleFragmenter(20));//设置每次返回的字符数
  39. Analyzer analyzer = new StandardAnalyzer();
  40. for(int i=0;i<hits.length;i++){
  41. Document doc = searcher.doc(hits[i].doc);
  42. String str = highlighter.getBestFragment(analyzer, "content", doc.get("content")) ;
  43. System.out.println(str);
  44. }
  46. } catch (IOException e1) {
  47. // TODO Auto-generated catch block
  48. e1.printStackTrace();
  49. } catch (InvalidTokenOffsetsException e) {
  50. // TODO Auto-generated catch block
  51. e.printStackTrace();
  52. }finally{
  53. service.shutdown();
  54. }
  55. }
  56. }

lucene的highlighter高亮展示的原理：

  • 根据Formatter和Scorer创建highlighter对象，formatter定义了高亮的显示方式，而scorer定义了高亮的评分；

评分的算法是先根据term的评分值获取对应的document的权重，在此基础上对文本的内容进行轮询,获取对应的文本出现的次数，和它在term对应的文本中出现的位置（便于高亮处理），评分并分词的算法为：

[java] view plain copy
  1. public float getTokenScore() {
  2. position += posIncAtt.getPositionIncrement();//记录出现的位置
  3. String termText = termAtt.toString();
  5. WeightedSpanTerm weightedSpanTerm;
  7. if ((weightedSpanTerm = fieldWeightedSpanTerms.get(
  8. termText)) == null) {
  9. return 0;
  10. }
  12. if (weightedSpanTerm.positionSensitive &&
  13. !weightedSpanTerm.checkPosition(position)) {
  14. return 0;
  15. }
  17. float score = weightedSpanTerm.getWeight();//获取权重
  19. // found a query term - is it unique in this doc?
  20. if (!foundTerms.contains(termText)) {//结果排重处理
  21. totalScore += score;
  22. foundTerms.add(termText);
  23. }
  25. return score;
  26. }

formatter的原理为：对搜索的文本进行判断，如果scorer获取的totalScore不小于0，即查询内容在对应的term中存在，则按照格式拼接成preTag+查询内容+postTag的格式

详细算法如下：

[java] view plain copy
  1. public String highlightTerm(String originalText, TokenGroup tokenGroup) {
  2. if (tokenGroup.getTotalScore() <= 0) {
  3. return originalText;
  4. }
  6. // Allocate StringBuilder with the right number of characters from the
  7. // beginning, to avoid char[] allocations in the middle of appends.
  8. StringBuilder returnBuffer = new StringBuilder(preTag.length() + originalText.length() + postTag.length());
  9. returnBuffer.append(preTag);
  10. returnBuffer.append(originalText);
  11. returnBuffer.append(postTag);
  12. return returnBuffer.toString();
  13. }

其默认格式为“<B></B>”的形式；

  • Highlighter根据scorer和formatter，对document进行分析，查询结果调用getBestTextFragments,TokenStream tokenStream,String text,boolean mergeContiguousFragments,int maxNumFragments)，其过程为
  1. scorer首先初始化查询内容对应的出现位置的下标，然后对tokenstream添加PositionIncrementAttribute，此类记录单词出现的位置；
  2. 对文本内容进行轮询，判断查询的文本长度是否超出限制，如果超出文本长度提示过长内容；
  3. 如果获取到指定的文本，先对单次查询的内容进行内容的截取（截取值根据setTextFragmenter指定的值决定），再调用formatter的highlightTerm方法对文本进行重新构建
  4. 在本次轮询和下次单词出现之前对文本内容进行处理

查询工具类

[java] view plain copy
  1. package com.lucene.search.util;
  3. import java.io.File;
  4. import java.io.IOException;
  5. import java.nio.file.Paths;
  6. import java.util.Set;
  7. import java.util.concurrent.ExecutorService;
  9. import org.apache.lucene.analysis.Analyzer;
  10. import org.apache.lucene.analysis.standard.StandardAnalyzer;
  11. import org.apache.lucene.document.Document;
  12. import org.apache.lucene.index.DirectoryReader;
  13. import org.apache.lucene.index.IndexReader;
  14. import org.apache.lucene.index.MultiReader;
  15. import org.apache.lucene.index.Term;
  16. import org.apache.lucene.queryparser.classic.ParseException;
  17. import org.apache.lucene.queryparser.classic.QueryParser;
  18. import org.apache.lucene.search.BooleanQuery;
  19. import org.apache.lucene.search.IndexSearcher;
  20. import org.apache.lucene.search.MatchAllDocsQuery;
  21. import org.apache.lucene.search.NumericRangeQuery;
  22. import org.apache.lucene.search.Query;
  23. import org.apache.lucene.search.ScoreDoc;
  24. import org.apache.lucene.search.TermQuery;
  25. import org.apache.lucene.search.TopDocs;
  26. import org.apache.lucene.search.BooleanClause.Occur;
  27. import org.apache.lucene.search.highlight.Highlighter;
  28. import org.apache.lucene.search.highlight.InvalidTokenOffsetsException;
  29. import org.apache.lucene.search.highlight.QueryScorer;
  30. import org.apache.lucene.search.highlight.SimpleFragmenter;
  31. import org.apache.lucene.search.highlight.SimpleHTMLFormatter;
  32. import org.apache.lucene.store.FSDirectory;
  33. import org.apache.lucene.util.BytesRef;
  35. /**lucene索引查询工具类
  36. * @author lenovo
  37. *
  38. */
  39. public class SearchUtil {
  40. /**获取IndexSearcher对象
  41. * @param indexPath
  42. * @param service
  43. * @return
  44. * @throws IOException
  45. */
  46. public static IndexSearcher getIndexSearcherByParentPath(String parentPath,ExecutorService service) throws IOException{
  47. MultiReader reader = null;
  48. //设置
  49. try {
  50. File[] files = new File(parentPath).listFiles();
  51. IndexReader[] readers = new IndexReader[files.length];
  52. for (int i = 0 ; i < files.length ; i ++) {
  53. readers[i] = DirectoryReader.open(FSDirectory.open(Paths.get(files[i].getPath(), new String[0])));
  54. }
  55. reader = new MultiReader(readers);
  56. } catch (IOException e) {
  57. // TODO Auto-generated catch block
  58. e.printStackTrace();
  59. }
  60. return new IndexSearcher(reader,service);
  61. }
  62. /**多目录多线程查询
  63. * @param parentPath 父级索引目录
  64. * @param service 多线程查询
  65. * @return
  66. * @throws IOException
  67. */
  68. public static IndexSearcher getMultiSearcher(String parentPath,ExecutorService service) throws IOException{
  69. File file = new File(parentPath);
  70. File[] files = file.listFiles();
  71. IndexReader[] readers = new IndexReader[files.length];
  72. for (int i = 0 ; i < files.length ; i ++) {
  73. readers[i] = DirectoryReader.open(FSDirectory.open(Paths.get(files[i].getPath(), new String[0])));
  74. }
  75. MultiReader multiReader = new MultiReader(readers);
  76. IndexSearcher searcher = new IndexSearcher(multiReader,service);
  77. return searcher;
  78. }
  79. /**根据索引路径获取IndexReader
  80. * @param indexPath
  81. * @return
  82. * @throws IOException
  83. */
  84. public static DirectoryReader getIndexReader(String indexPath) throws IOException{
  85. return DirectoryReader.open(FSDirectory.open(Paths.get(indexPath, new String[0])));
  86. }
  87. /**根据索引路径获取IndexSearcher
  88. * @param indexPath
  89. * @param service
  90. * @return
  91. * @throws IOException
  92. */
  93. public static IndexSearcher getIndexSearcherByIndexPath(String indexPath,ExecutorService service) throws IOException{
  94. IndexReader reader = getIndexReader(indexPath);
  95. return new IndexSearcher(reader,service);
  96. }
  98. /**如果索引目录会有变更用此方法获取新的IndexSearcher这种方式会占用较少的资源
  99. * @param oldSearcher
  100. * @param service
  101. * @return
  102. * @throws IOException
  103. */
  104. public static IndexSearcher getIndexSearcherOpenIfChanged(IndexSearcher oldSearcher,ExecutorService service) throws IOException{
  105. DirectoryReader reader = (DirectoryReader) oldSearcher.getIndexReader();
  106. DirectoryReader newReader = DirectoryReader.openIfChanged(reader);
  107. return new IndexSearcher(newReader, service);
  108. }
  110. /**多条件查询类似于sql in
  111. * @param querys
  112. * @return
  113. */
  114. public static Query getMultiQueryLikeSqlIn(Query ... querys){
  115. BooleanQuery query = new BooleanQuery();
  116. for (Query subQuery : querys) {
  117. query.add(subQuery,Occur.SHOULD);
  118. }
  119. return query;
  120. }
  122. /**多条件查询类似于sql and
  123. * @param querys
  124. * @return
  125. */
  126. public static Query getMultiQueryLikeSqlAnd(Query ... querys){
  127. BooleanQuery query = new BooleanQuery();
  128. for (Query subQuery : querys) {
  129. query.add(subQuery,Occur.MUST);
  130. }
  131. return query;
  132. }
  133. /**从指定配置项中查询
  134. * @return
  135. * @param analyzer 分词器
  136. * @param field 字段
  137. * @param fieldType 字段类型
  138. * @param queryStr 查询条件
  139. * @param range 是否区间查询
  140. * @return
  141. */
  142. public static Query getQuery(String field,String fieldType,String queryStr,boolean range){
  143. Query q = null;
  144. try {
  145. if(queryStr != null && !"".equals(queryStr)){
  146. if(range){
  147. String[] strs = queryStr.split("\\|");
  148. if("int".equals(fieldType)){
  149. int min = new Integer(strs[0]);
  150. int max = new Integer(strs[1]);
  151. q = NumericRangeQuery.newIntRange(field, min, max, true, true);
  152. }else if("double".equals(fieldType)){
  153. Double min = new Double(strs[0]);
  154. Double max = new Double(strs[1]);
  155. q = NumericRangeQuery.newDoubleRange(field, min, max, true, true);
  156. }else if("float".equals(fieldType)){
  157. Float min = new Float(strs[0]);
  158. Float max = new Float(strs[1]);
  159. q = NumericRangeQuery.newFloatRange(field, min, max, true, true);
  160. }else if("long".equals(fieldType)){
  161. Long min = new Long(strs[0]);
  162. Long max = new Long(strs[1]);
  163. q = NumericRangeQuery.newLongRange(field, min, max, true, true);
  164. }
  165. }else{
  166. if("int".equals(fieldType)){
  167. q = NumericRangeQuery.newIntRange(field, new Integer(queryStr), new Integer(queryStr), true, true);
  168. }else if("double".equals(fieldType)){
  169. q = NumericRangeQuery.newDoubleRange(field, new Double(queryStr), new Double(queryStr), true, true);
  170. }else if("float".equals(fieldType)){
  171. q = NumericRangeQuery.newFloatRange(field, new Float(queryStr), new Float(queryStr), true, true);
  172. }else{
  173. Analyzer analyzer = new StandardAnalyzer();
  174. q = new QueryParser(field, analyzer).parse(queryStr);
  175. }
  176. }
  177. }else{
  178. q= new MatchAllDocsQuery();
  179. }
  181. System.out.println(q);
  182. } catch (ParseException e) {
  183. // TODO Auto-generated catch block
  184. e.printStackTrace();
  185. }
  186. return q;
  187. }
  188. /**根据field和值获取对应的内容
  189. * @param fieldName
  190. * @param fieldValue
  191. * @return
  192. */
  193. public static Query getQuery(String fieldName,Object fieldValue){
  194. Term term = new Term(fieldName, new BytesRef(fieldValue.toString()));
  195. return new TermQuery(term);
  196. }
  197. /**根据IndexSearcher和docID获取默认的document
  198. * @param searcher
  199. * @param docID
  200. * @return
  201. * @throws IOException
  202. */
  203. public static Document getDefaultFullDocument(IndexSearcher searcher,int docID) throws IOException{
  204. return searcher.doc(docID);
  205. }
  206. /**根据IndexSearcher和docID
  207. * @param searcher
  208. * @param docID
  209. * @param listField
  210. * @return
  211. * @throws IOException
  212. */
  213. public static Document getDocumentByListField(IndexSearcher searcher,int docID,Set<String> listField) throws IOException{
  214. return searcher.doc(docID, listField);
  215. }
  217. /**分页查询
  218. * @param page 当前页数
  219. * @param perPage 每页显示条数
  220. * @param searcher searcher查询器
  221. * @param query 查询条件
  222. * @return
  223. * @throws IOException
  224. */
  225. public static TopDocs getScoreDocsByPerPage(int page,int perPage,IndexSearcher searcher,Query query) throws IOException{
  226. TopDocs result = null;
  227. if(query == null){
  228. System.out.println(" Query is null return null ");
  229. return null;
  230. }
  231. ScoreDoc before = null;
  232. if(page != 1){
  233. TopDocs docsBefore = searcher.search(query, (page-1)*perPage);
  234. ScoreDoc[] scoreDocs = docsBefore.scoreDocs;
  235. if(scoreDocs.length > 0){
  236. before = scoreDocs[scoreDocs.length - 1];
  237. }
  238. }
  239. result = searcher.searchAfter(before, query, perPage);
  240. return result;
  241. }
  242. public static TopDocs getScoreDocs(IndexSearcher searcher,Query query) throws IOException{
  243. TopDocs docs = searcher.search(query, getMaxDocId(searcher));
  244. return docs;
  245. }
  246. /**高亮显示字段
  247. * @param searcher
  248. * @param field
  249. * @param keyword
  250. * @param preTag
  251. * @param postTag
  252. * @param fragmentSize
  253. * @return
  254. * @throws IOException
  255. * @throws InvalidTokenOffsetsException
  256. */
  257. public static String[] highlighter(IndexSearcher searcher,String field,String keyword,String preTag, String postTag,int fragmentSize) throws IOException, InvalidTokenOffsetsException{
  258. Term term = new Term("content",new BytesRef("lucene"));
  259. TermQuery termQuery = new TermQuery(term);
  260. TopDocs docs = getScoreDocs(searcher, termQuery);
  261. ScoreDoc[] hits = docs.scoreDocs;
  262. QueryScorer scorer = new QueryScorer(termQuery);
  263. SimpleHTMLFormatter simpleHtmlFormatter = new SimpleHTMLFormatter(preTag,postTag);//设定高亮显示的格式<B>keyword</B>,此为默认的格式
  264. Highlighter highlighter = new Highlighter(simpleHtmlFormatter,scorer);
  265. highlighter.setTextFragmenter(new SimpleFragmenter(fragmentSize));//设置每次返回的字符数
  266. Analyzer analyzer = new StandardAnalyzer();
  267. String[] result = new String[hits.length];
  268. for (int i = 0; i < result.length ; i++) {
  269. Document doc = searcher.doc(hits[i].doc);
  270. result[i] = highlighter.getBestFragment(analyzer, field, doc.get(field));
  271. }
  272. return result;
  273. }
  274. /**统计document的数量,此方法等同于matchAllDocsQuery查询
  275. * @param searcher
  276. * @return
  277. */
  278. public static int getMaxDocId(IndexSearcher searcher){
  279. return searcher.getIndexReader().maxDoc();
  280. }
  282. }
回复
分享到 :
0 人收藏
返回列表
高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

16级独孤
匿名技术用户

积分:7942463
帖子:1588486
精华:0
+ 关注 私信
期权论坛 期权论坛
期权网络科技版权所有
关于我们
联系我们
加入我们
反馈问题
免责声明
积分充值
统一社会信用代码
积分规则
网站地图
爱文库
下属网站
官方
新浪微博
微信公众号
◆
下载
表情包
App下载
期权论坛 期权论坛

QQ咨询|关于我们|Archiver|手机版|小黑屋|( 辽ICP备15012455号-4 ) Powered by 期权论坛 X3.2 © 2001-2016 期权工具网&期权论坛 Inc.

发布
内容

下载期权论坛手机APP