ACM Home Page
Please provide us with feedback. Feedback
Digital Library logoTake a look at the new version of this page: [ beta version ]. Tell us what you think.
Bursty and hierarchical structure in streams
Full text PdfPdf (1.21 MB)
Source International Conference on Knowledge Discovery and Data Mining archive
Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining table of contents
Edmonton, Alberta, Canada
SESSION: Streams and time series table of contents
Pages: 91 - 101  
Year of Publication: 2002
ISBN:1-58113-567-X
Author
Jon Kleinberg  Cornell University, Ithaca NY
Sponsors
SIGKDD: ACM Special Interest Group on Knowledge Discovery in Data
SIGMOD: ACM Special Interest Group on Management of Data
: AAAI
Publisher
ACM  New York, NY, USA
Bibliometrics
Downloads (6 Weeks): 19,   Downloads (12 Months): 154,   Citation Count: 85
Additional Information:

abstract   references   cited by   index terms   collaborative colleagues  

Tools and Actions: Request Permissions Request Permissions    Review this Article  
DOI Bookmark: Use this link to bookmark this Article: http://doi.acm.org/10.1145/775047.775061
What is a DOI?

ABSTRACT

A fundamental problem in text data mining is to extract meaningful structure from document streams that arrive continuously over time. E-mail and news articles are two natural examples of such streams, each characterized by topics that appear, grow in intensity for a period of time, and then fade away. The published literature in a particular research field can be seen to exhibit similar phenomena over a much longer time scale. Underlying much of the text mining work in this area is the following intuitive premise --- that the appearance of a topic in a document stream is signaled by a "burst of activity," with certain features rising sharply in frequency as the topic emerges.The goal of the present work is to develop a formal approach for modeling such "bursts," in such a way that they can be robustly and efficiently identified, and can provide an organizational framework for analyzing the underlying content. The approach is based on modeling the stream using an infinite-state automaton, in which bursts appear naturally as state transitions; in some ways, it can be viewed as drawing an analogy with models from queueing theory for bursty network traffic. The resulting algorithms are highly efficient, and yield a nested representation of the set of bursts that imposes a hierarchical structure on the overall stream. Experiments with e-mail and research paper archives suggest that the resulting structures have a natural meaning in terms of the content that gave rise to them.


REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

 
1
 
2
J. Allan, J.G. Carbonell, G. Doddington, J. Yamron, Y. Yang, 'Topic Detection and Tracking Pilot Study: Final Report," Proc. DARPA Broadcast News Transcription and Understanding Workshop, Feb. 1998.
3
4
 
5
6
 
7
A. Birrell, S. Perl, M. Schroeder, T. Wobber, The Pachyderm E-mail System, 1997, at http://www.research.compaq.com/SRC/pachyderm/.
 
8
T. Blanton, Ed., White House E-mail, New Press, 1995.
9
 
10
C. Chatfield, The Analysis of time Series: An Introduction, Chapman and Hall, 1996.
 
11
S. Chatman, Story and Discourse: Narrative Structure in Fiction and Film, Cornell Univ. Press, 1978.
 
12
D. Chudova, P. Smyth, "Unsupervised identification of sequential patterns under a Markov assumption," KDD Workshop on Temporal Data Mining, 2001.
 
13
W. Cohen. "Learning rules that classify e-mail." Proc. AAAI Spring Symp. Machine Learning and Information Access, 1996.
 
14
T. Cover, P. Hart, "Nearest neighbor pattern classification," IEEE Trans. Information Theory IT-13(1967), pp. 21--27.
 
15
W. Davison, L. Wall, S. Barber, trn, 1993 http://web.mit.edu/afs/sipb/project/trn/src/trn-3.6/.
 
16
R. Ehrich, J. Foith, "Representation of Random Waveforms by Relational 'Trees," IEEE Trans. Computers, C25:7(1976).
 
17
 
18
E.M. Forster, Aspects of the Novel, Harcourt, Brace, and World, Inc. 1927.
 
19
G. Gay, M. Stefanone, M. Grace-Martin, H. Hembrooke, "The effect of wireless computing in collaborative learning environments," Intl. J. Human-Computer Interaction, to appear.
 
20
G. Genette, Narrative Discourse: An Essay in Method, English translation (J.E. Lewin), Cornell Univ. Press, 1980.
 
21
G. Genette, Narrative Discourse Revisited, English translation (J.E. Lewin), Cornell Univ. Press, 1988.
 
22
 
23
T. Gruber, Hypermail, Enterprise Integration Technologies.
24
 
25
J. Han, W. Gong, Y. Yin, "Mining Segment-Wise Periodic Patterns in Time-Related Databases", Proc. Intl. Conf. Knowledge Discovery and Data Mining, 1998.
 
26
 
27
 
28
D. Hawkins, "Point estimation of the parameters of piecewise regression models," Applied Statistics 25(1976)
29
 
30
J. Helfman, C. Isbell, "Ishmail: Immediate identification of important information," AT&T Labs Technical Report, 1995.
31
 
32
D. Hudson, "Fitting segmented curves whose join points have to be estimated," Journal of the American Statistical Association 61(1966) pp. 1097--1129.
 
33
F. P. Kelly, "Notes on effective bandwidths," in Stochastic Networks: Theory and Applications, (F.P. Kelly, S. Zachary, I. Ziedins, eds.) Oxford Univ. Press, 1996.
 
34
E. Keogh, P. Smyth, "A probabilistic approach to fast pattern matching in time series databases," Proc. Intl. Conf. Knowledge Discovery and Data Mining, 1997.
 
35
J.l. Klein et al., Plaintiffs' Memorandum in Support of Proposed Final Judgment, United States of America v. Microsoft Corporation and State of New York, ex rel. Attorney General Eliot Spitzer, et al., v. Microsoft Corporation, Civil Actions No. 98-1232 (TPJ) and 98-1233 (TPJ), April 2000.
 
36
M. Last, Y. Klein, A. Kandel, "Knowledge Discovery in Time Series Databases," IEEE Transactions on Systems, Man, and Cybernetics 31B(2001).
 
37
V. Lavrenko, M. Schmill, D. Lawrie, P. Ogilvie, D. Jensen, J. Allan, "Mining of Concurrent Text and Time-Series," KDD-2000 Workshop on Text Mining, 2000.
 
38
 
39
S.S. Lukesh, "E-mail and potential loss to future archives and scholarship, or, The dog that didn't bark," First Monday 4(9) (September 1999), at http://firstmonday.org
40
41
 
42
H. Mannila, H. Toivonen, A.I. Verkamo, "Discovering frequent episodes in sequences," Proc. Intl. Conf. on Knowledge Discovery and Data Mining, 1995.
 
43
R. Martin, V. Yohai, "Data mining for unusual movements in temporal data," KDD Wkshp. Temporal Data Mining, 2001.
44
 
45
 
46
R. Moore, C. Baru, A. Rajasekar, B. Ludaescher, R. Marciano, M. Wan, W. Schroeder, A. Gupta, "Collection-Based Persistent Digital Archives -- Part 2," D-Lib Magazine, 6(2000).
 
47
K. Murphy, M. Paskin, "Linear time inference in hierarchical HMMs," Advances in Neural Information Processing Systems (NIPS) 14, 2001.
 
48
F. Olsen, "Facing Flood of E-Mail, Archives Seeks Help From Supercomputer Researchers," Chronicle of Higher Education, August 24, 1999.
 
49
T. Payne, P. Edwards, "Interface agents that learn: An investigation of learning issues in a mail agent interface," Applied Artificial Intelligence 11(1997), pp. 1--32.
50
 
51
L. Rabiner, "A tutorial on hidden Markov models and selected applications in speech recognition," Proc. IEEE 77(1989).
 
52
M. Redmond, B. Adelson, "AlterEgo e-mail filtering agent," Proc. AAAI Workshop on Case-Based Reasoning, 1998.
 
53
J. Rennie, "ifile: An application of machine learning to e-mail filtering," Proc. KDD Workshop on Text Mining, 2000.
 
54
M. Sahami, S. Dumais, D. Heckerman, E. Horvitz. "A Bayesian approach to filtering junk email," Proc. AAAI Workshop on Learning for Text Categorization, 1998.
 
55
B. Schneier, Applied Cryptography Wiley, 1996.
56
 
57
 
58
S. Shaw, R. DeFigueiredo, "Structural Processing of Waveforms as Trees," IEEE Transactions on Acoustics, Speech, and Signal Processing 38:2(1990)
59
60
 
61
R. Swan, D. Jensen, "TimeMines: Constructing Timelines with Statistical Models of Word Usage," KDD-2000 Workshop on Text Mining, 2000.
62
 
63
64
65

CITED BY  85