2014-07-24 06:56:10 -07:00
talon
=====
Mailgun library to extract message quotations and signatures.
2014-07-25 02:40:37 +00:00
If you ever tried to parse message quotations or signatures you know that absence of any formatting standards in this area could make this task a nightmare. Hopefully this library will make your life much easier. The name of the project is inspired by TALON - multipurpose robot designed to perform missions ranging from reconnaissance to combat and operate in a number of hostile environments. That’
2014-07-24 06:56:10 -07:00
Usage
-----
Here’
message:
.. code :: python
import talon
from talon import quotations
talon.init()
text = """Reply
-----Original Message-----
Quote"""
reply = quotations.extract_from(text, 'text/plain')
reply = quotations.extract_from_plain(text)
# reply == "Reply"
To extract a reply from html:
.. code :: python
html = """Reply
<blockquote>
<div>
On 11-Apr-2011, at 6:54 PM, Bob <bob@example.com> wrote:
</div>
<div>
Quote
</div>
</blockquote>"""
reply = quotations.extract_from(html, 'text/html')
reply = quotations.extract_from_html(html)
# reply == "<html><body><p>Reply</p></body></html>"
2014-07-24 07:22:25 -07:00
Often the best way is the easiest one. Here’
2014-07-24 06:56:10 -07:00
signature from email message without any
2014-07-24 07:22:25 -07:00
machine learning fancy stuff:
2014-07-24 06:56:10 -07:00
.. code :: python
from talon.signature.bruteforce import extract_signature
message = """Wow. Awesome!
--
Bob Smith"""
text, signature = extract_signature(message)
# text == "Wow. Awesome!"
# signature == "--\nBob Smith"
Quick and works like a charm 90% of the time. For other 10% you can use
the power of machine learning algorithms:
.. code :: python
2014-09-14 09:03:10 -07:00
import talon
# don't forget to init the library first
# it loads machine learning classifiers
talon.init()
2014-07-24 06:56:10 -07:00
from talon import signature
message = """Thanks Sasha, I can't go any higher and is why I limited it to the
homepage.
John Doe
via mobile"""
text, signature = signature.extract(message, sender='john.doe@example.com')
# text == "Thanks Sasha, I can't go any higher and is why I limited it to the\nhomepage."
# signature == "John Doe\nvia mobile"
2015-07-02 21:46:27 -04:00
For machine learning talon currently uses the `scikit-learn`_ library to build SVM
2014-07-24 06:56:10 -07:00
classifiers. The core of machine learning algorithm lays in
`` talon.signature.learning package `` . It defines a set of features to
apply to a message (`` featurespace.py `` ), how data sets are built
(`` dataset.py `` ), classifier’ `` classifier.py `` ).
2016-05-31 15:14:32 -07:00
Currently the data used for training is taken from our personal email
2014-07-24 06:56:10 -07:00
conversations and from `ENRON`_ dataset. As a result of applying our set
of features to the dataset we provide files `` classifier `` and
`` train.data `` that don’
used to load trained classifier. Those files should be regenerated every
time the feature/data set is changed.
2015-07-02 21:46:27 -04:00
To regenerate the model files, you can run
.. code :: sh
python train.py
or
.. code :: python
from talon.signature import EXTRACTOR_FILENAME, EXTRACTOR_DATA
from talon.signature.learning.classifier import train, init
train(init(), EXTRACTOR_DATA, EXTRACTOR_FILENAME)
2016-06-10 14:10:53 -07:00
Open-source Dataset
-------------------
Recently we started a `forge`_ project to create an open-source, annotated dataset of raw emails. In the project we
used a subset of `ENRON`_ data, cleansed of private, health and financial information by `EDRM`_ . At the moment over 190
emails are annotated. Any contribution and collaboration on the project are welcome. Once the dataset is ready we plan to
start using it for talon.
2015-07-02 21:46:27 -04:00
.. _scikit-learn: http://scikit-learn.org
2014-07-24 06:56:10 -07:00
.. _ENRON: https://www.cs.cmu.edu/~enron/
2016-06-10 14:10:53 -07:00
.. _EDRM: http://www.edrm.net/resources/data-sets/edrm-enron-email-data-set
.. _forge: https://github.com/mailgun/forge
2018-11-02 15:21:36 +03:00
Training on your dataset
------------------------
talon comes with a pre-processed dataset and a pre-trained classifier. To retrain the classifier on your own dataset of raw emails, structure and annotate them in the same way the `forge`_ project does. Then do:
.. code :: python
from talon.signature.learning.dataset import build_extraction_dataset
from talon.signature.learning import classifier as c
build_extraction_dataset("/path/to/your/P/folder", "/path/to/talon/signature/data/train.data")
c.train(c.init(), "/path/to/talon/signature/data/train.data", "/path/to/talon/signature/data/classifier")
Note that for signature extraction you need just the folder with the positive samples with annotated signature lines (P folder).
.. _forge: https://github.com/mailgun/forge
2014-07-24 06:56:10 -07:00
Research
--------
The library is inspired by the following research papers and projects:
- http://www.cs.cmu.edu/~vitor/papers/sigFilePaper_finalversion.pdf
- http://www.cs.cornell.edu/people/tj/publications/joachims_01a.pdf
