3.9.1

Fix deprecation warning regarding invalid escape sequences.

.travis.yml

@@ -1,8 +1,13 @@
 language: python
 python:
-  - 2.6
+  - 3.5
-  - 2.7
+  - 3.6
-  - 3.3
+# Enable 3.7 without globally enabling sudo and dist: xenial for other build jobs
+matrix:
+  include:
+    - python: 3.7
+      dist: xenial
+      sudo: true
 env:
   global:
   - secure: USjLDneiXlVvEjkUVqTt+LBi0XJ4QhkRcJzqVXA9gEau1NTjAkNTPmHjUbOygp0dkfoV0uWrZKCw6fL1g+HJgWl0vHeHzcNl4mUkA+OwkGFHgaeIhvUfnyyJA8P3Zm21XHC+ehzMpEFN5fVNNhREjnRj+CXMc0FgA6knwBRobu4=
@@ -17,15 +22,11 @@ env:
   - TEST_LIST_SLUG=team
   - TEST_LIST_OWNER_SCREEN_NAME=twitterapi
   - ACCESS_TOKEN_B64=U2FsdGVkX18QdBhvMNshM4PGy04tU3HLwKP+nNSoNZHKsvGLjELcWEXN2LIu/T+yngX1vGONf9lo14ElnfS4k7sfhiru8phR4+rZuBVP3bDvC2A6fXJuhuLqNhBrWqg32WQewvxLWDWBoKmnvRHg5b74GHh+IN/12tU0cBF2HK8=
-install: 
+install:
   - pip install -r requirements.txt
-  - if [[ $TRAVIS_PYTHON_VERSION == '2.6' ]]; then pip install unittest2; fi
 script: nosetests -v -w tests/ --logging-filter="twython" --with-cov --cov twython --cov-config .coveragerc --cov-report term-missing
 notifications:
   email: false
-branches:
-  only:
-    - master
 after_success:
   - coveralls
 before_script:

HISTORY.md

@@ -1,23 +1,39 @@
-.. :changelog:
+# History
 
-History
+## 3.8.0 (2020-04-02)
--------
+- Bump release with latest patches from GitHub.
+- Fix Direct Messages with patches from @manuelcortez.
 
-3.4.0 (2016-30-04)
+## 3.7.0 (2018-07-05)
-++++++++++++++++++
+- Fixes for cursoring API endpoints
+- Improve `html_for_tweet()` parsing
+- Documentation cleanup
+- Documentation for cursor's `return_pages` keyword argument
+- Update links to Twitter API in documentation
+- Added `create_metadata` endpoint
+- Raise error for when cursor is not provided a callable
+
+## 3.6.0 (2017-23-08)
+- Improve replacing of entities with links in `html_for_tweet()`
+- Update classifiers for PyPI
+
+## 3.5.0 (2017-06-06)
+- Added support for "symbols" in `Twython.html_for_tweet()`
+- Added support for extended tweets in `Twython.html_for_tweet()`
+- You can now check progress of video uploads to Twitter when using `Twython.upload_video()`
+
+## 3.4.0 (2016-30-04)
 - Added `upload_video` endpoint
 - Fix quoted status checks in `html_for_tweet`
 - Fix `html_for_tweet` method response when hashtag/mention is a substring of another
 
-3.3.0 (2015-18-07)
+## 3.3.0 (2015-18-07)
-++++++++++++++++++
 - Added support for muting users
 - Fix typos in documentation
 - Updated documentation examples
 - Added dynamic filtering to streamer
 
-3.2.0 (2014-10-30)
+## 3.2.0 (2014-10-30)
-++++++++++++++++++
 - PEP8'd some code
 - Added `lookup_status` function to `endpoints.py`
 - Added keyword argument to `cursor` to return full pages rather than individual results
@@ -30,23 +46,16 @@ History
 - Deprecating `update_with_media` per Twitter API 1.1 (https://dev.twitter.com/rest/reference/post/statuses/update_with_media)
 - Unpin `requests` and `requests-oauthlib` in `requirements.txt`
 
-
+## 3.1.2 (2013-12-05)
-3.1.2 (2013-12-05)
-++++++++++++++++++
-
 - Fixed Changelog (HISTORY.rst)
 
-3.1.1 (2013-12-05)
+## 3.1.1 (2013-12-05)
-++++++++++++++++++
-
 - Update `requests` version to 2.1.0.
 - Fixed: Streaming issue where `Exceptions` in handlers or `on_success` which subclass `ValueError` would previously be caught and reported as a JSON decoding problem, and `on_error()` would be called (with status_code=200)
 - Fixed issue where XML was returned when bad tokens were passed to `get_authorized_tokens`
 - Fixed import for `setup` causing installation to fail on some devices (eg. Nokia N9/MeeGo)
 
-3.1.0 (2013-09-25)
+## 3.1.0 (2013-09-25)
-++++++++++++++++++
-
 - Added ``html_for_tweet`` static method. This method accepts a tweet object returned from a Twitter API call and will return a string with urls, mentions and hashtags in the tweet replaced with HTML.
 - Pass ``client_args`` to the streaming ``__init__``, much like in core Twython (you can pass headers, timeout, hooks, proxies, etc.).
 - Streamer has new parameter ``handlers`` which accepts a list of strings related to functions that are apart of the Streaming class and start with "on\_". i.e. ['delete'] is passed, when 'delete' is received from a stream response; ``on_delete`` will be called.
@@ -58,9 +67,7 @@ History
 - Fixed streaming issue where results wouldn't be returned for streams that weren't so active (See https://github.com/ryanmcgrath/twython/issues/202#issuecomment-19915708)
 - Streaming API now uses ``_transparent_params`` so when passed ``True`` or ``False`` or an array, etc. Twython formats it to meet Twitter parameter standards (i.e. ['ryanmcgrath', 'mikehelmick', 'twitterapi'] would convert to string 'ryanmcgrath,mikehelmick,twitterapi')
 
-3.0.0 (2013-06-18)
+## 3.0.0 (2013-06-18)
-++++++++++++++++++
-
 - Changed ``twython/twython.py`` to ``twython/api.py`` in attempt to make structure look a little neater
 - Removed all camelCase function access (anything like ``getHomeTimeline`` is now ``get_home_timeline``)
 - Removed ``shorten_url``. With the ``requests`` library, shortening a URL on your own is simple enough
@@ -79,9 +86,7 @@ History
 - Added ``invalidate_token`` API method which allows registed apps to revoke an access token presenting its client credentials
 - ``get_lastfunction_header`` now accepts a ``default_return_value`` parameter. This means that if you pass a second value (ex. ``Twython.get_lastfunction_header('x-rate-limit-remaining', 0)``) and the value is not found, it returns your default value
 
-2.10.1 (2013-05-29)
+## 2.10.1 (2013-05-29)
-++++++++++++++++++
-
 - More test coverage!
 - Fix ``search_gen``
 - Fixed ``get_lastfunction_header`` to actually do what its docstring says, returns ``None`` if header is not found
@@ -91,9 +96,7 @@ History
 - No longer raise ``TwythonStreamError`` when stream line can't be decoded. Instead, sends signal to ``TwythonStreamer.on_error``
 - Allow for (int, long, float) params to be passed to Twython Twitter API functions in Python 2, and (int, float) in Python 3
 
-2.10.0 (2013-05-21)
+## 2.10.0 (2013-05-21)
-++++++++++++++++++
-
 - Added ``get_retweeters_ids`` method
 - Fixed ``TwythonDeprecationWarning`` on camelCase functions if the camelCase was the same as the PEP8 function (i.e. ``Twython.retweet`` did not change)
 - Fixed error message bubbling when error message returned from Twitter was not an array (i.e. if you try to retweet something twice, the error is not found at index 0)
@@ -104,20 +107,14 @@ History
 - Cleaned up ``Twython.construct_api_url``, uses "transparent" parameters (see 4th bullet in this version for explaination)
 - Update ``requests`` and ``requests-oauthlib`` requirements, fixing posting files AND post data together, making authenticated requests in general in Python 3.3
 
-2.9.1 (2013-05-04)
+## 2.9.1 (2013-05-04)
-++++++++++++++++++
-
 - "PEP8" all the functions. Switch functions from camelCase() to underscore_funcs(). (i.e. ``updateStatus()`` is now ``update_status()``)
 
-2.9.0 (2013-05-04)
+## 2.9.0 (2013-05-04)
-++++++++++++++++++
-
 - Fixed streaming issue #144, added ``TwythonStreamer`` to aid users in a friendly streaming experience (streaming examples in ``examples`` and README's have been updated as well)
 - ``Twython`` now requires ``requests-oauthlib`` 0.3.1, fixes #154 (unable to upload media when sending POST data with the file)
 
-2.8.0 (2013-04-29)
+## 2.8.0 (2013-04-29)
-++++++++++++++++++
-
 - Added a ``HISTORY.rst`` to start tracking history of changes
 - Updated ``twitter_endpoints.py`` to ``endpoints.py`` for cleanliness
 - Removed twython3k directory, no longer needed
@@ -140,36 +137,24 @@ History
 - Twython now takes ``ssl_verify`` parameter, defaults True. Set False if you're having development server issues
 - Removed internal ``_media_update`` function, we could have always just used ``self.post``
 
-2.7.3 (2013-04-12)
+## 2.7.3 (2013-04-12)
-++++++++++++++++++
-
 - Fixed issue where Twython Exceptions were not being logged correctly
 
-2.7.2 (2013-04-08)
+## 2.7.2 (2013-04-08)
-++++++++++++++++++
-
 - Fixed ``AttributeError`` when trying to decode the JSON response via ``Response.json()``
 
-2.7.1 (2013-04-08)
+## 2.7.1 (2013-04-08)
-++++++++++++++++++
-
 - Removed ``simplejson`` dependency
 - Fixed ``destroyDirectMessage``, ``createBlock``, ``destroyBlock`` endpoints in ``twitter_endpoints.py``
 - Added ``getProfileBannerSizes`` method to ``twitter_endpoints.py``
 - Made oauth_verifier argument required in ``get_authorized_tokens``
 - Update ``updateProfileBannerImage`` to use v1.1 endpoint
 
-2.7.0 (2013-04-04)
+## 2.7.0 (2013-04-04)
-++++++++++++++++++
-
 - New ``showOwnedLists`` method
 
-2.7.0 (2013-03-31)
+## 2.7.0 (2013-03-31)
-++++++++++++++++++
-
 - Added missing slash to ``getMentionsTimeline`` in ``twitter_endpoints.py``
 
-2.6.0 (2013-03-29)
+## 2.6.0 (2013-03-29)
-++++++++++++++++++
-
 - Updated ``twitter_endpoints.py`` to better reflect order of API endpoints on the Twitter API v1.1 docs site

MANIFEST.in

@@ -1 +1,9 @@
-include README.rst HISTORY.rst LICENSE requirements.txt
+include README.md HISTORY.md LICENSE requirements.txt
+
+recursive-include docs *
+prune docs/_build
+
+recursive-include examples *.py
+
+recursive-include tests *.py
+recursive-include tests/tweets *.json

README.md

@@ -0,0 +1,169 @@
+# Twython
+
+<a href="https://pypi.python.org/pypi/twython"><img src="https://img.shields.io/pypi/v/twython.svg?style=flat-square"></a>
+<a href="https://pypi.python.org/pypi/twython"><img src="https://img.shields.io/pypi/dw/twython.svg?style=flat-square"></a>
+<a href="https://travis-ci.org/ryanmcgrath/twython"><img src="https://img.shields.io/travis/ryanmcgrath/twython.svg?style=flat-square"></a>
+<a href="https://coveralls.io/r/ryanmcgrath/twython?branch=master"><img src="https://img.shields.io/coveralls/ryanmcgrath/twython/master.svg?style=flat-square"></a>
+
+`Twython` is a Python library providing an easy way to access Twitter data. Supports Python 3. It's been battle tested by companies, educational institutions and individuals alike. Try it today!
+
+**Note**: As of Twython 3.7.0, there's a general call for maintainers put out. If you find the project useful and want to help out, reach out to Ryan with the info from the bottom of this README. Great open source project to get your feet wet with!
+
+## Features
+- Query data for:
+    - User information
+    - Twitter lists
+    - Timelines
+    - Direct Messages
+    - and anything found in [the docs](https://developer.twitter.com/en/docs)
+- Image Uploading:
+    - Update user status with an image
+    - Change user avatar
+    - Change user background image
+    - Change user banner image
+- OAuth 2 Application Only (read-only) Support
+- Support for Twitter's Streaming API
+- Seamless Python 3 support!
+
+## Installation
+Install Twython via pip:
+
+```bash
+$ pip install twython
+```
+
+If you're on a legacy project that needs Python 2.7 support, you can install the last version of Twython that supported 2.7:
+
+```
+pip install twython==3.7.0`
+```
+
+Or, if you want the code that is currently on GitHub:
+
+```bash
+git clone git://github.com/ryanmcgrath/twython.git
+cd twython
+python setup.py install
+```
+
+## Documentation
+Documentation is available at https://twython.readthedocs.io/en/latest/
+
+## Starting Out
+First, you'll want to head over to https://apps.twitter.com and register an application!
+
+After you register, grab your applications `Consumer Key` and `Consumer Secret` from the application details tab.
+
+The most common type of authentication is Twitter user authentication using OAuth 1. If you're a web app planning to have users sign up with their Twitter account and interact with their timelines, updating their status, and stuff like that this **is** the authentication for you!
+
+First, you'll want to import Twython
+
+```python
+from twython import Twython
+```
+
+## Obtain Authorization URL
+Now, you'll want to create a Twython instance with your `Consumer Key` and `Consumer Secret`:
+
+- Only pass *callback_url* to *get_authentication_tokens* if your application is a Web Application
+- Desktop and Mobile Applications **do not** require a callback_url
+
+```python
+APP_KEY = 'YOUR_APP_KEY'
+APP_SECRET = 'YOUR_APP_SECRET'
+
+twitter = Twython(APP_KEY, APP_SECRET)
+
+auth = twitter.get_authentication_tokens(callback_url='http://mysite.com/callback')
+```
+
+From the `auth` variable, save the `oauth_token` and `oauth_token_secret` for later use (these are not the final auth tokens). In Django or other web frameworks, you might want to store it to a session variable
+
+```python
+OAUTH_TOKEN = auth['oauth_token']
+OAUTH_TOKEN_SECRET = auth['oauth_token_secret']
+```
+
+Send the user to the authentication url, you can obtain it by accessing
+
+```python
+auth['auth_url']
+```
+
+## Handling the Callback
+If your application is a Desktop or Mobile Application *oauth_verifier* will be the PIN code
+
+After they authorize your application to access some of their account details, they'll be redirected to the callback url you specified in `get_authentication_tokens`.
+
+You'll want to extract the `oauth_verifier` from the url.
+
+Django example:
+
+```python
+oauth_verifier = request.GET['oauth_verifier']
+```
+
+Now that you have the `oauth_verifier` stored to a variable, you'll want to create a new instance of Twython and grab the final user tokens
+
+```python
+twitter = Twython(
+    APP_KEY, APP_SECRET,
+    OAUTH_TOKEN, OAUTH_TOKEN_SECRET
+)
+
+final_step = twitter.get_authorized_tokens(oauth_verifier)
+```
+
+Once you have the final user tokens, store them in a database for later use::
+
+```python
+    OAUTH_TOKEN = final_step['oauth_token']
+    OAUTH_TOKEN_SECRET = final_step['oauth_token_secret']
+```
+
+For OAuth 2 (Application Only, read-only) authentication, see [our documentation](https://twython.readthedocs.io/en/latest/usage/starting_out.html#oauth-2-application-authentication).
+
+## Dynamic Function Arguments
+Keyword arguments to functions are mapped to the functions available for each endpoint in the Twitter API docs. Doing this allows us to be incredibly flexible in querying the Twitter API, so changes to the API aren't held up from you using them by this library.
+
+Basic Usage
+-----------
+
+**Function definitions (i.e. get_home_timeline()) can be found by reading over twython/endpoints.py**
+
+Create a Twython instance with your application keys and the users OAuth tokens
+
+```python
+from twython import Twython
+twitter = Twython(APP_KEY, APP_SECRET, OAUTH_TOKEN, OAUTH_TOKEN_SECRET)
+```
+
+## Authenticated Users Home Timeline
+```python
+twitter.get_home_timeline()
+```
+
+## Updating Status
+This method makes use of dynamic arguments, [read more about them](https://twython.readthedocs.io/en/latest/usage/starting_out.html#dynamic-function-arguments).
+
+```python
+twitter.update_status(status='See how easy using Twython is!')
+```
+
+## Advanced Usage
+- [Advanced Twython Usage](https://twython.readthedocs.io/en/latest/usage/advanced_usage.html)
+- [Streaming with Twython](https://twython.readthedocs.io/en/latest/usage/streaming_api.html)
+
+
+## Questions, Comments, etc?
+My hope is that Twython is so simple that you'd never *have* to ask any questions, but if you feel the need to contact me for this (or other) reasons, you can hit me up at ryan@venodesigns.net.
+
+Or if I'm to busy to answer, feel free to ping mikeh@ydekproductions.com as well.
+
+Follow us on Twitter:
+
+- [@ryanmcgrath](https://twitter.com/ryanmcgrath)
+- [@mikehelmick](https://twitter.com/mikehelmick)
+
+## Want to help?
+Twython is useful, but ultimately only as useful as the people using it (say that ten times fast!). If you'd like to help, write example code, contribute patches, document things on the wiki, tweet about it. Your help is always appreciated!

README.rst

@@ -1,222 +0,0 @@
-Twython
-=======
-
-
-.. image:: https://img.shields.io/pypi/v/twython.svg?style=flat-square
-  :target: https://pypi.python.org/pypi/twython 
-
-.. image:: https://img.shields.io/pypi/dw/twython.svg?style=flat-square
-  :target: https://pypi.python.org/pypi/twython 
- 
-.. image:: https://img.shields.io/travis/ryanmcgrath/twython.svg?style=flat-square
-  :target: https://travis-ci.org/ryanmcgrath/twython 
-
-.. image:: https://img.shields.io/coveralls/ryanmcgrath/twython/master.svg?style=flat-square
-  :target: https://coveralls.io/r/ryanmcgrath/twython?branch=master 
-
-``Twython`` is the premier Python library providing an easy (and up-to-date) way to access Twitter data. Actively maintained and featuring support for Python 2.6+ and Python 3. It's been battle tested by companies, educational institutions and individuals alike. Try it today!
-
-Features
---------
-
-- Query data for:
-    - User information
-    - Twitter lists
-    - Timelines
-    - Direct Messages
-    - and anything found in `the docs <https://dev.twitter.com/docs/api/1.1>`_
-- Image Uploading:
-    - Update user status with an image
-    - Change user avatar
-    - Change user background image
-    - Change user banner image
-- OAuth 2 Application Only (read-only) Support
-- Support for Twitter's Streaming API
-- Seamless Python 3 support!
-
-Installation
-------------
-
-Install Twython via `pip <http://www.pip-installer.org/>`_
-
-.. code-block:: bash
-
-    $ pip install twython
-
-or, with `easy_install <http://pypi.python.org/pypi/setuptools>`_
-
-.. code-block:: bash
-
-    $ easy_install twython
-
-But, hey... `that's up to you <http://www.pip-installer.org/en/latest/other-tools.html#pip-compared-to-easy-install>`_.
-
-Or, if you want the code that is currently on GitHub
-
-.. code-block:: bash
-
-    git clone git://github.com/ryanmcgrath/twython.git
-    cd twython
-    python setup.py install
-
-Documentation
--------------
-
-Documentation is available at https://twython.readthedocs.org/en/latest/
-
-Starting Out
-------------
-
-First, you'll want to head over to https://dev.twitter.com/apps and register an application!
-
-After you register, grab your applications ``Consumer Key`` and ``Consumer Secret`` from the application details tab.
-
-The most common type of authentication is Twitter user authentication using OAuth 1. If you're a web app planning to have users sign up with their Twitter account and interact with their timelines, updating their status, and stuff like that this **is** the authentication for you!
-
-First, you'll want to import Twython
-
-.. code-block:: python
-
-    from twython import Twython
-
-Authentication
-~~~~~~~~~~~~~~
-
-Obtain Authorization URL
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-Now, you'll want to create a Twython instance with your ``Consumer Key`` and ``Consumer Secret``
-
-    Only pass *callback_url* to *get_authentication_tokens* if your application is a Web Application
-
-    Desktop and Mobile Applications **do not** require a callback_url
-
-.. code-block:: python
-
-    APP_KEY = 'YOUR_APP_KEY'
-    APP_SECRET = 'YOUR_APP_SECRET'
-
-    twitter = Twython(APP_KEY, APP_SECRET)
-
-    auth = twitter.get_authentication_tokens(callback_url='http://mysite.com/callback')
-
-From the ``auth`` variable, save the ``oauth_token`` and ``oauth_token_secret`` for later use (these are not the final auth tokens). In Django or other web frameworks, you might want to store it to a session variable
-
-.. code-block:: python
-
-    OAUTH_TOKEN = auth['oauth_token']
-    OAUTH_TOKEN_SECRET = auth['oauth_token_secret']
-
-Send the user to the authentication url, you can obtain it by accessing
-
-.. code-block:: python
-
-    auth['auth_url']
-
-Handling the Callback
-^^^^^^^^^^^^^^^^^^^^^
-
-    If your application is a Desktop or Mobile Application *oauth_verifier* will be the PIN code
-
-After they authorize your application to access some of their account details, they'll be redirected to the callback url you specified in ``get_authentication_tokens``
-
-You'll want to extract the ``oauth_verifier`` from the url.
-
-Django example:
-
-.. code-block:: python
-
-    oauth_verifier = request.GET['oauth_verifier']
-
-Now that you have the ``oauth_verifier`` stored to a variable, you'll want to create a new instance of Twython and grab the final user tokens
-
-.. code-block:: python
-
-    twitter = Twython(APP_KEY, APP_SECRET,
-                      OAUTH_TOKEN, OAUTH_TOKEN_SECRET)
-
-    final_step = twitter.get_authorized_tokens(oauth_verifier)
-
-Once you have the final user tokens, store them in a database for later use::
-
-    OAUTH_TOKEN = final_step['oauth_token']
-    OAUTH_TOKEN_SECRET = final_step['oauth_token_secret']
-
-For OAuth 2 (Application Only, read-only) authentication, see `our documentation <https://twython.readthedocs.org/en/latest/usage/starting_out.html#oauth-2-application-authentication>`_
-
-Dynamic Function Arguments
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Keyword arguments to functions are mapped to the functions available for each endpoint in the Twitter API docs. Doing this allows us to be incredibly flexible in querying the Twitter API, so changes to the API aren't held up from you using them by this library.
-
-Basic Usage
------------
-
-**Function definitions (i.e. get_home_timeline()) can be found by reading over twython/endpoints.py**
-
-Create a Twython instance with your application keys and the users OAuth tokens
-
-.. code-block:: python
-
-    from twython import Twython
-    twitter = Twython(APP_KEY, APP_SECRET,
-                      OAUTH_TOKEN, OAUTH_TOKEN_SECRET)
-
-Authenticated Users Home Timeline
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Documentation: https://dev.twitter.com/docs/api/1.1/get/statuses/home_timeline
-
-.. code-block:: python
-
-    twitter.get_home_timeline()
-
-Updating Status
-~~~~~~~~~~~~~~~
-
-This method makes use of dynamic arguments, `read more about them <https://twython.readthedocs.org/en/latest/usage/starting_out.html#dynamic-function-arguments>`_
-
-Documentation: https://dev.twitter.com/docs/api/1/post/statuses/update
-
-.. code-block:: python
-
-    twitter.update_status(status='See how easy using Twython is!')
-
-Searching
-~~~~~~~~~
-
-    https://dev.twitter.com/docs/api/1.1/get/search/tweets says it takes "q" and "result_type" amongst other arguments
-
-.. code-block:: python
-
-    twitter.search(q='twitter')
-    twitter.search(q='twitter', result_type='popular')
-
-Advanced Usage
---------------
-
-- `Advanced Twython Usage <https://twython.readthedocs.org/en/latest/usage/advanced_usage.html>`_
-- `Streaming with Twython <https://twython.readthedocs.org/en/latest/usage/streaming_api.html>`_
-
-
-Notes
------
-
-- Twython 3.0.0 has been injected with 1000mgs of pure awesomeness! OAuth 2 application authentication is now supported. And a *whole lot* more! See the `CHANGELOG <https://github.com/ryanmcgrath/twython/blob/master/HISTORY.rst#300-2013-06-18>`_ for more details!
-
-Questions, Comments, etc?
--------------------------
-
-My hope is that Twython is so simple that you'd never *have* to ask any questions, but if you feel the need to contact me for this (or other) reasons, you can hit me up at ryan@venodesigns.net.
-
-Or if I'm to busy to answer, feel free to ping mikeh@ydekproductions.com as well.
-
-Follow us on Twitter:
-
-- `@ryanmcgrath <https://twitter.com/ryanmcgrath>`_
-- `@mikehelmick <https://twitter.com/mikehelmick>`_
-
-Want to help?
--------------
-
-Twython is useful, but ultimately only as useful as the people using it (say that ten times fast!). If you'd like to help, write example code, contribute patches, document things on the wiki, tweet about it. Your help is always appreciated!

docs/_themes/basicstrap/customsidebar.html

@@ -1,18 +1,5 @@
-<h4>{{ _('Donate') }}</h4>
-
-<p>
-Find Twython useful? Consider supporting the author on <a href="https://www.gittip.com/ryanmcgrath/">Gittip</a>:
-</p>
-
-<p>
-  <iframe style="border: 0; margin: 0; padding: 0;"
-      src="https://www.gittip.com/ryanmcgrath/widget.html"
-      width="48pt" height="20pt"></iframe>
-</p>
-
 <h4>{{ _('Links') }}</h4>
 <ul>
-  <li><a href="https://crate.io/packages/twython/">Twython on Crate</a></li>
   <li><a href="https://github.com/ryanmcgrath/twython">Twython on GitHub</a></li>
   <li><a href="https://github.com/ryanmcgrath/twython/issues">Bug Tracker</a></li>
-</ul>
+</ul>

docs/conf.py

@@ -50,9 +50,9 @@ copyright = u'2013, Ryan McGrath'
 # built documents.
 #
 # The short X.Y version.
-version = '3.4.0'
+version = '3.8.0'
 # The full version, including alpha/beta/rc tags.
-release = '3.4.0'
+release = '3.8.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

docs/index.rst

@@ -16,7 +16,7 @@ Features
    - Twitter lists
    - Timelines
    - Direct Messages
-   - and anything found in `the Twitter API docs <https://dev.twitter.com/docs/api/1.1>`_.
+   - and anything found in `the Twitter API docs <https://developer.twitter.com/en/docs>`_.
 - Image Uploading:
    - Update user status with an image
    - Change user avatar

docs/usage/advanced_usage.rst

@@ -29,11 +29,11 @@ This uploads an image as a media object and associates it with a status update.
     photo = open('/path/to/file/image.jpg', 'rb')
     response = twitter.upload_media(media=photo)
     twitter.update_status(status='Checkout this cool image!', media_ids=[response['media_id']])
-    
+
 Documentation:
 
-* https://dev.twitter.com/rest/reference/post/statuses/update
+* https://developer.twitter.com/en/docs/api-reference-index
-* https://dev.twitter.com/rest/reference/post/media/upload
+* https://developer.twitter.com/en/docs/media/upload-media/overview
 
 Updating Status with Video
 --------------------------
@@ -48,8 +48,8 @@ This uploads a video as a media object and associates it with a status update.
 
 Documentation:
 
-* https://dev.twitter.com/rest/reference/post/statuses/update
+* https://developer.twitter.com/en/docs/api-reference-index
-* https://dev.twitter.com/rest/reference/post/media/upload
+* https://developer.twitter.com/en/docs/media/upload-media/overview
 
 Posting a Status with an Editing Image
 --------------------------------------
@@ -59,10 +59,10 @@ with a status update.
 
 .. code-block:: python
 
-    # Assume you are working with a JPEG
+    # Assuming that you are working with a JPEG
 
     from PIL import Image
-    from StringIO import StringIO
+    from io import BytesIO
 
     photo = Image.open('/path/to/file/image.jpg')
 
@@ -71,14 +71,13 @@ with a status update.
     height = int((float(photo.size[1]) * float(wpercent)))
     photo = photo.resize((basewidth, height), Image.ANTIALIAS)
 
-    image_io = StringIO.StringIO()
+    image_io = BytesIO()
     photo.save(image_io, format='JPEG')
 
     # If you do not seek(0), the image will be at the end of the file and
     # unable to be read
     image_io.seek(0)
 
-
     response = twitter.upload_media(media=image_io)
     twitter.update_status(status='Checkout this cool image!', media_ids=[response['media_id']])
 
@@ -98,7 +97,7 @@ That being said, Twython offers a generator for search results and can be access
 
     results = twitter.cursor(twitter.search, q='python')
     for result in results:
-        print result
+        print(result)
 
 Manipulate the Request (headers, proxies, etc.)
 -----------------------------------------------

docs/usage/basic_usage.rst

@@ -28,7 +28,7 @@ Create a Twython instance with your application keys and the users OAuth tokens
 User Information
 ^^^^^^^^^^^^^^^^
 
-Documentation: https://dev.twitter.com/docs/api/1.1/get/account/verify_credentials
+Documentation: https://developer.twitter.com/en/docs/accounts-and-users/manage-account-settings/api-reference/get-account-verify_credentials
 
 .. code-block:: python
 
@@ -37,7 +37,7 @@ Documentation: https://dev.twitter.com/docs/api/1.1/get/account/verify_credentia
 Authenticated Users Home Timeline
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Documentation: https://dev.twitter.com/docs/api/1.1/get/statuses/home_timeline
+Documentation: https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-home_timeline
 
 .. code-block:: python
 
@@ -48,7 +48,7 @@ Updating Status
 
 This method makes use of dynamic arguments, :ref:`read more about them <dynamicargexplaination>`
 
-Documentation: https://dev.twitter.com/docs/api/1.1/post/statuses/update
+Documentation: https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/post-statuses-update
 
 .. code-block:: python
 
@@ -72,7 +72,7 @@ Searching
 
 .. note:: Searching can be done whether you're authenticated via OAuth 1 or OAuth 2
 
-Documentation: https://dev.twitter.com/docs/api/1.1/get/search/tweets
+Documentation: https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets
 
 .. code-block:: python
 
@@ -80,7 +80,7 @@ Documentation: https://dev.twitter.com/docs/api/1.1/get/search/tweets
 
 .. _dynamicargexplaination:
 
-.. important:: To help explain :ref:`dynamic function arguments <dynamicfunctionarguments>` a little more, you can see that the previous call used the keyword argument ``q``, that is because Twitter specifies in their `search documentation <https://dev.twitter.com/docs/api/1.1/get/search/tweets>`_ that the search call accepts the parameter "q". You can pass mutiple keyword arguments. The search documentation also specifies that the call accepts the parameter "result_type"
+.. important:: To help explain :ref:`dynamic function arguments <dynamicfunctionarguments>` a little more, you can see that the previous call used the keyword argument ``q``, that is because Twitter specifies in their `search documentation <https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets>`_ that the search call accepts the parameter "q". You can pass mutiple keyword arguments. The search documentation also specifies that the call accepts the parameter "result_type"
 
 .. code-block:: python
 

docs/usage/special_functions.rst

@@ -25,7 +25,7 @@ The Old Way
     results = twitter.search(q='twitter')
     if results.get('statuses'):
         for result in results['statuses']:
-            print result['id_str']
+            print(result['id_str'])
 
 The New Way
 ^^^^^^^^^^^
@@ -39,15 +39,28 @@ The New Way
 
     results = twitter.cursor(twitter.search, q='twitter')
     for result in results:
-        print result['id_str']
+        print(result['id_str'])
 
 Another example:
 
 .. code-block:: python
 
-    results = twitter.cursor(t.get_mentions_timeline)
+    results = twitter.cursor(twitter.get_mentions_timeline)
     for result in results:
-        print result['id_str']
+        print(result['id_str'])
+
+Items vs Pages
+^^^^^^^^^^^^^^
+
+By default, the cursor yields one item at a time. If instead you prefer to work with entire pages of results, specify ``return_pages=True`` as a keyword argument.
+
+.. code-block:: python
+
+    results = twitter.cursor(twitter.get_mentions_timeline, return_pages=True)
+    # page is a list
+    for page in results:
+        for result in page:
+            print(result['id_str'])
 
 
 HTML for Tweet
@@ -66,17 +79,17 @@ This function takes a tweet object received from the Twitter API and returns an
                                             include_rts=True)
     for tweet in user_tweets:
         tweet['text'] = Twython.html_for_tweet(tweet)
-        print tweet['text']
+        print(tweet['text'])
 
 The above code takes all the tweets from a specific users timeline, loops over them and replaces the value of ``tweet['text']`` with HTML.
 
 So:
 
-    http://t.co/FCmXyI6VHd is a #cool site, lol! @mikehelmick shd #checkitout. Love, @__twython__ https://t.co/67pwRvY6z9 http://t.co/N6InAO4B71
+    http://t.co/FCmXyI6VHd is #cool, lol! @mikehelmick shd #checkitout. Love, @__twython__ $IBM https://t.co/67pwRvY6z9 http://t.co/N6InAO4B71
 
 will be replaced with:
 
-    <a href="http://t.co/FCmXyI6VHd" class="twython-url">google.com</a> is a <a href="https://twitter.com/search?q=%23cool" class="twython-hashtag">#cool</a> site, lol! <a href="https://twitter.com/mikehelmick" class="twython-mention">@mikehelmick</a> shd <a href="https://twitter.com/search?q=%23checkitout" class="twython-hashtag">#checkitout</a>. Love, <a href="https://twitter.com/__twython__" class="twython-mention">@__twython__</a> <a href="https://t.co/67pwRvY6z9" class="twython-url">github.com</a> <a href="http://t.co/N6InAO4B71" class="twython-media">pic.twitter.com/N6InAO4B71</a>
+    <a href="http://t.co/FCmXyI6VHd" class="twython-url">google.com</a> is <a href="https://twitter.com/search?q=%23cool" class="twython-hashtag">#cool</a>, lol! <a href="https://twitter.com/mikehelmick" class="twython-mention">@mikehelmick</a> shd <a href="https://twitter.com/search?q=%23checkitout" class="twython-hashtag">#checkitout</a>. Love, <a href="https://twitter.com/__twython__" class="twython-mention">@__twython__</a> <a href="https://twitter.com/?q=%24IBM" class="twython-symbol">$IBM</a> <a href="https://t.co/67pwRvY6z9" class="twython-url">github.com</a> <a href="http://t.co/N6InAO4B71" class="twython-media">pic.twitter.com/N6InAO4B71</a>
 
 .. note:: When converting the string to HTML we add a class to each HTML tag so that you can maninpulate the DOM later on.
 
@@ -84,6 +97,7 @@ will be replaced with:
 - For media urls that are replaced we add ``class="twython-media"`` to the anchor tag
 - For user mentions that are replaced we add ``class="twython-mention"`` to the anchor tag
 - For hashtags that are replaced we add ``class="twython-hashtag"`` to the anchor tag
+- For symbols that are replaced we add ``class="twython-symbol"`` to the anchor tag
 
 This function accepts two parameters: ``use_display_url`` and ``use_expanded_url``
 By default, ``use_display_url`` is ``True``. Meaning the link displayed in the tweet text will appear as (ex. google.com, github.com)

docs/usage/starting_out.rst

@@ -10,7 +10,7 @@ This section is going to help you understand creating a Twitter Application, aut
 Beginning
 ---------
 
-First, you'll want to head over to https://dev.twitter.com/apps and register an application!
+First, you'll want to head over to https://apps.twitter.com/ and register an application!
 
 After you register, grab your applications ``Consumer Key`` and ``Consumer Secret`` from the application details tab.
 

docs/usage/streaming_api.rst

@@ -5,7 +5,7 @@ Streaming API
 
 This section will cover how to use Twython and interact with the Twitter Streaming API.
 
-Streaming Documentation: https://dev.twitter.com/docs/streaming-apis
+Streaming Documentation: https://developer.twitter.com/en/docs/tweets/filter-realtime/guides/streaming-message-types
 
 .. important:: The Streaming API requires that you have OAuth 1 authentication credentials. If you don't have credentials, head over to the :ref:`authentication section <oauth1>` and find out how!
 
@@ -27,10 +27,10 @@ Now set up how you want to handle the signals.
     class MyStreamer(TwythonStreamer):
         def on_success(self, data):
             if 'text' in data:
-                print data['text'].encode('utf-8')
+                print(data['text'])
 
         def on_error(self, status_code, data):
-            print status_code
+            print(status_code)
 
             # Want to stop trying to get data because of the error?
             # Uncomment the next line!

examples/get_direct_messages.py

@@ -0,0 +1,5 @@
+from twython import Twython, TwythonError
+twitter = Twython(APP_KEY, APP_SECRET, OAUTH_TOKEN, OAUTH_TOKEN_SECRET)
+
+get_list = twitter.get_direct_messages()
+print(get_list)

examples/get_user_timeline.py

@@ -7,4 +7,4 @@ try:
 except TwythonError as e:
     print e
 
-print user_timeline
+print(user_timeline)

setup.py

@@ -8,8 +8,8 @@ try:
 except ImportError:
     from distutils.core import setup
 
-__author__ = 'Ryan McGrath <ryan@venodesigns.net>'
+__author__ = 'Ryan McGrath <ryan@rymc.io>'
-__version__ = '3.4.0'
+__version__ = '3.9.1'
 
 packages = [
     'twython',
@@ -24,15 +24,15 @@ setup(
     name='twython',
     version=__version__,
     install_requires=['requests>=2.1.0', 'requests_oauthlib>=0.4.0'],
+    python_requires='>=3.5',
     author='Ryan McGrath',
-    author_email='ryan@venodesigns.net',
+    author_email='ryan@rymc.io',
-    license=open('LICENSE').read(),
+    license='MIT',
     url='https://github.com/ryanmcgrath/twython/tree/master',
     keywords='twitter search api tweet twython stream',
-    description='Actively maintained, pure Python wrapper for the \
+    description='Actively maintained, pure Python wrapper for the Twitter API. Supports both normal and streaming Twitter APIs',
-    Twitter API. Supports both normal and streaming Twitter APIs',
+    long_description=open('README.md', encoding='utf-8').read() + '\n\n' +open('HISTORY.md', encoding='utf-8').read(),
-    long_description=open('README.rst').read() + '\n\n' +
+    long_description_content_type='text/markdown',
-        open('HISTORY.rst').read(),
     include_package_data=True,
     packages=packages,
     classifiers=[
@@ -41,6 +41,11 @@ setup(
         'License :: OSI Approved :: MIT License',
         'Topic :: Software Development :: Libraries :: Python Modules',
         'Topic :: Communications :: Chat',
-        'Topic :: Internet'
+        'Topic :: Internet',
+        'Programming Language :: Python :: 3',
+        'Programming Language :: Python :: 3 :: Only',
+        'Programming Language :: Python :: 3.5',
+        'Programming Language :: Python :: 3.6',
+        'Programming Language :: Python :: 3.7',
     ]
 )

tests/config.py

@@ -1,10 +1,8 @@
+# -*- coding: utf-8 -*-
 import os
 
 import sys
-if sys.version_info[0] == 2 and sys.version_info[1] == 6:
+import unittest
-    import unittest2 as unittest
-else:
-    import unittest
 
 app_key = os.environ.get('APP_KEY')
 app_secret = os.environ.get('APP_SECRET')
@@ -26,5 +24,3 @@ test_list_slug = os.environ.get('TEST_LIST_SLUG', 'team')
 test_list_owner_screen_name = os.environ.get('TEST_LIST_OWNER_SCREEN_NAME',
                                              'twitterapi')
 
-test_tweet_object = {u'contributors': None, u'truncated': False, u'text': u'http://t.co/FCmXyI6VHd is a #cool site, lol! @mikehelmick shd #checkitout. Love, @__twython__ https://t.co/67pwRvY6z9 http://t.co/N6InAO4B71', u'in_reply_to_status_id': None, u'id': 349683012054683648, u'favorite_count': 0, u'source': u'web', u'retweeted': False, u'coordinates': None, u'entities': {u'symbols': [], u'user_mentions': [{u'id': 29251354, u'indices': [45, 57], u'id_str': u'29251354', u'screen_name': u'mikehelmick', u'name': u'Mike Helmick'}, {u'id': 1431865928, u'indices': [81, 93], u'id_str': u'1431865928', u'screen_name': u'__twython__', u'name': u'Twython'}], u'hashtags': [{u'indices': [28, 33], u'text': u'cool'}, {u'indices': [62, 73], u'text': u'checkitout'}], u'urls': [{u'url': u'http://t.co/FCmXyI6VHd', u'indices': [0, 22], u'expanded_url': u'http://google.com', u'display_url': u'google.com'}, {u'url': u'https://t.co/67pwRvY6z9', u'indices': [94, 117], u'expanded_url': u'https://github.com', u'display_url': u'github.com'}], u'media': [{u'id': 537884378513162240, u'id_str': u'537884378513162240', u'indices': [118, 140], u'media_url': u'http://pbs.twimg.com/media/B3by_g-CQAAhrO5.jpg', u'media_url_https': u'https://pbs.twimg.com/media/B3by_g-CQAAhrO5.jpg', u'url': u'http://t.co/N6InAO4B71', u'display_url': u'pic.twitter.com/N6InAO4B71', u'expanded_url': u'http://twitter.com/pingofglitch/status/537884380060844032/photo/1', u'type': u'photo', u'sizes': {u'large': {u'w': 1024, u'h': 640, u'resize': u'fit'}, u'thumb': {u'w': 150, u'h': 150, u'resize': u'crop'}, u'medium': {u'w': 600, u'h': 375, u'resize': u'fit'}, u'small': {u'w': 340, u'h': 212, u'resize': u'fit'}}}]}, u'in_reply_to_screen_name': None, u'id_str': u'349683012054683648', u'retweet_count': 0, u'in_reply_to_user_id': None, u'favorited': False, u'user': {u'follow_request_sent': False, u'profile_use_background_image': True, u'default_profile_image': True, u'id': 1431865928, u'verified': False, u'profile_text_color': u'333333', u'profile_image_url_https': u'https://si0.twimg.com/sticky/default_profile_images/default_profile_3_normal.png', u'profile_sidebar_fill_color': u'DDEEF6', u'entities': {u'description': {u'urls': []}}, u'followers_count': 1, u'profile_sidebar_border_color': u'C0DEED', u'id_str': u'1431865928', u'profile_background_color': u'3D3D3D', u'listed_count': 0, u'profile_background_image_url_https': u'https://si0.twimg.com/images/themes/theme1/bg.png', u'utc_offset': None, u'statuses_count': 2, u'description': u'', u'friends_count': 1, u'location': u'', u'profile_link_color': u'0084B4', u'profile_image_url': u'http://a0.twimg.com/sticky/default_profile_images/default_profile_3_normal.png', u'following': False, u'geo_enabled': False, u'profile_background_image_url': u'http://a0.twimg.com/images/themes/theme1/bg.png', u'screen_name': u'__twython__', u'lang': u'en', u'profile_background_tile': False, u'favourites_count': 0, u'name': u'Twython', u'notifications': False, u'url': None, u'created_at': u'Thu May 16 01:11:09 +0000 2013', u'contributors_enabled': False, u'time_zone': None, u'protected': False, u'default_profile': False, u'is_translator': False}, u'geo': None, u'in_reply_to_user_id_str': None, u'possibly_sensitive': False, u'lang': u'en', u'created_at': u'Wed Jun 26 00:18:21 +0000 2013', u'in_reply_to_status_id_str': None, u'place': None}
-test_tweet_html = '<a href="http://t.co/FCmXyI6VHd" class="twython-url">google.com</a> is a <a href="https://twitter.com/search?q=%23cool" class="twython-hashtag">#cool</a> site, lol! <a href="https://twitter.com/mikehelmick" class="twython-mention">@mikehelmick</a> shd <a href="https://twitter.com/search?q=%23checkitout" class="twython-hashtag">#checkitout</a>. Love, <a href="https://twitter.com/__twython__" class="twython-mention">@__twython__</a> <a href="https://t.co/67pwRvY6z9" class="twython-url">github.com</a> <a href="http://t.co/N6InAO4B71" class="twython-media">pic.twitter.com/N6InAO4B71</a>'

tests/test_core.py

@@ -1,8 +1,7 @@
+# -*- coding: utf-8 -*-
 from twython import Twython, TwythonError, TwythonAuthError, TwythonRateLimitError
 
-from .config import (
+from .config import unittest
-    test_tweet_object, test_tweet_html, unittest
-)
 
 import responses
 import requests
@@ -299,21 +298,15 @@ class TwythonAPITestCase(unittest.TestCase):
         """Test encoding UTF-8 works"""
         self.api.encode('Twython is awesome!')
 
-    def test_html_for_tweet(self):
+    def test_cursor_requires_twython_function(self):
-        """Test HTML for Tweet returns what we want"""
+        """Test that cursor() raises when called without a Twython function"""
-        tweet_text = self.api.html_for_tweet(test_tweet_object)
+        def init_and_iterate_cursor(*args, **kwargs):
-        self.assertEqual(test_tweet_html, tweet_text)
+            cursor = self.api.cursor(*args, **kwargs)
+            return next(cursor)
 
-    def test_html_for_tweet_expanded_url(self):
+        non_function = object()
-        """Test using expanded url in HTML for Tweet displays full urls"""
+        non_twython_function = lambda x: x
-        tweet_text = self.api.html_for_tweet(test_tweet_object,
+
-                                             use_expanded_url=True)
+        self.assertRaises(TypeError, init_and_iterate_cursor, non_function)
-        # Make sure full url is in HTML
+        self.assertRaises(TwythonError, init_and_iterate_cursor, non_twython_function)
-        self.assertTrue('http://google.com' in tweet_text)
 
-    def test_html_for_tweet_short_url(self):
-        """Test using expanded url in HTML for Tweet displays full urls"""
-        tweet_text = self.api.html_for_tweet(test_tweet_object, False)
-        # Make sure HTML doesn't contain the display OR expanded url
-        self.assertTrue('http://google.com' not in tweet_text)
-        self.assertTrue('google.com' not in tweet_text)

tests/test_endpoints.py

@@ -4,7 +4,7 @@ from .config import (
     app_key, app_secret, oauth_token, oauth_token_secret,
     protected_twitter_1, protected_twitter_2, screen_name,
     test_tweet_id, test_list_slug, test_list_owner_screen_name,
-    access_token, test_tweet_object, test_tweet_html, unittest
+    access_token, unittest
 )
 
 import time

tests/test_html_for_tweet.py

@@ -0,0 +1,130 @@
+# -*- coding: utf-8 -*-
+import json
+import os
+
+from twython import Twython, TwythonError
+
+from .config import unittest
+
+
+class TestHtmlForTweetTestCase(unittest.TestCase):
+    def setUp(self):
+        self.api = Twython('', '', '', '')
+
+    def load_tweet(self, name):
+        f = open(os.path.join(
+                os.path.dirname(__file__),
+                'tweets',
+                '%s.json' % name
+            ))
+        tweet = json.load(f)
+        f.close()
+        return tweet
+
+    def test_basic(self):
+        """Test HTML for Tweet returns what we want"""
+        tweet_object = self.load_tweet('basic')
+        tweet_text = self.api.html_for_tweet(tweet_object)
+        self.assertEqual(tweet_text,
+            '<a href="http://t.co/FCmXyI6VHd" class="twython-url">google.com</a> is a <a href="https://twitter.com/search?q=%23cool" class="twython-hashtag">#cool</a> site, lol! <a href="https://twitter.com/mikehelmick" class="twython-mention">@mikehelmick</a> shd <a href="https://twitter.com/search?q=%23checkitout" class="twython-hashtag">#checkitout</a>. Love, <a href="https://twitter.com/__twython__" class="twython-mention">@__twython__</a> <a href="https://t.co/67pwRvY6z9" class="twython-url">github.com</a> <a href="http://t.co/N6InAO4B71" class="twython-media">pic.twitter.com/N6InAO4B71</a>')
+
+    def test_reply(self):
+        """Test HTML for Tweet links the replied-to username."""
+        tweet_object = self.load_tweet('reply')
+        tweet_text = self.api.html_for_tweet(tweet_object)
+        self.assertEqual(tweet_text,
+                u'<span class="twython-tweet-prefix"><a href="https://twitter.com/philgyford" class="twython-mention">@philgyford</a> </span>Here’s a test tweet that goes on as much as possible and includes an image. Hi to my fans in testland!<span class="twython-tweet-suffix"> https://t.co/tzhyk2QWSr</span>')
+
+    def test_expanded_url(self):
+        """Test using expanded url in HTML for Tweet displays full urls"""
+        tweet_object = self.load_tweet('basic')
+        tweet_text = self.api.html_for_tweet(tweet_object,
+                                             use_expanded_url=True)
+        # Make sure full url is in HTML
+        self.assertTrue('http://google.com' in tweet_text)
+
+    def test_short_url(self):
+        """Test using expanded url in HTML for Tweet displays full urls"""
+        tweet_object = self.load_tweet('basic')
+        tweet_text = self.api.html_for_tweet(tweet_object, False)
+        # Make sure HTML doesn't contain the display OR expanded url
+        self.assertTrue('http://google.com' not in tweet_text)
+        self.assertTrue('google.com' not in tweet_text)
+
+    def test_identical_urls(self):
+        """If the 'url's for different url entities are identical, they should link correctly."""
+        tweet_object = self.load_tweet('identical_urls')
+        tweet_text = self.api.html_for_tweet(tweet_object)
+        self.assertEqual(tweet_text,
+                u'Use Cases, Trials and Making 5G a Reality <a href="https://t.co/W0uArTMk9N" class="twython-url">buff.ly/2sEhrgO</a> #5G #innovation via @5GWorldSeries <a href="https://t.co/W0uArTMk9N" class="twython-url">buff.ly/2sEhrgO</a>')
+
+    def test_symbols(self):
+        tweet_object = self.load_tweet('symbols')
+        tweet_text = self.api.html_for_tweet(tweet_object)
+        # Should only link symbols listed in entities:
+        self.assertTrue('<a href="https://twitter.com/search?q=%24AAPL" class="twython-symbol">$AAPL</a>' in tweet_text)
+        self.assertTrue('<a href="https://twitter.com/search?q=%24ANOTHER" class="twython-symbol">$ANOTHER</a>' not in tweet_text)
+
+    def test_no_symbols(self):
+        """Should still work if tweet object has no symbols list"""
+        tweet = self.load_tweet('symbols')
+        # Save a copy:
+        symbols = tweet['entities']['symbols']
+        del tweet['entities']['symbols']
+        tweet_text = self.api.html_for_tweet(tweet)
+        self.assertTrue('symbols: $AAPL and' in tweet_text)
+        self.assertTrue('and $ANOTHER and $A.' in tweet_text)
+
+    def test_compatmode(self):
+        tweet_object = self.load_tweet('compat')
+        tweet_text = self.api.html_for_tweet(tweet_object)
+        # link to compat web status link
+        self.assertTrue(
+            u'<a href="https://t.co/SRmsuks2ru" class="twython-url">twitter.com/i/web/status/7…</a>' in tweet_text)
+
+    def test_extendedmode(self):
+        tweet_object = self.load_tweet('extended')
+        tweet_text = self.api.html_for_tweet(tweet_object)
+        # full tweet rendered with suffix
+        self.assertEqual(tweet_text,
+            'Say more about what\'s happening! Rolling out now: photos, videos, GIFs, polls, and Quote Tweets no longer count toward your 140 characters.<span class="twython-tweet-suffix"> <a href="https://t.co/I9pUC0NdZC" class="twython-media">pic.twitter.com/I9pUC0NdZC</a></span>')
+
+    def test_entities_with_prefix(self):
+        """
+        If there is a username mention at the start of a tweet it's in the
+        "prefix" and so isn't part of the main tweet display text.
+        But its length is still counted in the indices of any subsequent
+        mentions, urls, hashtags, etc.
+        """
+        self.maxDiff = 2200
+        tweet_object = self.load_tweet('entities_with_prefix')
+        tweet_text = self.api.html_for_tweet(tweet_object)
+        self.assertEqual(tweet_text,
+            u'<span class="twython-tweet-prefix"><a href="https://twitter.com/philgyford" class="twython-mention">@philgyford</a> </span>This is a test for <a href="https://twitter.com/visionphil" class="twython-mention">@visionphil</a> that includes a link <a href="https://t.co/sKw4J3A8SZ" class="twython-url">example.org</a> and <a href="https://twitter.com/search?q=%23hashtag" class="twython-hashtag">#hashtag</a> and X for good measure AND that is longer than 140 characters. <a href="https://t.co/jnQdy7Zg7u" class="twython-url">example.com</a>')
+
+    def test_media(self):
+        tweet_object = self.load_tweet('media')
+        tweet_text = self.api.html_for_tweet(tweet_object)
+
+        self.assertEqual(
+            u"""I made some D3.js charts showing the years covered by books in a series compared to their publishing dates <a href="https://t.co/2yUmmn3TOc" class="twython-url">gyford.com/phil/writing/2\u2026</a><span class="twython-tweet-suffix"> <a href="https://t.co/OwNc6uJklg" class="twython-media">pic.twitter.com/OwNc6uJklg</a></span>""",
+            tweet_text)
+
+    def test_quoted(self):
+        "With expand_quoted_status=True it should include a quoted tweet."
+        tweet_object = self.load_tweet('quoted')
+        tweet_text = self.api.html_for_tweet(tweet_object,
+                                             expand_quoted_status=True)
+        self.assertEqual(
+                u"""Here\u2019s a quoted tweet. <a href="https://t.co/3neKzof0gT" class="twython-url">twitter.com/philgyford/sta\u2026</a><blockquote class="twython-quote">The quoted tweet text.<cite><a href="https://twitter.com/philgyford/status/917699069916729344"><span class="twython-quote-user-name">Phil Gyford</span><span class="twython-quote-user-screenname">@philgyford</span></a></cite></blockquote>""",
+                tweet_text)
+
+    def test_retweet(self):
+        "With expand_quoted_status=True it should include a quoted tweet."
+        tweet_object = self.load_tweet('retweet')
+        tweet_text = self.api.html_for_tweet(tweet_object)
+
+        self.assertEqual(
+                u"""My aunt and uncle in a very ill humour one with another, but I made shift with much ado to keep them from scolding.""",
+                tweet_text)
+

tests/tweets/basic.json

@@ -0,0 +1,170 @@
+{
+  "contributors":null,
+  "truncated":false,
+  "text":"http://t.co/FCmXyI6VHd is a #cool site, lol! @mikehelmick shd #checkitout. Love, @__twython__ https://t.co/67pwRvY6z9 http://t.co/N6InAO4B71",
+  "in_reply_to_status_id":null,
+  "id":349683012054683648,
+  "favorite_count":0,
+  "source":"<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>",
+  "retweeted":false,
+  "coordinates":null,
+  "entities":{
+    "symbols":[
+
+    ],
+    "user_mentions":[
+      {
+        "id":29251354,
+        "indices":[
+          45,
+          57
+        ],
+        "id_str":"29251354",
+        "screen_name":"mikehelmick",
+        "name":"Mike Helmick"
+      },
+      {
+        "id":1431865928,
+        "indices":[
+          81,
+          93
+        ],
+        "id_str":"1431865928",
+        "screen_name":"__twython__",
+        "name":"Twython"
+      }
+    ],
+    "hashtags":[
+      {
+        "indices":[
+          28,
+          33
+        ],
+        "text":"cool"
+      },
+      {
+        "indices":[
+          62,
+          73
+        ],
+        "text":"checkitout"
+      }
+    ],
+    "urls":[
+      {
+        "url":"http://t.co/FCmXyI6VHd",
+        "indices":[
+          0,
+          22
+        ],
+        "expanded_url":"http://google.com",
+        "display_url":"google.com"
+      },
+      {
+        "url":"https://t.co/67pwRvY6z9",
+        "indices":[
+          94,
+          117
+        ],
+        "expanded_url":"https://github.com",
+        "display_url":"github.com"
+      }
+    ],
+    "media":[
+      {
+        "id":537884378513162240,
+        "id_str":"537884378513162240",
+        "indices":[
+          118,
+          140
+        ],
+        "media_url":"http://pbs.twimg.com/media/B3by_g-CQAAhrO5.jpg",
+        "media_url_https":"https://pbs.twimg.com/media/B3by_g-CQAAhrO5.jpg",
+        "url":"http://t.co/N6InAO4B71",
+        "display_url":"pic.twitter.com/N6InAO4B71",
+        "expanded_url":"http://twitter.com/pingofglitch/status/537884380060844032/photo/1",
+        "type":"photo",
+        "sizes":{
+          "large":{
+            "w":1024,
+            "h":640,
+            "resize":"fit"
+          },
+          "thumb":{
+            "w":150,
+            "h":150,
+            "resize":"crop"
+          },
+          "medium":{
+            "w":600,
+            "h":375,
+            "resize":"fit"
+          },
+          "small":{
+            "w":340,
+            "h":212,
+            "resize":"fit"
+          }
+        }
+      }
+    ]
+  },
+  "in_reply_to_screen_name":null,
+  "id_str":"349683012054683648",
+  "retweet_count":0,
+  "in_reply_to_user_id":null,
+  "favorited":false,
+  "user":{
+    "follow_request_sent":false,
+    "profile_use_background_image":true,
+    "default_profile_image":true,
+    "id":1431865928,
+    "verified":false,
+    "profile_text_color":"333333",
+    "profile_image_url_https":"https://si0.twimg.com/sticky/default_profile_images/default_profile_3_normal.png",
+    "profile_sidebar_fill_color":"DDEEF6",
+    "entities":{
+      "description":{
+        "urls":[
+
+        ]
+      }
+    },
+    "followers_count":1,
+    "profile_sidebar_border_color":"C0DEED",
+    "id_str":"1431865928",
+    "profile_background_color":"3D3D3D",
+    "listed_count":0,
+    "profile_background_image_url_https":"https://si0.twimg.com/images/themes/theme1/bg.png",
+    "utc_offset":null,
+    "statuses_count":2,
+    "description":"",
+    "friends_count":1,
+    "location":"",
+    "profile_link_color":"0084B4",
+    "profile_image_url":"http://a0.twimg.com/sticky/default_profile_images/default_profile_3_normal.png",
+    "following":false,
+    "geo_enabled":false,
+    "profile_background_image_url":"http://a0.twimg.com/images/themes/theme1/bg.png",
+    "screen_name":"__twython__",
+    "lang":"en",
+    "profile_background_tile":false,
+    "favourites_count":0,
+    "name":"Twython",
+    "notifications":false,
+    "url":null,
+    "created_at":"Thu May 16 01:11:09 +0000 2013",
+    "contributors_enabled":false,
+    "time_zone":null,
+    "protected":false,
+    "default_profile":false,
+    "is_translator":false
+  },
+  "geo":null,
+  "in_reply_to_user_id_str":null,
+  "possibly_sensitive":false,
+  "lang":"en",
+  "created_at":"Wed Jun 26 00:18:21 +0000 2013",
+  "in_reply_to_status_id_str":null,
+  "place":null
+}

tests/tweets/compat.json

@@ -0,0 +1,51 @@
+{
+  "contributors":null,
+  "truncated":true,
+  "text":"Say more about what's happening! Rolling out now: photos, videos, GIFs, polls, and Quote Tweets no longer count tow\u2026 https://t.co/SRmsuks2ru",
+  "is_quote_status":false,
+  "in_reply_to_status_id":null,
+  "id":777915304261193728,
+  "favorite_count":13856,
+  "source":"<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>",
+  "retweeted":false,
+  "coordinates":null,
+  "entities":{
+    "symbols":[
+
+    ],
+    "user_mentions":[
+
+    ],
+    "hashtags":[
+
+    ],
+    "urls":[
+      {
+        "url":"https://t.co/SRmsuks2ru",
+        "indices":[
+          117,
+          140
+        ],
+        "expanded_url":"https://twitter.com/i/web/status/777915304261193728",
+        "display_url":"twitter.com/i/web/status/7\u2026"
+      }
+    ]
+  },
+  "in_reply_to_screen_name":null,
+  "id_str":"777915304261193728",
+  "retweet_count":14767,
+  "in_reply_to_user_id":null,
+  "favorited":false,
+  "user":{
+    "name":"Twitter",
+    "screen_name":"twitter"
+  },
+  "geo":null,
+  "in_reply_to_user_id_str":null,
+  "possibly_sensitive":false,
+  "possibly_sensitive_appealable":false,
+  "lang":"en",
+  "created_at":"Mon Sep 19 17:00:36 +0000 2016",
+  "in_reply_to_status_id_str":null,
+  "place":null
+}

tests/tweets/entities_with_prefix.json

@@ -0,0 +1,68 @@
+{  
+  "created_at":"Sat Jan 06 18:56:35 +0000 2018",
+  "id":949716340755091458,
+  "id_str":"949716340755091458",
+  "full_text":"@philgyford This is a test for @visionphil that includes a link https://t.co/sKw4J3A8SZ and #hashtag and X for good measure AND that is longer than 140 characters. https://t.co/jnQdy7Zg7u",
+  "truncated":false,
+  "display_text_range":[ 12, 187 ],
+  "entities":{  
+    "hashtags":[  
+      {  
+        "text":"hashtag",
+        "indices":[ 92, 100 ]
+      }
+    ],
+    "symbols":[  ],
+    "user_mentions":[  
+      {  
+        "screen_name":"philgyford",
+        "name":"Phil Gyford",
+        "id":12552,
+        "id_str":"12552",
+        "indices":[ 0, 11 ]
+      },
+      {  
+        "screen_name":"visionphil",
+        "name":"Vision Phil",
+        "id":104456050,
+        "id_str":"104456050",
+        "indices":[ 31, 42 ]
+      }
+    ],
+    "urls":[  
+      {  
+        "url":"https://t.co/sKw4J3A8SZ",
+        "expanded_url":"http://example.org",
+        "display_url":"example.org",
+        "indices":[ 64, 87 ]
+      },
+      {  
+        "url":"https://t.co/jnQdy7Zg7u",
+        "expanded_url":"http://example.com",
+        "display_url":"example.com",
+        "indices":[ 164, 187 ]
+      }
+    ]
+  },
+  "source":"<a href=\"http://tapbots.com/software/tweetbot/mac\" rel=\"nofollow\">Tweetbot for Mac</a>",
+  "in_reply_to_status_id":948561036889722880,
+  "in_reply_to_status_id_str":"948561036889722880",
+  "in_reply_to_user_id":12552,
+  "in_reply_to_user_id_str":"12552",
+  "in_reply_to_screen_name":"philgyford",
+  "user":{  
+    "id":2030131,
+    "id_str":"2030131"
+  },
+  "geo":null,
+  "coordinates":null,
+  "place":null,
+  "contributors":null,
+  "is_quote_status":false,
+  "retweet_count":0,
+  "favorite_count":0,
+  "favorited":false,
+  "retweeted":false,
+  "possibly_sensitive":false,
+  "lang":"en"
+}

tests/tweets/extended.json

@@ -0,0 +1,204 @@
+{
+  "full_text":"Say more about what's happening! Rolling out now: photos, videos, GIFs, polls, and Quote Tweets no longer count toward your 140 characters. https://t.co/I9pUC0NdZC",
+  "truncated":false,
+  "is_quote_status":false,
+  "in_reply_to_status_id":null,
+  "id":777915304261193728,
+  "favorite_count":13856,
+  "contributors":null,
+  "source":"<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>",
+  "retweeted":false,
+  "coordinates":null,
+  "entities":{
+    "symbols":[
+
+    ],
+    "user_mentions":[
+
+    ],
+    "hashtags":[
+
+    ],
+    "urls":[
+
+    ],
+    "media":[
+      {
+        "expanded_url":"https://twitter.com/twitter/status/777915304261193728/photo/1",
+        "sizes":{
+          "small":{
+            "h":340,
+            "w":340,
+            "resize":"fit"
+          },
+          "large":{
+            "h":700,
+            "w":700,
+            "resize":"fit"
+          },
+          "medium":{
+            "h":600,
+            "w":600,
+            "resize":"fit"
+          },
+          "thumb":{
+            "h":150,
+            "w":150,
+            "resize":"crop"
+          }
+        },
+        "url":"https://t.co/I9pUC0NdZC",
+        "media_url_https":"https://pbs.twimg.com/tweet_video_thumb/Csu1TzEVMAAAEv7.jpg",
+        "id_str":"777914712382058496",
+        "indices":[
+          140,
+          163
+        ],
+        "media_url":"http://pbs.twimg.com/tweet_video_thumb/Csu1TzEVMAAAEv7.jpg",
+        "type":"photo",
+        "id":777914712382058496,
+        "display_url":"pic.twitter.com/I9pUC0NdZC"
+      }
+    ]
+  },
+  "in_reply_to_screen_name":null,
+  "id_str":"777915304261193728",
+  "display_text_range":[
+    0,
+    139
+  ],
+  "retweet_count":14767,
+  "in_reply_to_user_id":null,
+  "favorited":false,
+  "user":{
+    "follow_request_sent":false,
+    "has_extended_profile":false,
+    "profile_use_background_image":true,
+    "id":783214,
+    "verified":true,
+    "profile_text_color":"333333",
+    "profile_image_url_https":"https://pbs.twimg.com/profile_images/767879603977191425/29zfZY6I_normal.jpg",
+    "profile_sidebar_fill_color":"F6F6F6",
+    "is_translator":false,
+    "geo_enabled":true,
+    "entities":{
+      "url":{
+        "urls":[
+          {
+            "url":"http://t.co/5iRhy7wTgu",
+            "indices":[
+              0,
+              22
+            ],
+            "expanded_url":"http://blog.twitter.com/",
+            "display_url":"blog.twitter.com"
+          }
+        ]
+      },
+      "description":{
+        "urls":[
+          {
+            "url":"https://t.co/qq1HEzvnrA",
+            "indices":[
+              84,
+              107
+            ],
+            "expanded_url":"http://support.twitter.com",
+            "display_url":"support.twitter.com"
+          }
+        ]
+      }
+    },
+    "followers_count":56827498,
+    "protected":false,
+    "location":"San Francisco, CA",
+    "default_profile_image":false,
+    "id_str":"783214",
+    "lang":"en",
+    "utc_offset":-25200,
+    "statuses_count":3161,
+    "description":"Your official source for news, updates and tips from Twitter, Inc. Need help? Visit https://t.co/qq1HEzvnrA.",
+    "friends_count":145,
+    "profile_link_color":"226699",
+    "profile_image_url":"http://pbs.twimg.com/profile_images/767879603977191425/29zfZY6I_normal.jpg",
+    "notifications":false,
+    "profile_background_image_url_https":"https://pbs.twimg.com/profile_background_images/657090062/l1uqey5sy82r9ijhke1i.png",
+    "profile_background_color":"ACDED6",
+    "profile_banner_url":"https://pbs.twimg.com/profile_banners/783214/1471929200",
+    "profile_background_image_url":"http://pbs.twimg.com/profile_background_images/657090062/l1uqey5sy82r9ijhke1i.png",
+    "name":"Twitter",
+    "is_translation_enabled":false,
+    "profile_background_tile":true,
+    "favourites_count":2332,
+    "screen_name":"twitter",
+    "url":"http://t.co/5iRhy7wTgu",
+    "created_at":"Tue Feb 20 14:35:54 +0000 2007",
+    "contributors_enabled":false,
+    "time_zone":"Pacific Time (US & Canada)",
+    "profile_sidebar_border_color":"FFFFFF",
+    "default_profile":false,
+    "following":false,
+    "listed_count":90445
+  },
+  "geo":null,
+  "in_reply_to_user_id_str":null,
+  "possibly_sensitive":false,
+  "possibly_sensitive_appealable":false,
+  "lang":"en",
+  "created_at":"Mon Sep 19 17:00:36 +0000 2016",
+  "in_reply_to_status_id_str":null,
+  "place":null,
+  "extended_entities":{
+    "media":[
+      {
+        "expanded_url":"https://twitter.com/twitter/status/777915304261193728/photo/1",
+        "display_url":"pic.twitter.com/I9pUC0NdZC",
+        "url":"https://t.co/I9pUC0NdZC",
+        "media_url_https":"https://pbs.twimg.com/tweet_video_thumb/Csu1TzEVMAAAEv7.jpg",
+        "video_info":{
+          "aspect_ratio":[
+            1,
+            1
+          ],
+          "variants":[
+            {
+              "url":"https://pbs.twimg.com/tweet_video/Csu1TzEVMAAAEv7.mp4",
+              "bitrate":0,
+              "content_type":"video/mp4"
+            }
+          ]
+        },
+        "id_str":"777914712382058496",
+        "sizes":{
+          "small":{
+            "h":340,
+            "w":340,
+            "resize":"fit"
+          },
+          "large":{
+            "h":700,
+            "w":700,
+            "resize":"fit"
+          },
+          "medium":{
+            "h":600,
+            "w":600,
+            "resize":"fit"
+          },
+          "thumb":{
+            "h":150,
+            "w":150,
+            "resize":"crop"
+          }
+        },
+        "indices":[
+          140,
+          163
+        ],
+        "type":"animated_gif",
+        "id":777914712382058496,
+        "media_url":"http://pbs.twimg.com/tweet_video_thumb/Csu1TzEVMAAAEv7.jpg"
+      }
+    ]
+  }
+}

tests/tweets/identical_urls.json

@@ -0,0 +1,34 @@
+{
+  "entities":{
+    "hashtags":[
+
+    ],
+    "user_mentions":[
+
+    ],
+    "symbols":[
+
+    ],
+    "urls":[
+      {
+        "display_url":"buff.ly/2sEhrgO",
+        "expanded_url":"http://buff.ly/2sEhrgO",
+        "indices":[
+          42,
+          65
+        ],
+        "url":"https://t.co/W0uArTMk9N"
+      },
+      {
+        "display_url":"buff.ly/2sEhrgO",
+        "expanded_url":"http://buff.ly/2sEhrgO",
+        "indices":[
+          101,
+          124
+        ],
+        "url":"https://t.co/W0uArTMk9N"
+      }
+    ]
+  },
+  "full_text":"Use Cases, Trials and Making 5G a Reality https://t.co/W0uArTMk9N #5G #innovation via @5GWorldSeries https://t.co/W0uArTMk9N"
+}

tests/tweets/media.json

@@ -0,0 +1,205 @@
+{
+  "contributors":null,
+  "truncated":false,
+  "is_quote_status":false,
+  "in_reply_to_status_id":null,
+  "id":905105588279013377,
+  "favorite_count":10,
+  "full_text":"I made some D3.js charts showing the years covered by books in a series compared to their publishing dates https://t.co/2yUmmn3TOc https://t.co/OwNc6uJklg",
+  "source":"<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>",
+  "retweeted":false,
+  "coordinates":null,
+  "entities":{
+    "symbols":[
+
+    ],
+    "user_mentions":[
+
+    ],
+    "hashtags":[
+
+    ],
+    "urls":[
+      {
+        "url":"https://t.co/2yUmmn3TOc",
+        "indices":[
+          107,
+          130
+        ],
+        "expanded_url":"http://www.gyford.com/phil/writing/2017/09/05/book-series-charts.php",
+        "display_url":"gyford.com/phil/writing/2\u2026"
+      }
+    ],
+    "media":[
+      {
+        "expanded_url":"https://twitter.com/philgyford/status/905105588279013377/photo/1",
+        "display_url":"pic.twitter.com/OwNc6uJklg",
+        "url":"https://t.co/OwNc6uJklg",
+        "media_url_https":"https://pbs.twimg.com/media/DI-UuNlWAAA_cvz.jpg",
+        "id_str":"905105571765944320",
+        "sizes":{
+          "small":{
+            "h":256,
+            "resize":"fit",
+            "w":680
+          },
+          "large":{
+            "h":376,
+            "resize":"fit",
+            "w":1000
+          },
+          "medium":{
+            "h":376,
+            "resize":"fit",
+            "w":1000
+          },
+          "thumb":{
+            "h":150,
+            "resize":"crop",
+            "w":150
+          }
+        },
+        "indices":[
+          131,
+          154
+        ],
+        "type":"photo",
+        "id":905105571765944320,
+        "media_url":"http://pbs.twimg.com/media/DI-UuNlWAAA_cvz.jpg"
+      }
+    ]
+  },
+  "in_reply_to_screen_name":null,
+  "in_reply_to_user_id":null,
+  "display_text_range":[
+    0,
+    130
+  ],
+  "retweet_count":1,
+  "id_str":"905105588279013377",
+  "favorited":false,
+  "user":{
+    "screen_name":"philgyford",
+    "name":"Phil Gyford"
+  },
+  "geo":null,
+  "in_reply_to_user_id_str":null,
+  "possibly_sensitive":false,
+  "possibly_sensitive_appealable":false,
+  "lang":"en",
+  "created_at":"Tue Sep 05 16:29:22 +0000 2017",
+  "in_reply_to_status_id_str":null,
+  "place":null,
+  "extended_entities":{
+    "media":[
+      {
+        "expanded_url":"https://twitter.com/philgyford/status/905105588279013377/photo/1",
+        "display_url":"pic.twitter.com/OwNc6uJklg",
+        "url":"https://t.co/OwNc6uJklg",
+        "media_url_https":"https://pbs.twimg.com/media/DI-UuNlWAAA_cvz.jpg",
+        "id_str":"905105571765944320",
+        "sizes":{
+          "small":{
+            "h":256,
+            "resize":"fit",
+            "w":680
+          },
+          "large":{
+            "h":376,
+            "resize":"fit",
+            "w":1000
+          },
+          "medium":{
+            "h":376,
+            "resize":"fit",
+            "w":1000
+          },
+          "thumb":{
+            "h":150,
+            "resize":"crop",
+            "w":150
+          }
+        },
+        "indices":[
+          131,
+          154
+        ],
+        "type":"photo",
+        "id":905105571765944320,
+        "media_url":"http://pbs.twimg.com/media/DI-UuNlWAAA_cvz.jpg"
+      },
+      {
+        "expanded_url":"https://twitter.com/philgyford/status/905105588279013377/photo/1",
+        "display_url":"pic.twitter.com/OwNc6uJklg",
+        "url":"https://t.co/OwNc6uJklg",
+        "media_url_https":"https://pbs.twimg.com/media/DI-UuQbXUAQ1WlR.jpg",
+        "id_str":"905105572529393668",
+        "sizes":{
+          "large":{
+            "h":399,
+            "resize":"fit",
+            "w":1000
+          },
+          "small":{
+            "h":271,
+            "resize":"fit",
+            "w":680
+          },
+          "medium":{
+            "h":399,
+            "resize":"fit",
+            "w":1000
+          },
+          "thumb":{
+            "h":150,
+            "resize":"crop",
+            "w":150
+          }
+        },
+        "indices":[
+          131,
+          154
+        ],
+        "type":"photo",
+        "id":905105572529393668,
+        "media_url":"http://pbs.twimg.com/media/DI-UuQbXUAQ1WlR.jpg"
+      },
+      {
+        "expanded_url":"https://twitter.com/philgyford/status/905105588279013377/photo/1",
+        "display_url":"pic.twitter.com/OwNc6uJklg",
+        "url":"https://t.co/OwNc6uJklg",
+        "media_url_https":"https://pbs.twimg.com/media/DI-UuTIXcAA-t1_.jpg",
+        "id_str":"905105573255016448",
+        "sizes":{
+          "small":{
+            "h":195,
+            "resize":"fit",
+            "w":680
+          },
+          "large":{
+            "h":287,
+            "resize":"fit",
+            "w":1000
+          },
+          "medium":{
+            "h":287,
+            "resize":"fit",
+            "w":1000
+          },
+          "thumb":{
+            "h":150,
+            "resize":"crop",
+            "w":150
+          }
+        },
+        "indices":[
+          131,
+          154
+        ],
+        "type":"photo",
+        "id":905105573255016448,
+        "media_url":"http://pbs.twimg.com/media/DI-UuTIXcAA-t1_.jpg"
+      }
+    ]
+  }
+}

tests/tweets/quoted.json

@@ -0,0 +1,94 @@
+{  
+  "contributors":null,
+  "truncated":false,
+  "text":"Here\u2019s a quoted tweet. https://t.co/3neKzof0gT",
+  "is_quote_status":true,
+  "in_reply_to_status_id":null,
+  "id":917706313785905157,
+  "favorite_count":0,
+  "source":"<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>",
+  "quoted_status_id":917699069916729344,
+  "retweeted":false,
+  "coordinates":null,
+  "quoted_status":{  
+    "contributors":null,
+    "truncated":false,
+    "text":"The quoted tweet text.",
+    "is_quote_status":false,
+    "in_reply_to_status_id":null,
+    "id":917699069916729344,
+    "favorite_count":1,
+    "source":"<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>",
+    "retweeted":false,
+    "coordinates":null,
+    "entities":{  
+      "symbols":[  
+
+      ],
+      "user_mentions":[  
+
+      ],
+      "hashtags":[  
+
+      ],
+      "urls":[  
+
+      ]
+    },
+    "in_reply_to_screen_name":null,
+    "in_reply_to_user_id":null,
+    "retweet_count":0,
+    "id_str":"917699069916729344",
+    "favorited":false,
+    "user":{  
+      "screen_name":"philgyford",
+      "name":"Phil Gyford"
+    },
+    "geo":null,
+    "in_reply_to_user_id_str":null,
+    "lang":"ht",
+    "created_at":"Tue Oct 10 10:31:22 +0000 2017",
+    "in_reply_to_status_id_str":null,
+    "place":null
+  },
+  "entities":{  
+    "symbols":[  
+
+    ],
+    "user_mentions":[  
+
+    ],
+    "hashtags":[  
+
+    ],
+    "urls":[  
+      {  
+        "url":"https://t.co/3neKzof0gT",
+        "indices":[  
+          23,
+          46
+        ],
+        "expanded_url":"https://twitter.com/philgyford/status/917699069916729344",
+        "display_url":"twitter.com/philgyford/sta\u2026"
+      }
+    ]
+  },
+  "in_reply_to_screen_name":null,
+  "in_reply_to_user_id":null,
+  "retweet_count":0,
+  "id_str":"917706313785905157",
+  "favorited":false,
+  "user":{  
+    "screen_name":"philgyfordtest",
+    "name":"Phil Gyford Test"
+  },
+  "geo":null,
+  "in_reply_to_user_id_str":null,
+  "possibly_sensitive":false,
+  "possibly_sensitive_appealable":false,
+  "lang":"en",
+  "created_at":"Tue Oct 10 11:00:10 +0000 2017",
+  "quoted_status_id_str":"917699069916729344",
+  "in_reply_to_status_id_str":null,
+  "place":null
+}

tests/tweets/reply.json

@@ -0,0 +1,62 @@
+{
+  "display_text_range":[
+    12,
+    114
+  ],
+  "in_reply_to_status_id_str":"742374355531923456",
+  "source":"<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>",
+  "geo":null,
+  "full_text":"@philgyford Here\u2019s a test tweet that goes on as much as possible and includes an image. Hi to my fans in testland! https://t.co/tzhyk2QWSr",
+  "extended_entities":{
+    "media":[
+
+    ]
+  },
+  "id_str":"300",
+  "in_reply_to_status_id":742374355531923456,
+  "id":300,
+  "in_reply_to_screen_name":"philgyford",
+  "retweet_count":0,
+  "user":{
+
+  },
+  "created_at":"Mon Jun 13 15:48:06 +0000 2016",
+  "lang":"en",
+  "favorite_count":0,
+  "coordinates":null,
+  "place":null,
+  "contributors":null,
+  "in_reply_to_user_id":12552,
+  "in_reply_to_user_id_str":"12552",
+  "retweeted":false,
+  "favorited":false,
+  "truncated":false,
+  "entities":{
+    "user_mentions":[
+      {
+        "id_str":"12552",
+        "id":12552,
+        "screen_name":"philgyford",
+        "name":"Phil Gyford",
+        "indices":[
+          0,
+          11
+        ]
+      }
+    ],
+    "media":[
+
+    ],
+    "hashtags":[
+
+    ],
+    "symbols":[
+
+    ],
+    "urls":[
+
+    ]
+  },
+  "is_quote_status":false,
+  "possibly_sensitive":false
+}

tests/tweets/retweet.json

@@ -0,0 +1,91 @@
+{
+  "coordinates":null,
+  "source":"web",
+  "in_reply_to_user_id_str":null,
+  "truncated":false,
+  "in_reply_to_user_id":null,
+  "favorite_count":0,
+  "user":{
+    "name":"Phil Gyford Test",
+    "screen_name":"philgyfordtest"
+  },
+  "favorited":false,
+  "retweeted_status":{
+    "coordinates":null,
+    "source":"<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>",
+    "in_reply_to_user_id_str":null,
+    "truncated":false,
+    "in_reply_to_user_id":null,
+    "favorite_count":21,
+    "user":{
+      "name":"Samuel Pepys",
+      "screen_name":"samuelpepys"
+    },
+    "favorited":false,
+    "id":917459832885653506,
+    "contributors":null,
+    "in_reply_to_screen_name":null,
+    "geo":null,
+    "in_reply_to_status_id_str":null,
+    "id_str":"917459832885653506",
+    "entities":{
+      "hashtags":[
+
+      ],
+      "symbols":[
+
+      ],
+      "user_mentions":[
+
+      ],
+      "urls":[
+
+      ]
+    },
+    "in_reply_to_status_id":null,
+    "text":"My aunt and uncle in a very ill humour one with another, but I made shift with much ado to keep them from scolding.",
+    "retweet_count":3,
+    "place":null,
+    "lang":"en",
+    "created_at":"Mon Oct 09 18:40:44 +0000 2017",
+    "is_quote_status":false,
+    "retweeted":false
+  },
+  "id":917712989649801216,
+  "contributors":null,
+  "in_reply_to_screen_name":null,
+  "geo":null,
+  "in_reply_to_status_id_str":null,
+  "id_str":"917712989649801216",
+  "entities":{
+    "hashtags":[
+
+    ],
+    "symbols":[
+
+    ],
+    "user_mentions":[
+      {
+        "id_str":"14475268",
+        "indices":[
+          3,
+          15
+        ],
+        "name":"Samuel Pepys",
+        "id":14475268,
+        "screen_name":"samuelpepys"
+      }
+    ],
+    "urls":[
+
+    ]
+  },
+  "in_reply_to_status_id":null,
+  "text":"RT @samuelpepys: My aunt and uncle in a very ill humour one with another, but I made shift with much ado to keep them from scolding.",
+  "retweet_count":3,
+  "place":null,
+  "lang":"en",
+  "created_at":"Tue Oct 10 11:26:41 +0000 2017",
+  "is_quote_status":false,
+  "retweeted":false
+}

tests/tweets/symbols.json

@@ -0,0 +1,61 @@
+{
+  "text":"Some symbols: $AAPL and $PEP and $ANOTHER and $A.",
+  "contributors":null,
+  "geo":null,
+  "favorited":true,
+  "in_reply_to_user_id_str":null,
+  "user":{
+    "screen_name":"philgyfordtest",
+    "name":"Phil Gyford Test"
+  },
+  "in_reply_to_user_id":null,
+  "retweeted":false,
+  "coordinates":null,
+  "place":null,
+  "in_reply_to_status_id":null,
+  "lang":"en",
+  "in_reply_to_status_id_str":null,
+  "truncated":false,
+  "retweet_count":0,
+  "is_quote_status":false,
+  "id":662694880657989632,
+  "id_str":"662694880657989632",
+  "in_reply_to_screen_name":null,
+  "favorite_count":1,
+  "entities":{
+    "hashtags":[
+
+    ],
+    "user_mentions":[
+
+    ],
+    "symbols":[
+      {
+        "indices":[
+          14,
+          19
+        ],
+        "text":"AAPL"
+      },
+      {
+        "indices":[
+          24,
+          28
+        ],
+        "text":"PEP"
+      },
+      {
+        "indices":[
+          46,
+          48
+        ],
+        "text":"A"
+      }
+    ],
+    "urls":[
+
+    ]
+  },
+  "created_at":"Fri Nov 06 18:15:46 +0000 2015",
+  "source":"<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>"
+}

twython/__init__.py

@@ -19,7 +19,7 @@ Questions, comments? ryan@venodesigns.net
 """
 
 __author__ = 'Ryan McGrath <ryan@venodesigns.net>'
-__version__ = '3.4.0'
+__version__ = '3.9.1'
 
 from .api import Twython
 from .streaming import TwythonStreamer

twython/advisory.py

@@ -17,6 +17,6 @@ only TwythonDeprecationWarnings.
 class TwythonDeprecationWarning(DeprecationWarning):
     """Custom DeprecationWarning to be raised when methods/variables
     are being deprecated in Twython. Python 2.7 > ignores DeprecationWarning
-    so we want to specifcally bubble up ONLY Twython Deprecation Warnings
+    so we want to specifically bubble up ONLY Twython Deprecation Warnings
     """
     pass

twython/api.py

@@ -9,6 +9,7 @@ Twitter Authentication, and miscellaneous methods that are useful when
 dealing with the Twitter API
 """
 
+from __future__ import generator_stop
 import warnings
 import re
 
@@ -19,6 +20,7 @@ from requests_oauthlib import OAuth1, OAuth2
 from . import __version__
 from .advisory import TwythonDeprecationWarning
 from .compat import json, urlencode, parse_qsl, quote_plus, str, is_py2
+from .compat import urlsplit
 from .endpoints import EndpointsMixin
 from .exceptions import TwythonError, TwythonAuthError, TwythonRateLimitError
 from .helpers import _transparent_params
@@ -134,13 +136,17 @@ class Twython(EndpointsMixin, object):
     def __repr__(self):
         return '<Twython: %s>' % (self.app_key)
 
-    def _request(self, url, method='GET', params=None, api_call=None):
+    def _request(self, url, method='GET', params=None, api_call=None, json_encoded=False):
         """Internal request method"""
         method = method.lower()
         params = params or {}
 
         func = getattr(self.client, method)
-        params, files = _transparent_params(params)
+        if isinstance(params, dict) and json_encoded is False:
+            params, files = _transparent_params(params)
+        else:
+            params = params
+            files = list()
 
         requests_args = {}
         for k, v in self.client_args.items():
@@ -148,11 +154,16 @@ class Twython(EndpointsMixin, object):
             if k in ('timeout', 'allow_redirects', 'stream', 'verify'):
                 requests_args[k] = v
 
-        if method == 'get':
+        if method == 'get' or method == 'delete':
             requests_args['params'] = params
         else:
+            # Check for json_encoded so we will sent params as "data" or "json"
+            if json_encoded:
+                data_key = 'json'
+            else:
+                data_key = 'data'
             requests_args.update({
-                'data': params,
+                data_key: params,
                 'files': files,
             })
         try:
@@ -192,15 +203,16 @@ class Twython(EndpointsMixin, object):
                 error_message,
                 error_code=response.status_code,
                 retry_after=response.headers.get('X-Rate-Limit-Reset'))
-
+        content = ''
         try:
             if response.status_code == 204:
                 content = response.content
             else:
                 content = response.json()
         except ValueError:
-            raise TwythonError('Response was not valid JSON. \
+            if response.content != '':
-                               Unable to decode.')
+                raise TwythonError('Response was not valid JSON. \
+                                   Unable to decode.')
 
         return content
 
@@ -224,14 +236,14 @@ class Twython(EndpointsMixin, object):
 
         return error_message
 
-    def request(self, endpoint, method='GET', params=None, version='1.1'):
+    def request(self, endpoint, method='GET', params=None, version='1.1', json_encoded=False):
         """Return dict of response received from Twitter's API
 
         :param endpoint: (required) Full url or Twitter API endpoint
                          (e.g. search/tweets)
         :type endpoint: string
         :param method: (optional) Method of accessing data, either
-                       GET or POST. (default GET)
+                       GET, POST or DELETE. (default GET)
         :type method: string
         :param params: (optional) Dict of parameters (if any) accepted
                        the by Twitter API endpoint you are trying to
@@ -240,6 +252,9 @@ class Twython(EndpointsMixin, object):
         :param version: (optional) Twitter API version to access
                         (default 1.1)
         :type version: string
+        :param json_encoded: (optional) Flag to indicate if this method should send data encoded as json
+                        (default False)
+        :type json_encoded: bool
 
         :rtype: dict
         """
@@ -255,7 +270,7 @@ class Twython(EndpointsMixin, object):
             url = '%s/%s.json' % (self.api_url % version, endpoint)
 
         content = self._request(url, method=method, params=params,
-                                api_call=url)
+                                api_call=url, json_encoded=json_encoded)
 
         return content
 
@@ -263,9 +278,13 @@ class Twython(EndpointsMixin, object):
         """Shortcut for GET requests via :class:`request`"""
         return self.request(endpoint, params=params, version=version)
 
-    def post(self, endpoint, params=None, version='1.1'):
+    def post(self, endpoint, params=None, version='1.1', json_encoded=False):
         """Shortcut for POST requests via :class:`request`"""
-        return self.request(endpoint, 'POST', params=params, version=version)
+        return self.request(endpoint, 'POST', params=params, version=version, json_encoded=json_encoded)
+
+    def delete(self, endpoint, params=None, version='1.1', json_encoded=False):
+        """Shortcut for delete requests via :class:`request`"""
+        return self.request(endpoint, 'DELETE', params=params, version=version, json_encoded=json_encoded)
 
     def get_lastfunction_header(self, header, default_return_value=None):
         """Returns a specific header from the last API call
@@ -412,7 +431,7 @@ class Twython(EndpointsMixin, object):
 
     @staticmethod
     def construct_api_url(api_url, **params):
-        """Construct a Twitter API url, encoded, with parameters
+        r"""Construct a Twitter API url, encoded, with parameters
 
         :param api_url: URL of the Twitter API endpoint you are attempting
         to construct
@@ -451,7 +470,7 @@ class Twython(EndpointsMixin, object):
         return self.cursor(self.search, q=search_query, **params)
 
     def cursor(self, function, return_pages=False, **params):
-        """Returns a generator for results that match a specified query.
+        r"""Returns a generator for results that match a specified query.
 
         :param function: Instance of a Twython function
         (Twython.get_home_timeline, Twython.search)
@@ -470,6 +489,11 @@ class Twython(EndpointsMixin, object):
           >>>   print result
 
         """
+        if not callable(function):
+            raise TypeError('.cursor() takes a Twython function as its first \
+                            argument. Did you provide the result of a \
+                            function call?')
+
         if not hasattr(function, 'iter_mode'):
             raise TwythonError('Unable to create generator for Twython \
                                method "%s"' % function.__name__)
@@ -478,7 +502,7 @@ class Twython(EndpointsMixin, object):
             content = function(**params)
 
             if not content:
-                raise StopIteration
+                return
 
             if hasattr(function, 'iter_key'):
                 results = content.get(function.iter_key)
@@ -493,23 +517,31 @@ class Twython(EndpointsMixin, object):
 
             if function.iter_mode == 'cursor' and \
                content['next_cursor_str'] == '0':
-                raise StopIteration
+                return
 
             try:
                 if function.iter_mode == 'id':
-                    if 'max_id' not in params:
+                    # Set max_id in params to one less than lowest tweet id
-                        # Add 1 to the id because since_id and
+                    if hasattr(function, 'iter_metadata'):
-                        # max_id are inclusive
+                        # Get supplied next max_id
-                        if hasattr(function, 'iter_metadata'):
+                        metadata = content.get(function.iter_metadata)
-                            since_id = content[function.iter_metadata].get('since_id_str')
+                        if 'next_results' in metadata:
+                            next_results = urlsplit(metadata['next_results'])
+                            params = dict(parse_qsl(next_results.query))
                         else:
-                            since_id = content[0]['id_str']
+                            # No more results
-                        params['since_id'] = (int(since_id) - 1)
+                            return
+                    else:
+                        # Twitter gives tweets in reverse chronological order:
+                        params['max_id'] = str(int(content[-1]['id_str']) - 1)
                 elif function.iter_mode == 'cursor':
                     params['cursor'] = content['next_cursor_str']
             except (TypeError, ValueError):  # pragma: no cover
                 raise TwythonError('Unable to generate next page of search \
                                    results, `page` is not a number.')
+            except (KeyError, AttributeError):  #pragma no cover
+                raise TwythonError('Unable to generate next page of search \
+                                   results, content has unexpected structure.')
 
     @staticmethod
     def unicode2utf8(text):
@@ -528,7 +560,7 @@ class Twython(EndpointsMixin, object):
 
     @staticmethod
     def html_for_tweet(tweet, use_display_url=True, use_expanded_url=False, expand_quoted_status=False):
-        """Return HTML for a tweet (urls, mentions, hashtags replaced with links)
+        """Return HTML for a tweet (urls, mentions, hashtags, symbols replaced with links)
 
         :param tweet: Tweet object from received from Twitter API
         :param use_display_url: Use display URL to represent link
@@ -544,62 +576,122 @@ class Twython(EndpointsMixin, object):
         if 'retweeted_status' in tweet:
             tweet = tweet['retweeted_status']
 
+        if 'extended_tweet' in tweet:
+            tweet = tweet['extended_tweet']
+
+        orig_tweet_text = tweet.get('full_text') or tweet['text']
+
+        display_text_range = tweet.get('display_text_range') or [0, len(orig_tweet_text)]
+        display_text_start, display_text_end = display_text_range[0], display_text_range[1]
+        display_text = orig_tweet_text[display_text_start:display_text_end]
+        prefix_text = orig_tweet_text[0:display_text_start]
+        suffix_text = orig_tweet_text[display_text_end:len(orig_tweet_text)]
+
         if 'entities' in tweet:
-            text = tweet['text']
+            # We'll put all the bits of replacement HTML and their starts/ends
-            entities = tweet['entities']
+            # in this list:
+            entities = []
 
             # Mentions
-            for entity in sorted(entities['user_mentions'],
+            if 'user_mentions' in tweet['entities']:
-                                 key=lambda mention: len(mention['screen_name']), reverse=True):
+                for entity in tweet['entities']['user_mentions']:
-                start, end = entity['indices'][0], entity['indices'][1]
+                    temp = {}
+                    temp['start'] = entity['indices'][0]
+                    temp['end'] = entity['indices'][1]
 
-                mention_html = '<a href="https://twitter.com/%(screen_name)s" class="twython-mention">@%(screen_name)s</a>'
+                    mention_html = '<a href="https://twitter.com/%(screen_name)s" class="twython-mention">@%(screen_name)s</a>' % {'screen_name': entity['screen_name']}
-                text = re.sub(r'(?<!>)' + tweet['text'][start:end] + '(?!</a>)',
+
-                              mention_html % {'screen_name': entity['screen_name']}, text)
+                    if display_text_start <= temp['start'] <= display_text_end:
+                        temp['replacement'] = mention_html
+                        temp['start'] -= display_text_start
+                        temp['end'] -= display_text_start
+                        entities.append(temp)
+                    else:
+                        # Make the '@username' at the start, before
+                        # display_text, into a link:
+                        sub_expr = r'(?<!>)' + orig_tweet_text[temp['start']:temp['end']] + '(?!</a>)'
+                        prefix_text = re.sub(sub_expr, mention_html, prefix_text)
 
             # Hashtags
-            for entity in sorted(entities['hashtags'],
+            if 'hashtags' in tweet['entities']:
-                                 key=lambda hashtag: len(hashtag['text']), reverse=True):
+                for entity in tweet['entities']['hashtags']:
-                start, end = entity['indices'][0], entity['indices'][1]
+                    temp = {}
+                    temp['start'] = entity['indices'][0] - display_text_start
+                    temp['end'] = entity['indices'][1] - display_text_start
 
-                hashtag_html = '<a href="https://twitter.com/search?q=%%23%(hashtag)s" class="twython-hashtag">#%(hashtag)s</a>'
+                    url_html = '<a href="https://twitter.com/search?q=%%23%(hashtag)s" class="twython-hashtag">#%(hashtag)s</a>' % {'hashtag': entity['text']}
-                text = re.sub(r'(?<!>)' + tweet['text'][start:end] + '(?!</a>)',
-                              hashtag_html % {'hashtag': entity['text']}, text)
 
-            # Urls
+                    temp['replacement'] = url_html
-            for entity in entities['urls']:
+                    entities.append(temp)
-                start, end = entity['indices'][0], entity['indices'][1]
-                if use_display_url and entity.get('display_url') \
-                   and not use_expanded_url:
-                    shown_url = entity['display_url']
-                elif use_expanded_url and entity.get('expanded_url'):
-                    shown_url = entity['expanded_url']
-                else:
-                    shown_url = entity['url']
 
-                url_html = '<a href="%s" class="twython-url">%s</a>'
+            # Symbols
-                text = text.replace(tweet['text'][start:end],
+            if 'symbols' in tweet['entities']:
-                                    url_html % (entity['url'], shown_url))
+                for entity in tweet['entities']['symbols']:
+                    temp = {}
+                    temp['start'] = entity['indices'][0] - display_text_start
+                    temp['end'] = entity['indices'][1] - display_text_start
 
-             # Media
+                    url_html = '<a href="https://twitter.com/search?q=%%24%(symbol)s" class="twython-symbol">$%(symbol)s</a>' % {'symbol': entity['text']}
-            if 'media' in entities:
+
-                for entity in entities['media']:
+                    temp['replacement'] = url_html
-                    start, end = entity['indices'][0], entity['indices'][1]
+                    entities.append(temp)
-                    if use_display_url and entity.get('display_url') \
+
-                       and not use_expanded_url:
+            # URLs
+            if 'urls' in tweet['entities']:
+                for entity in tweet['entities']['urls']:
+                    temp = {}
+                    temp['start'] = entity['indices'][0] - display_text_start
+                    temp['end'] = entity['indices'][1] - display_text_start
+
+                    if use_display_url and entity.get('display_url') and not use_expanded_url:
                         shown_url = entity['display_url']
                     elif use_expanded_url and entity.get('expanded_url'):
                         shown_url = entity['expanded_url']
                     else:
                         shown_url = entity['url']
 
-                    url_html = '<a href="%s" class="twython-media">%s</a>'
+                    url_html = '<a href="%s" class="twython-url">%s</a>' % (entity['url'], shown_url)
-                    text = text.replace(tweet['text'][start:end],
-                                        url_html % (entity['url'], shown_url))
 
+                    if display_text_start <= temp['start'] <= display_text_end:
+                        temp['replacement'] = url_html
+                        entities.append(temp)
+                    else:
+                        suffix_text = suffix_text.replace(orig_tweet_text[temp['start']:temp['end']], url_html)
+
+            if 'media' in tweet['entities'] and len(tweet['entities']['media']) > 0:
+                # We just link to the overall URL for the tweet's media,
+                # rather than to each individual item.
+                # So, we get the URL from the first media item:
+                entity = tweet['entities']['media'][0]
+
+                temp = {}
+                temp['start'] = entity['indices'][0]
+                temp['end'] = entity['indices'][1]
+
+                if use_display_url and entity.get('display_url') and not use_expanded_url:
+                    shown_url = entity['display_url']
+                elif use_expanded_url and entity.get('expanded_url'):
+                    shown_url = entity['expanded_url']
+                else:
+                    shown_url = entity['url']
+
+                url_html = '<a href="%s" class="twython-media">%s</a>' % (entity['url'], shown_url)
+
+                if display_text_start <= temp['start'] <= display_text_end:
+                    temp['replacement'] = url_html
+                    entities.append(temp)
+                else:
+                    suffix_text = suffix_text.replace(orig_tweet_text[temp['start']:temp['end']], url_html)
+
+            # Now do all the replacements, starting from the end, so that the
+            # start/end indices still work:
+            for entity in sorted(entities, key=lambda e: e['start'], reverse=True):
+                display_text = display_text[0:entity['start']] + entity['replacement'] + display_text[entity['end']:]
+
+        quote_text = ''
         if expand_quoted_status and tweet.get('is_quote_status') and tweet.get('quoted_status'):
             quoted_status = tweet['quoted_status']
-            text += '<blockquote class="twython-quote">%(quote)s<cite><a href="%(quote_tweet_link)s">' \
+            quote_text += '<blockquote class="twython-quote">%(quote)s<cite><a href="%(quote_tweet_link)s">' \
                     '<span class="twython-quote-user-name">%(quote_user_name)s</span>' \
                     '<span class="twython-quote-user-screenname">@%(quote_user_screen_name)s</span></a>' \
                     '</cite></blockquote>' % \
@@ -609,4 +701,9 @@ class Twython(EndpointsMixin, object):
                      'quote_user_name': quoted_status['user']['name'],
                      'quote_user_screen_name': quoted_status['user']['screen_name']}
 
-        return text
+        return '%(prefix)s%(display)s%(suffix)s%(quote)s' % {
+            'prefix': '<span class="twython-tweet-prefix">%s</span>' % prefix_text if prefix_text else '',
+            'display': display_text,
+            'suffix': '<span class="twython-tweet-suffix">%s</span>' % suffix_text if suffix_text else '',
+            'quote': quote_text
+        }

twython/compat.py

@@ -25,7 +25,7 @@ except ImportError:
 
 if is_py2:
     from urllib import urlencode, quote_plus
-    from urlparse import parse_qsl
+    from urlparse import parse_qsl, urlsplit
 
     str = unicode
     basestring = basestring
@@ -33,7 +33,7 @@ if is_py2:
 
 
 elif is_py3:
-    from urllib.parse import urlencode, quote_plus, parse_qsl
+    from urllib.parse import urlencode, quote_plus, parse_qsl, urlsplit
 
     str = str
     basestring = (str, bytes)

twython/streaming/api.py

@@ -86,8 +86,6 @@ class TwythonStreamer(object):
         # Set up type methods
         StreamTypes = TwythonStreamerTypes(self)
         self.statuses = StreamTypes.statuses
-        self.user = StreamTypes.user
-        self.site = StreamTypes.site
 
         self.connected = False
 
@@ -125,7 +123,7 @@ class TwythonStreamer(object):
                     self.on_timeout()
                 else:
                     if response.status_code != 200:
-                        self.on_error(response.status_code, response.content)
+                        self.on_error(response.status_code, response.content, response.headers)
 
                     if self.retry_count and \
                        (self.retry_count - retry_counter) > 0:
@@ -169,16 +167,16 @@ class TwythonStreamer(object):
         Returns True if other handlers for this message should be invoked.
 
         Feel free to override this to handle your streaming data how you
-        want it handled.
+        want it handled. See
-        See https://dev.twitter.com/docs/streaming-apis/messages for messages
+        https://developer.twitter.com/en/docs/tweets/filter-realtime/guides/streaming-message-types
-        sent along in stream responses.
+        for messages sent along in stream responses.
 
         :param data: data recieved from the stream
         :type data: dict
         """
         return True
 
-    def on_error(self, status_code, data):  # pragma: no cover
+    def on_error(self, status_code, data, headers=None):  # pragma: no cover
         """Called when stream returns non-200 status code
 
         Feel free to override this to handle your streaming data how you
@@ -189,6 +187,9 @@ class TwythonStreamer(object):
 
         :param data: Error message sent from stream
         :type data: dict
+
+        :param headers: Response headers sent from the stream (i.e. Retry-After)
+        :type headers: dict
         """
         return
 

twython/streaming/types.py

@@ -20,33 +20,14 @@ class TwythonStreamerTypes(object):
         self.streamer = streamer
         self.statuses = TwythonStreamerTypesStatuses(streamer)
 
-    def user(self, **params):
-        """Stream user
-
-        Accepted params found at:
-        https://dev.twitter.com/docs/api/1.1/get/user
-        """
-        url = 'https://userstream.twitter.com/%s/user.json' \
-              % self.streamer.api_version
-        self.streamer._request(url, params=params)
-
-    def site(self, **params):
-        """Stream site
-
-        Accepted params found at:
-        https://dev.twitter.com/docs/api/1.1/get/site
-        """
-        url = 'https://sitestream.twitter.com/%s/site.json' \
-              % self.streamer.api_version
-        self.streamer._request(url, params=params)
-
 
 class TwythonStreamerTypesStatuses(object):
     """Class for different statuses endpoints
 
-    Available so TwythonStreamer.statuses.filter() is available.
+    Available so :meth:`TwythonStreamer.statuses.filter()` is available.
-    Just a bit cleaner than TwythonStreamer.statuses_filter(),
+    Just a bit cleaner than :meth:`TwythonStreamer.statuses_filter()`,
-    statuses_sample(), etc. all being single methods in TwythonStreamer
+    :meth:`statuses_sample()`, etc. all being single methods in 
+    :class:`TwythonStreamer`.
 
     """
     def __init__(self, streamer):
@@ -54,31 +35,31 @@ class TwythonStreamerTypesStatuses(object):
         self.params = None
 
     def filter(self, **params):
-        """Stream statuses/filter
+        r"""Stream statuses/filter
 
         :param \*\*params: Parameters to send with your stream request
 
         Accepted params found at:
-        https://dev.twitter.com/docs/api/1.1/post/statuses/filter
+        https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter
         """
         url = 'https://stream.twitter.com/%s/statuses/filter.json' \
               % self.streamer.api_version
         self.streamer._request(url, 'POST', params=params)
 
     def sample(self, **params):
-        """Stream statuses/sample
+        r"""Stream statuses/sample
 
         :param \*\*params: Parameters to send with your stream request
 
         Accepted params found at:
-        https://dev.twitter.com/docs/api/1.1/get/statuses/sample
+        https://developer.twitter.com/en/docs/tweets/sample-realtime/api-reference/get-statuses-sample
         """
         url = 'https://stream.twitter.com/%s/statuses/sample.json' \
               % self.streamer.api_version
         self.streamer._request(url, params=params)
 
     def firehose(self, **params):
-        """Stream statuses/firehose
+        r"""Stream statuses/firehose
 
         :param \*\*params: Parameters to send with your stream request
 
@@ -90,12 +71,12 @@ class TwythonStreamerTypesStatuses(object):
         self.streamer._request(url, params=params)
 
     def set_dynamic_filter(self, **params):
-        """Set/update statuses/filter
+        r"""Set/update statuses/filter
 
         :param \*\*params: Parameters to send with your stream request
 
         Accepted params found at:
-        https://dev.twitter.com/docs/api/1.1/post/statuses/filter
+        https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter
         """
         self.params = params
 
@@ -104,4 +85,4 @@ class TwythonStreamerTypesStatuses(object):
 
         url = 'https://stream.twitter.com/%s/statuses/filter.json' \
               % self.streamer.api_version
-        self.streamer._request(url, 'POST', params=self.params)	
+        self.streamer._request(url, 'POST', params=self.params)