Use a stub for the requests library in unit tests

cash commented

2014-01-06 17:27:15 -08:00

(Migrated from github.com)

The majority of the unit tests for Twython test the Twitter API rather than Twython. Using a requests stub would allow the unit tests to focus on the code in Twython and work around the issues with failing tests due to Twitter hiccups and rate limits.

Here is an example using HTTMock (https://github.com/patrys/httmock):

class TwythonAPITestCase(unittest.TestCase):
    def setUp(self):
        [snip]

    def _make_url_checker(self, path):
        def check_url(url, request):
            self.assertEqual(path, url.path)
            return "empty response"
        return check_url

    def test_get(self):
        """Test Twython generic GET request works"""
        with httmock.HTTMock(self._make_url_checker("/1.1/account/verify_credentials.json")):
            self.api.get('account/verify_credentials')

In the example, I use a closure to create a url checker, but it could be expanded to check the request method and parameters.

This shows how the unit tests can now focus on making sure the request (url, params, header) is correct and skip hitting the Twitter API entirely. For those functions that process the response, the mock can return a hard coded response.

To run just this unit test: nosetests tests.test_core:TwythonAPITestCase.test_get

What do you think?

The majority of the unit tests for Twython test the Twitter API rather than Twython. Using a requests stub would allow the unit tests to focus on the code in Twython and work around the issues with failing tests due to Twitter hiccups and rate limits. Here is an example using HTTMock (https://github.com/patrys/httmock): ``` class TwythonAPITestCase(unittest.TestCase): def setUp(self): [snip] def _make_url_checker(self, path): def check_url(url, request): self.assertEqual(path, url.path) return "empty response" return check_url def test_get(self): """Test Twython generic GET request works""" with httmock.HTTMock(self._make_url_checker("/1.1/account/verify_credentials.json")): self.api.get('account/verify_credentials') ``` In the example, I use a closure to create a url checker, but it could be expanded to check the request method and parameters. This shows how the unit tests can now focus on making sure the request (url, params, header) is correct and skip hitting the Twitter API entirely. For those functions that process the response, the mock can return a hard coded response. To run just this unit test: `nosetests tests.test_core:TwythonAPITestCase.test_get` What do you think?

michaelhelmick commented

2014-01-07 08:26:13 -08:00

(Migrated from github.com)

I think this is a good idea. In a newer library I'm writing -- I decided to go with @sigmavirus24 library, BetaMax. Another option would be using responses by Dropbox.

Runscope is another good option, but if anyone wanted to make a PR; they wouldn't be able to create new/edit old responses (if need-be).

Also, thanks for all the work you're planning to do with Twython and finding bugs! It's really appreciated!

I think this is a good idea. In a newer library I'm writing -- I decided to go with @sigmavirus24 library, `BetaMax`. Another option would be using `responses` by Dropbox. Runscope is another good option, but if anyone wanted to make a PR; they wouldn't be able to create new/edit old responses (if need-be). Also, thanks for all the work you're planning to do with Twython and finding bugs! It's really appreciated!

cash commented

2014-01-07 08:47:51 -08:00

(Migrated from github.com)

The Twython tests rarely parse/check the responses of the Twitter API. Does sigmavirus24/betamax allow you to inspect the request or does it only play back cached responses?

michaelhelmick commented

2014-01-07 08:51:24 -08:00

(Migrated from github.com)

The data is returned and you may do as you wish with it

michaelhelmick commented

2014-01-07 08:51:33 -08:00

(Migrated from github.com)

Didn't mean to close this* haha, sorry.

Didn't mean to close this\* haha, sorry.

sigmavirus24 commented

2014-01-07 09:09:13 -08:00

(Migrated from github.com)

@cash but matching of the request to return the cached data can basically perform your request inspection for you. If you want you can even define your own custom matcher to match requests which can/will check the body, headers, etc. Each of those are individually available but for unordered data, e.g., if you're body is a JSON encoded body then the python dictionary might not be great so you could write a matcher for a JSON body instead to check the parsed dictionaries are the same.

If you're interested in using Betamax, let me know and we can collaborate on writing the tests. I'm getting pip started on Betamax too

@cash but matching of the request to return the cached data can basically perform your request inspection for you. If you want you can even define your own custom matcher to match requests which can/will check the body, headers, etc. Each of those are individually available but for unordered data, e.g., if you're body is a JSON encoded body then the python dictionary might not be great so you could write a matcher for a JSON body instead to check the parsed dictionaries are the same. If you're interested in using Betamax, let me know and we can collaborate on writing the tests. I'm getting pip started on Betamax too

cash commented

2014-01-07 19:09:28 -08:00

(Migrated from github.com)

Here is an example of unit tests using dropbox/responses:

    @responses.activate
    def test_request_with_full_endpoint(self):
        url = 'http://example.com/test'
        responses.add(responses.GET, url)

        self.api.request(url)
        self.assertEqual(1, len(responses.calls))
        self.assertEqual(url, responses.calls[0].request.url)

    @responses.activate
    def test_request_with_relative_endpoint(self):
        url = 'https://api.twitter.com/2.0/search/tweets.json'
        responses.add(responses.GET, url)

        self.api.request('search/tweets', version='2.0')
        self.assertEqual(1, len(responses.calls))
        self.assertEqual(url, responses.calls[0].request.url)

    @responses.activate
    def test_request_with_non_get_method(self):
        url = 'http://example.com/test'
        responses.add(responses.POST, url)

        self.api.request(url, method='POST')
        self.assertEqual(1, len(responses.calls))
        self.assertEqual(url, responses.calls[0].request.url)

    @responses.activate
    def test_request_with_invalid_http_method(self):
        #TODO(cash): should Twython catch the AttributeError and throw a TwythonError
        self.assertRaises(AttributeError, self.api.request, endpoint='search/tweets', method='INVALID')

That feels much cleaner than HTTMock.

I haven't looked at Betamax yet, but it sounds cool - especially for libraries that do a lot of parsing of responses.

Here is an example of unit tests using dropbox/responses: ``` @responses.activate def test_request_with_full_endpoint(self): url = 'http://example.com/test' responses.add(responses.GET, url) self.api.request(url) self.assertEqual(1, len(responses.calls)) self.assertEqual(url, responses.calls[0].request.url) @responses.activate def test_request_with_relative_endpoint(self): url = 'https://api.twitter.com/2.0/search/tweets.json' responses.add(responses.GET, url) self.api.request('search/tweets', version='2.0') self.assertEqual(1, len(responses.calls)) self.assertEqual(url, responses.calls[0].request.url) @responses.activate def test_request_with_non_get_method(self): url = 'http://example.com/test' responses.add(responses.POST, url) self.api.request(url, method='POST') self.assertEqual(1, len(responses.calls)) self.assertEqual(url, responses.calls[0].request.url) @responses.activate def test_request_with_invalid_http_method(self): #TODO(cash): should Twython catch the AttributeError and throw a TwythonError self.assertRaises(AttributeError, self.api.request, endpoint='search/tweets', method='INVALID') ``` That feels much cleaner than HTTMock. I haven't looked at Betamax yet, but it sounds cool - especially for libraries that do a lot of parsing of responses.

sigmavirus24 commented

2014-01-07 20:02:56 -08:00

(Migrated from github.com)

@cash Betamax hooks into the requests Session so that you can use it just like you would expect it to work. You also don't have to worry about keeping track of what responses there are or what their body is like you would with dropbox's responses. I used a testing method that was almost exactly the same as dropbox's responses and found it far more complex than the example you posted. It's not elegant and it didn't scale well, not for >450 tests at least. That's why I wrote Betamax, especially since in my tests I do rely on some parsing.

cash commented

2014-01-08 09:45:39 -08:00

(Migrated from github.com)

I have now looked through the Betamax code and the documentation for the ruby vcr gem. I think responses and Betamax have different purposes and strengths. responses makes it easy to inspect the request arguments without ever doing any over the wire communication to an external resource. I did not see an obvious way to do that with Betamax. Likewise, if a particular endpoint intermittently fails, responses makes it easy to test that. Betamax is great when an application does a lot of parsing of the responses. I would definitely prefer it over responses for other projects that interact with web service call responses.

Twython is focused on creating http requests and more or less just passes back the responses to the client code. I think the only checking of the responses it performs are some basic status code checks and some json format sanity checks.

@michaelhelmick, if you are satisfied with this discussion, I'll start a branch based on the code in my previous comment.

I have now looked through the Betamax code and the documentation for the ruby vcr gem. I think responses and Betamax have different purposes and strengths. responses makes it easy to inspect the request arguments without ever doing any over the wire communication to an external resource. I did not see an obvious way to do that with Betamax. Likewise, if a particular endpoint intermittently fails, responses makes it easy to test that. Betamax is great when an application does a lot of parsing of the responses. I would definitely prefer it over responses for other projects that interact with web service call responses. Twython is focused on creating http requests and more or less just passes back the responses to the client code. I think the only checking of the responses it performs are some basic status code checks and some json format sanity checks. @michaelhelmick, if you are satisfied with this discussion, I'll start a branch based on the code in my previous comment.

michaelhelmick commented

2014-01-08 11:16:23 -08:00

(Migrated from github.com)

If an endpoint fails, it's easy to tell in Betamax as well. Once the actual request is made, Betamax will save the response to a json file -- mocking the status code and the response.

In Twython core tests, we test our endpoints API to assure they are valid against the Twitter API (i.e. if an endpoint was removed from the Twitter API and Twython still supported it, that test would fail)

If an endpoint fails, it's easy to tell in Betamax as well. Once the actual request is made, Betamax will save the response to a `json` file -- mocking the status code and the response. In Twython core tests, we test our endpoints API to assure they are valid against the Twitter API (i.e. if an endpoint was removed from the Twitter API and Twython still supported it, that test would fail)

sigmavirus24 commented

2014-01-08 11:21:09 -08:00

(Migrated from github.com)

@michaelhelmick also if you write a very strict matcher for Betamax (which is really really easy) it can do all the checking of URLs and request bodies that you like. I'd be happy to help write one.

The defaults are rather relaxed but that's only because I'm imitating the defaults of VCR. If you guys use that matcher then all you need to do is hit the endpoint once to record it and then in the future if something changes the tests will tell you immediately.

If something should change, fixing the error is as easy as re-recording the cassette (i.e., rm path/to/cassette.json && run_tests).

I'm of course biased though.

@michaelhelmick also if you write a very strict matcher for Betamax (which is _really really_ easy) it can do all the checking of URLs and request bodies that you like. I'd be happy to help write one. The defaults are rather relaxed but that's only because I'm imitating the defaults of [VCR](/vcr/vcr). If you guys use that matcher then all you need to do is hit the endpoint once to record it and then in the future if something changes the tests will tell you immediately. If something _should_ change, fixing the error is as easy as re-recording the cassette (i.e., `rm path/to/cassette.json && run_tests`). I'm of course biased though.

cash commented

2014-01-08 14:57:28 -08:00

(Migrated from github.com)

@sigmavirus24 how would I turn this responses unit test into a Betamax test:

    @responses.activate
    def test_request_with_full_endpoint(self):
        url = 'http://example.com/test'
        responses.add(responses.GET, url)

        self.api.request(url)
        self.assertEqual(1, len(responses.calls))
        self.assertEqual(url, responses.calls[0].request.url)

Note that I don't want this to actually make the request. I'm just trying to validate the the request.

@sigmavirus24 how would I turn this responses unit test into a Betamax test: ``` @responses.activate def test_request_with_full_endpoint(self): url = 'http://example.com/test' responses.add(responses.GET, url) self.api.request(url) self.assertEqual(1, len(responses.calls)) self.assertEqual(url, responses.calls[0].request.url) ``` Note that I don't want this to actually make the request. I'm just trying to validate the the request.

sigmavirus24 commented

2014-01-08 19:23:10 -08:00

(Migrated from github.com)

But why would you write a test for Twython that would be using example.com?

cash commented

2014-01-08 19:33:18 -08:00

(Migrated from github.com)

That's not important for this example unit test, but if you really want to know: Twython's request method lets you specify the full url for an endpoint and does not validate if it is actually hitting Twitter. I don't know if that was an intentional feature or an oversight. Theoretically, you could implement the twitter api on a different application and use Twython with that. Maybe some sort of open source micro blogging platform. I'm not saying this should be an actual unit test, but it's close enough for comparison sake.

sigmavirus24 commented

2014-01-08 20:14:36 -08:00

(Migrated from github.com)

Granted I'm not very familiar with Twython and I got sucked into this by @michaelhelmick but what's the semantic purpose of that?

I'm assuming that self.api is an instance of Twython. In Twython#__init__ I see no way to customize api_url when you create a new instance. Yes everything in Python is mutable, but there are several URLs generated as a result of that decision in __init__. If you're going to use Twython for another client that implements Twitter's API, you'll have to do a lot more work to get the auto-generated URLs to work properly anyway. Beyond that, continuing with my assumption that self.api in your example test is in fact an instance of Twython, Twython#request does not take a full URL, but instead a path.

I'm not saying this should be an actual unit test

I'm glad you're not because semantically it makes little sense to me (since we're getting into semantics, I felt we should make it clear to everyone else we're discussing semantics). The purpose of Twython is to speak to Twitter. The happy path in Twython is to talk to Twitter. Users do weird and often unexpected things with libraries, but the tests do not need to cover every one of those possibilities and libraries don't need to be everything to everyone (in fact, when they are they become awful). All of that said, the tests should be designed to test Twython's expected interaction with Twitter's API. With that in mind, there is likely not a test you can write with responses (that is semantically relevant to Twython) that cannot be written in Betamax. I'm not saying you have to use Betamax, just that your example is irrelevant.

Granted I'm not very familiar with Twython and I got sucked into this by @michaelhelmick but what's the semantic purpose of that? I'm assuming that `self.api` is an instance of `Twython`. In `Twython#__init__` I see no way to customize `api_url` when you create a new instance. Yes everything in Python is mutable, but there are several URLs generated as a result of that decision in `__init__`. If you're going to use Twython for another client that implements Twitter's API, you'll have to do a lot more work to get the auto-generated URLs to work properly anyway. Beyond that, continuing with my assumption that `self.api` in your example test is in fact an instance of `Twython`, `Twython#request` does not take a full URL, but instead a path. > I'm not saying this should be an actual unit test I'm glad you're not because semantically it makes little sense to me (since we're getting into semantics, I felt we should make it clear to everyone else we're discussing semantics). The purpose of Twython is to speak to Twitter. The happy path in Twython is to talk to Twitter. Users do weird and often unexpected things with libraries, but the tests do not need to cover every one of those possibilities and libraries don't need to be everything to everyone (in fact, when they are they become awful). All of that said, the tests should be designed to test Twython's expected interaction with Twitter's API. With that in mind, there is likely not a test you can write with responses (that is semantically relevant to Twython) that cannot be written in Betamax. I'm not saying you have to use Betamax, just that your example is irrelevant.

cash commented

2014-01-09 05:42:52 -08:00

(Migrated from github.com)

That may not be an actual test, but I really want tests that do not send out actual requests so it is a relevant example. Betamax is not a stub library but is meant for playing back actual responses. That means at some point the tests have to run against the external dependency (the Twitter API in this case). That's great for integration tests and tests that are focused on working with the responses of web services. It is not a good fit for tests where you don't want to have any external dependencies (handling rare errors, inspecting requests without making them, etc.).

@michaelhelmick I think dropbox/responses is the better choice for the Twython unit tests. It sounds like you are interested in having both unit tests and integration tests. Betamax could be used for integration tests where you occasionally clear the cassettes and test whether the Twitter API has changed. I more interested in the unit tests because I was fixing bugs that the current unit tests miss. Let me know how you want to proceed.

That may not be an actual test, but I really want tests that do not send out actual requests so it is a relevant example. Betamax is not a stub library but is meant for playing back actual responses. That means at some point the tests have to run against the external dependency (the Twitter API in this case). That's great for integration tests and tests that are focused on working with the responses of web services. It is not a good fit for tests where you don't want to have any external dependencies (handling rare errors, inspecting requests without making them, etc.). @michaelhelmick I think dropbox/responses is the better choice for the Twython unit tests. It sounds like you are interested in having both unit tests and integration tests. Betamax could be used for integration tests where you occasionally clear the cassettes and test whether the Twitter API has changed. I more interested in the unit tests because I was fixing bugs that the current unit tests miss. Let me know how you want to proceed.

michaelhelmick commented

2014-01-09 07:26:14 -08:00

(Migrated from github.com)

@sigmavirus24 Sorry for dragging you into this :P haha Developers can implement a call to Twitter via Twython a few ways

twitter = Twython(app_key, app_secret)

twitter.update_status(...)
twitter.post('statuses/update', ...)
twitter.post('https://api.twitter.com/1.1/statuses/update.json')

All of those update a status.

@cash, I think either could be used for both. We could create our own results or we can record a result and use it in the future. Almost all errors are raised after a HTTP request has happened. So you're always going to need a response to test again, well.. almost always. If you want to test against a Rate Limit Error, you'd either have to generate a Rate Limit Error and then save that response and use it in a responses mock test or you could generate it and save it to a file via Betamax without copy and pasting and all that, and Betamax also saves a lot more than JUST the response body (so you could check the status code and stuff) Betamax may be a little more overhead but offers more data about the responses.

@sigmavirus24 Sorry for dragging you into this :P haha Developers can implement a call to Twitter via Twython a few ways ``` python twitter = Twython(app_key, app_secret) twitter.update_status(...) twitter.post('statuses/update', ...) twitter.post('https://api.twitter.com/1.1/statuses/update.json') ``` All of those update a status. @cash, I think either could be used for both. We could create our own results or we can record a result and use it in the future. Almost all errors are raised after a HTTP request has happened. So you're always going to need a response to test again, well.. almost always. If you want to test against a Rate Limit Error, you'd either have to generate a Rate Limit Error and then save that response and use it in a `responses` mock test or you could generate it and save it to a file via Betamax without copy and pasting and all that, and Betamax also saves a lot more than JUST the response body (so you could check the status code and stuff) Betamax may be a little more overhead but offers more data about the responses.

cash commented

2014-01-09 08:37:50 -08:00

(Migrated from github.com)

Testing rate limit errors is an example where using a stub is better:

@responses.activate
def test_request_with_rate_limit_error(self):
    url = 'https://api.twitter.com/1.1/statuses/update.json'
    responses.add(responses.GET, url, status=429)

    self.assertRaises(TwythonRateLimitError, self.api.request, endpoint=url)

That's one of the points of using a stub in testing rather than using the external dependency - you can simulate stuff like this without actually creating a rate limit error. Since Twython only checks the response codes and checks error messages, it is very easy to mock the responses. If a library does a lot of processing on the responses, it makes mocking that much more difficult which pushes people to use tools that play back actual responses.

I think another important point about testing Twython is that right now the testing is weak on request creation and is more focused on testing the Twitter API (send a known request and see if we get an error back). Would you be open to me writing tests that focus on the requests and the cursor capability using dropbox/responses and leaving it up to you how you test all the methods that deal with endpoints? Maybe split the core test file into api and endpoints?

Testing rate limit errors is an example where using a stub is better: ``` @responses.activate def test_request_with_rate_limit_error(self): url = 'https://api.twitter.com/1.1/statuses/update.json' responses.add(responses.GET, url, status=429) self.assertRaises(TwythonRateLimitError, self.api.request, endpoint=url) ``` That's one of the points of using a stub in testing rather than using the external dependency - you can simulate stuff like this without actually creating a rate limit error. Since Twython only checks the response codes and checks error messages, it is very easy to mock the responses. If a library does a lot of processing on the responses, it makes mocking that much more difficult which pushes people to use tools that play back actual responses. I think another important point about testing Twython is that right now the testing is weak on request creation and is more focused on testing the Twitter API (send a known request and see if we get an error back). Would you be open to me writing tests that focus on the requests and the cursor capability using dropbox/responses and leaving it up to you how you test all the methods that deal with endpoints? Maybe split the core test file into api and endpoints?

michaelhelmick commented

2014-01-09 16:44:38 -08:00

(Migrated from github.com)

I say go for responses if we need a little more in-depth details about the actual response, we'll implement Betamax

I say go for `responses` if we need a little more in-depth details about the actual response, we'll implement Betamax

Rows
Columns

Use a stub for the requests library in unit tests #297