twython3k search queries fail due to bytes vs. str

wescpy commented

2011-10-04 14:17:54 -07:00

(Migrated from github.com)

JSON unpack seems to be failing because a bytes object is subjected to a string operation. Repro code plus TB below:

#!/usr/bin/env python3

from pprint import pprint
import twython3k as twython

twitter = twython.Twython()
data = twitter.searchTwitter(q='twython3k')
res = data['results']
for tweet in res:
    pprint(tweet)

'''
Traceback (most recent call last):
  File "twython3k-bug.py", line 7, in <module>
    data = twitter.searchTwitter(q='twython3k')
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/twython-1.4.3-py3.2.egg/twython3k/twython.py", line 251, in searchTwitter
    return simplejson.loads(content)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/json/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/json/decoder.py", line 345, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: can't use a string pattern on a bytes-like object
'''

below is my half-attempt at a patch, which involves tweaking some stuff which may break other things, but it does work for searches.

$ diff -u twython-*py
--- twython-orig.py     2011-10-04 00:23:10.000000000 -0700
+++ twython-wescpy.py   2011-10-04 13:43:36.000000000 -0700
@@ -194,7 +194,7 @@

        @staticmethod
        def constructApiURL(base_url, params):
-               return base_url + "?" + "&".join(["%s=%s" %(Twython.unicode2utf8(key), urllib.parse.quote_plus(Twython.unicode2utf8(value))) for (key, value) in list(params.items())])
+               return base_url + "?" + "&".join(["%s=%s" % (key, urllib.parse.quote_plus(Twython.unicode2utf8(value))) for (key, value) in list(params.items())])

        @staticmethod
        def shortenURL(url_to_shorten, shortener = "http://is.gd/api.php", query = "longurl"):
@@ -248,7 +248,7 @@
                searchURL = Twython.constructApiURL("http://search.twitter.com/search.json", kwargs)
                try:
                        resp, content = self.client.request(searchURL, "GET")
-                       return simplejson.loads(content)
+                       return simplejson.loads(content.decode('utf-8'))
                except HTTPError as e:
                        raise TwythonError("getSearchTimeline() failed with a %s error code." % repr(e.code), e.code)

other functionality such as userTimeline and verifyCredentials work with or without the patch. at first, i thought the bug was going to be present for all commands that take parameters since clearly those that don't were still working. however, since updateStatus also works, i've got to conclude it's because the problem lies in twython3k.Twython.searchTwitter() which duplicates code in twython3k.Twython.get(). IOW, the bug was fixed in get() but not in searchTwitter() due to similar code being in 2 places. (i understand this may be required since searching is more complex than the other operations.)

JSON unpack seems to be failing because a bytes object is subjected to a string operation. Repro code plus TB below: ``` #!/usr/bin/env python3 from pprint import pprint import twython3k as twython twitter = twython.Twython() data = twitter.searchTwitter(q='twython3k') res = data['results'] for tweet in res: pprint(tweet) ''' Traceback (most recent call last): File "twython3k-bug.py", line 7, in <module> data = twitter.searchTwitter(q='twython3k') File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/twython-1.4.3-py3.2.egg/twython3k/twython.py", line 251, in searchTwitter return simplejson.loads(content) File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/json/__init__.py", line 307, in loads return _default_decoder.decode(s) File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/json/decoder.py", line 345, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) TypeError: can't use a string pattern on a bytes-like object ''' ``` below is my half-attempt at a patch, which involves tweaking some stuff which may break other things, but it does work for searches. ``` $ diff -u twython-*py --- twython-orig.py 2011-10-04 00:23:10.000000000 -0700 +++ twython-wescpy.py 2011-10-04 13:43:36.000000000 -0700 @@ -194,7 +194,7 @@ @staticmethod def constructApiURL(base_url, params): - return base_url + "?" + "&".join(["%s=%s" %(Twython.unicode2utf8(key), urllib.parse.quote_plus(Twython.unicode2utf8(value))) for (key, value) in list(params.items())]) + return base_url + "?" + "&".join(["%s=%s" % (key, urllib.parse.quote_plus(Twython.unicode2utf8(value))) for (key, value) in list(params.items())]) @staticmethod def shortenURL(url_to_shorten, shortener = "http://is.gd/api.php", query = "longurl"): @@ -248,7 +248,7 @@ searchURL = Twython.constructApiURL("http://search.twitter.com/search.json", kwargs) try: resp, content = self.client.request(searchURL, "GET") - return simplejson.loads(content) + return simplejson.loads(content.decode('utf-8')) except HTTPError as e: raise TwythonError("getSearchTimeline() failed with a %s error code." % repr(e.code), e.code) ``` other functionality such as userTimeline and verifyCredentials work with or without the patch. at first, i thought the bug was going to be present for all commands that take parameters since clearly those that don't were still working. however, since updateStatus also works, i've got to conclude it's because the problem lies in twython3k.Twython.searchTwitter() which duplicates code in twython3k.Twython.get(). IOW, the bug was fixed in get() but not in searchTwitter() due to similar code being in 2 places. (i understand this may be required since searching is more complex than the other operations.)

ryanmcgrath commented

2011-10-06 13:20:43 -07:00

(Migrated from github.com)

Whew! Sorry about this taking so long to get to, bunch of random things I had to deal with. I think this should be good now with the 1.4.4 release, but as always please feel free to test and confirm.

I also updated twitter_endpoints, as I believe it was you over Twitter who pointed out how woefully out of date it was. Cheers.

Whew! Sorry about this taking so long to get to, bunch of random things I had to deal with. I _think_ this should be good now with the 1.4.4 release, but as always please feel free to test and confirm. I also updated twitter_endpoints, as I believe it was you over Twitter who pointed out how woefully out of date it was. Cheers.

wescpy commented

2011-10-06 14:34:47 -07:00

(Migrated from github.com)

close, just some small issues remaining which i've filed separately... sorry to bother you with all this. i know the world isn't using Python 3 yet, but i'm trying to ensure a smooth experience for those who do moving forward! :-)

ryanmcgrath commented

2011-10-06 14:44:47 -07:00

(Migrated from github.com)

It's not a bother; if I put software out there for people to use, I'll do what I can to maintain it properly. Your efforts are well received, at least in my mind.

As always, let me know if you find anything else!

It's not a bother; if I put software out there for people to use, I'll do what I can to maintain it properly. Your efforts are well received, at least in my mind. As always, let me know if you find anything else!

Rows
Columns

twython3k search queries fail due to bytes vs. str #43