Unable to run the Stream API sample

cfu1 commented

2013-01-16 09:09:23 -08:00

(Migrated from github.com)

Hi, I'm testing the stream API sample code here https://github.com/ryanmcgrath/twython, but keep getting this error: line 560, in stream TwythonError: 'Response was not valid JSON, unable to decode.' Any help is appreciated.

seepel commented

2013-01-16 10:03:52 -08:00

(Migrated from github.com)

This is similar to Issue 110. I ended up just repeating the same procedure in the streaming method.

This is similar to [Issue 110](https://github.com/ryanmcgrath/twython/pull/110). I ended up just repeating the same procedure in the streaming method.

cfu1 commented

2013-01-16 11:42:53 -08:00

(Migrated from github.com)

The basic issue here is why this method can't retrieve valid streaming data as it supposes to do. I'd like to retrieve the tweets within a specific geographic area. After googling around, I didn't find many useful samples. I thought the stream API can do it but got stuck here. Are there any alternatives to do it?

michaelhelmick commented

2013-04-10 19:57:06 -07:00

(Migrated from github.com)

I messed around with the Stream API the other day and it was indeed broken. I'm working on a fix for it. Sorry for any inconvenience!

michaelhelmick commented

2013-04-10 20:30:08 -07:00

(Migrated from github.com)

Good news! I actually figured out the problem, BUT I'm in a little need of help thinking of the best way to structure the Streaming API calls since there is more than just POSTing now.

@ryanmcgrath any ideas?

Right now, we're always doing requests.post in the stream, BUT there are endpoints that require GET (i.e. https://dev.twitter.com/docs/api/1.1/get/user)

stream will no longer be a staticmethod, instead it must be on the Twython class because all streams now require OAuth authentication rather than XAuth.

Good news! I actually figured out the problem, BUT I'm in a little need of help thinking of the best way to structure the Streaming API calls since there is more than just POSTing now. @ryanmcgrath any ideas? Right now, we're always doing `requests.post` in the stream, BUT there are endpoints that require GET (i.e. https://dev.twitter.com/docs/api/1.1/get/user) `stream` will **no** longer be a `staticmethod`, instead it must be on the Twython class because all streams now require OAuth authentication rather than XAuth.

ryanmcgrath commented

2013-04-11 01:07:30 -07:00

(Migrated from github.com)

Mmm, I'd personally just offer two Classes to import - ala, a Twython and a TwythonStreamer or something. From what experiences I have with the Streaming API it has its own logic that is possibly worth just keeping somewhat separate.

Mmm, I'd personally just offer two Classes to import - ala, a `Twython` and a `TwythonStreamer` or something. From what experiences I have with the Streaming API it has its own logic that is possibly worth just keeping somewhat separate.

michaelhelmick commented

2013-04-11 09:09:10 -07:00

(Migrated from github.com)

Hmmm, alright; I'm still having trouble thinking of the best way to implement it though.

class TwythonStreamer(Twython):

That's about where I'm at, haha.

I'm trying to think of the best way to make the stream method as dynamic as possible; or do we want to do like:

def stream(self, data, callback, method='POST')
   func = getattr(self.client, method)
   if method == 'get':
     response = func(url, params=params)
   else:
     response = func(url, data=params, files=files)
   ...

def streamFilter(self, data, callback, 'POST')
  return self.stream(data, callback, method)

Hmmm, alright; I'm still having trouble thinking of the best way to implement it though. ``` python class TwythonStreamer(Twython): ``` That's about where I'm at, haha. I'm trying to think of the best way to make the `stream` method as dynamic as possible; or do we want to do like: ``` python def stream(self, data, callback, method='POST') func = getattr(self.client, method) if method == 'get': response = func(url, params=params) else: response = func(url, data=params, files=files) ... def streamFilter(self, data, callback, 'POST') return self.stream(data, callback, method) ```

ryanmcgrath commented

2013-04-13 15:49:33 -07:00

(Migrated from github.com)

Hey,

Anyone interested in this issue, we're looking for feedback on this approach to the Streaming API. It's kind of taking a cue from Django's class based views - the thought is to localize the number of callbacks and logic surrounding dealing with the nature of the streaming API itself:

from twython import Twython, TwythonStreamHandler

t = Twython(a, s, o, os)
t = TwythonStreamHandler(a, s, o, os)

class TwythonStreamHandler(object):
    def __init__(self, url, type, data=None):
        data = data or {}

class TwythonSiteStreamer(TwythonStreamHandler):
    url = 'https://sitestream.twitter.com/1.1/site.json'

class TwythonStatusFilterStreamer(TwythonStreamHandler):
    url = 'https://stream.twitter.com/1.1/statuses/filter.json'
    type = 'POST'

class StreamHandler(TwythonStatusFilterStreamer):
    follow = [...]
    delimited = False
    stall_warnings = False
    replies = True

    def onDisconnect(self, code, stream_name, reason):
      # handle disconnect - reconnect?

    def onDeletion(self, status):
      # Handle deleted status

    def onRateLimitNotice(self, notice):
      # Handle rate limit issue

    def onReceiveWarning(self, msg):
      self.unfollow(1424234)
      # Handle warning - throttle back

StreamHandler(
    tokens = t.tokens(),
    ...
)

Hey, Anyone interested in this issue, we're looking for feedback on this approach to the Streaming API. It's kind of taking a cue from Django's class based views - the thought is to localize the number of callbacks and logic surrounding dealing with the nature of the streaming API itself: ``` python from twython import Twython, TwythonStreamHandler t = Twython(a, s, o, os) t = TwythonStreamHandler(a, s, o, os) class TwythonStreamHandler(object): def __init__(self, url, type, data=None): data = data or {} class TwythonSiteStreamer(TwythonStreamHandler): url = 'https://sitestream.twitter.com/1.1/site.json' class TwythonStatusFilterStreamer(TwythonStreamHandler): url = 'https://stream.twitter.com/1.1/statuses/filter.json' type = 'POST' class StreamHandler(TwythonStatusFilterStreamer): follow = [...] delimited = False stall_warnings = False replies = True def onDisconnect(self, code, stream_name, reason): # handle disconnect - reconnect? def onDeletion(self, status): # Handle deleted status def onRateLimitNotice(self, notice): # Handle rate limit issue def onReceiveWarning(self, msg): self.unfollow(1424234) # Handle warning - throttle back StreamHandler( tokens = t.tokens(), ... ) ```

ryanmcgrath commented

2013-04-20 15:18:19 -07:00

(Migrated from github.com)

Well, it's been a week. We should go with this approach unless someone can indicate to me if there's something wrong with it.

michaelhelmick commented

2013-05-04 12:15:40 -07:00

(Migrated from github.com)

@cfu1 @seepel Streaming has been fixed! Changes are in master and Twython 2.9.0 is now available via PyPi!
pip install -I twython

Here is an example how to use the new TwythonStreamer class:

from twython import TwythonStreamer


class MyStreamer(TwythonStreamer):
    def on_success(self, data):
        print data

    def on_error(self, status_code, data):
        print status_code, data

# Requires Authentication as of Twitter API v1.1
stream = MyStreamer(APP_KEY, APP_SECRET,
                    OAUTH_TOKEN, OAUTH_TOKEN_SECRET)

stream.statuses.filter(track='twitter')

Any questions, feel free to ask! :)

@cfu1 @seepel Streaming has been fixed! Changes are in `master` and `Twython` 2.9.0 is now available via PyPi! `pip install -I twython` Here is an example how to use the new `TwythonStreamer` class: ``` python from twython import TwythonStreamer class MyStreamer(TwythonStreamer): def on_success(self, data): print data def on_error(self, status_code, data): print status_code, data # Requires Authentication as of Twitter API v1.1 stream = MyStreamer(APP_KEY, APP_SECRET, OAUTH_TOKEN, OAUTH_TOKEN_SECRET) stream.statuses.filter(track='twitter') ``` Any questions, feel free to ask! :)

sladiwala commented

2013-05-04 16:46:59 -07:00

(Migrated from github.com)

Installation from pip failed
pip install -I twython

from .streaming import TwythonStreamer
ImportError: No module named streaming

Had to download it from GitHub.

Installation from pip failed pip install -I twython from .streaming import TwythonStreamer ImportError: No module named streaming Had to download it from GitHub.

ryanmcgrath commented

2013-05-04 16:49:14 -07:00

(Migrated from github.com)

Hmmm, I just tried this and can't reproduce the error. The following code
has problems for you?

from twython import TwythonStreamer

Hmmm, I just tried this and can't reproduce the error. The following code has problems for you? ``` python from twython import TwythonStreamer ```

michaelhelmick commented

2013-05-04 16:52:41 -07:00

(Migrated from github.com)

I just reproduced this. On sec

michaelhelmick commented

2013-05-04 16:54:15 -07:00

(Migrated from github.com)

Needed to add

packages = [
    'twython',
    'twython.streaming'
]

to setup.py I believe. I'll push the new pep8 stuff along with this soon.

Needed to add ``` python packages = [ 'twython', 'twython.streaming' ] ``` to `setup.py` I believe. I'll push the new pep8 stuff along with this soon.

michaelhelmick commented

2013-05-04 17:16:34 -07:00

(Migrated from github.com)

Try it now @sladiwala! :D

sladiwala commented

2013-05-04 17:19:30 -07:00

(Migrated from github.com)

Works! Thanks for the quick turnaround.

michaelhelmick commented

2013-05-04 17:21:49 -07:00

(Migrated from github.com)

Ps @sladiwala, let us know if you find working with our Streaming API simple to understand! :)

harisibrahimkv commented

2013-06-23 00:58:15 -07:00

(Migrated from github.com)

I was playing around with the same streaming example itself. Following is my piece of code:

from twython import TwythonStreamer

class MyStreamer(TwythonStreamer):
    def on_success(self, data):
        if 'text' in data:
            print data['text'].encode('utf-8')

    def on_error(self, status_code, data):
        print status_code, data

stream = MyStreamer(APP_KEY, APP_SECRET,
                    OAUTH_TOKEN, OAUTH_TOKEN_SECRET)

stream.statuses.filter(follow='harisibrahimkv')

However, I was getting a 406 error as follows:

406 Parameter follow has unparseable items harisibrahimkv

406 Unable to decode response, not vaild JSON.

I thought the follow variable was supposed to take Twitter usernames as values and return their real time tweets. Please see if this is a bug. If not, I am ready to get scolded. :)

I was playing around with the same streaming example itself. Following is my piece of code: ``` from twython import TwythonStreamer class MyStreamer(TwythonStreamer): def on_success(self, data): if 'text' in data: print data['text'].encode('utf-8') def on_error(self, status_code, data): print status_code, data stream = MyStreamer(APP_KEY, APP_SECRET, OAUTH_TOKEN, OAUTH_TOKEN_SECRET) stream.statuses.filter(follow='harisibrahimkv') ``` However, I was getting a 406 error as follows: **406 Parameter follow has unparseable items harisibrahimkv** **406 Unable to decode response, not vaild JSON.** I thought the follow variable was supposed to take Twitter usernames as values and return their real time tweets. Please see if this is a bug. If not, I am ready to get scolded. :)

faheempatel commented

2013-06-23 05:55:42 -07:00

(Migrated from github.com)

@harisibrahimkv It's because the follow parameter takes in user ids as values and not usernames. Lucky for us it isn't difficult to get the user id for a given username:

from twython import Twython

t = Twython(APP_KEY, APP_SECRET,
                    OAUTH_TOKEN, OAUTH_TOKEN_SECRET)

user_id = t.lookup_user(screen_name='harisibrahimkv')[0]['id_str']

@harisibrahimkv It's because the _follow_ parameter takes in user ids as values and not usernames. Lucky for us it isn't difficult to get the user id for a given username: ``` python from twython import Twython t = Twython(APP_KEY, APP_SECRET, OAUTH_TOKEN, OAUTH_TOKEN_SECRET) user_id = t.lookup_user(screen_name='harisibrahimkv')[0]['id_str'] ```

harisibrahimkv commented

2013-06-23 08:38:17 -07:00

(Migrated from github.com)

Oh, ID literally meant a number huh? Maybe this should have been hinted in the documentation. Would be helpful to noobs.

Thanks again. I would have gone nuts trying to figure out how to get the user ID if you had not mentioned it.

Oh, ID literally meant a number huh? Maybe this should have been hinted in the documentation. Would be helpful to noobs. Thanks again. I would have gone nuts trying to figure out how to get the user ID if you had not mentioned it.

michaelhelmick commented

2013-06-23 09:00:01 -07:00

(Migrated from github.com)

Yes, it takes the IDs. Sorry about the wrong example. You can use track=yourusername

Although!! Until requests pushes a new update, all streams following users who don't get a lot of interaction will be one post behind. :( (there is an issue out for this in twython you can go check what exactly is going on) it will hopefully be merged soon though!

Sent from my iPhone

On Jun 23, 2013, at 11:38 AM, "Haris Ibrahim K. V." notifications@github.com wrote:

Oh, ID literally meant a number huh? Maybe this should have been hinted in the documentation. Would be helpful to noobs.

Thanks again. I would have gone nuts trying to figure out how to get the user ID if you had not mentioned it.

—
Reply to this email directly or view it on GitHub.

Yes, it takes the IDs. Sorry about the wrong example. You can use track=yourusername Although!! Until requests pushes a new update, all streams following users who don't get a lot of interaction will be one post behind. :( (there is an issue out for this in twython you can go check what exactly is going on) it will hopefully be merged soon though! Sent from my iPhone > On Jun 23, 2013, at 11:38 AM, "Haris Ibrahim K. V." notifications@github.com wrote: > > Oh, ID literally meant a number huh? Maybe this should have been hinted in the documentation. Would be helpful to noobs. > > Thanks again. I would have gone nuts trying to figure out how to get the user ID if you had not mentioned it. > > — > Reply to this email directly or view it on GitHub.

surank commented

2016-04-30 09:15:38 -07:00

(Migrated from github.com)

so im trying to stream and I'm using the exact same code as above, ive used Twython before to fetch data using the search function...

from twython import TwythonStreamer

class MyStreamer(TwythonStreamer):
def on_success(self, data):
if 'text' in data:
print data['text'].encode('utf-8')

def on_error(self, status_code, data):
    print status_code, data

stream = MyStreamer(APP_KEY, APP_SECRET,
OAUTH_TOKEN, OAUTH_TOKEN_SECRET)

stream.statuses.filter(track='#fashion')

but i am not seeing any data....

so im trying to stream and I'm using the exact same code as above, ive used Twython before to fetch data using the search function... from twython import TwythonStreamer class MyStreamer(TwythonStreamer): def on_success(self, data): if 'text' in data: print data['text'].encode('utf-8') ``` def on_error(self, status_code, data): print status_code, data ``` stream = MyStreamer(APP_KEY, APP_SECRET, OAUTH_TOKEN, OAUTH_TOKEN_SECRET) stream.statuses.filter(track='#fashion') but i am not seeing any data....

Rows
Columns

Unable to run the Stream API sample #144