Rice Cooker

A year and half ago, I did a project called Twitterball for one of my classes at PolyU on information architecture. It played on the idea of visualising the frequencies of tweets in time.

I quickly remashed Twitterball into Weiboball, using data from our archive and the search engine built with WeiboScope Search. The result is Weiboball and Weibobubble.

We belatedly announce a new service to retrieve all the comments or reposts of any given Sina Weibo post. The service will be very useful for researchers who want to study the contents of chatter surrounding any single post. We created a Google Form to submit posts to the system:

https://docs.google.com/spreadsheet/viewform?formkey=dEt6RmxDaGg4bEIzc3pDSWFiODNxSHc6MQ#gid=0

A Weibo ID is a 16-digit long numerical identifier. One way to find a post ID is to use one of our tools: WeiboScope and WeiboScope Search both expose the weibo ID). If you found your post via the Weibo.com website, once you are on the single post website (such as this one), go to the source code (press Ctrl-U if you are on Google Chrome) and do a Find (Ctrl-F) on “mid=”. The first digit starting with a 3 you find following “mid=” is your post ID (in the example we gave it’s “3433594570011824”).

Once you have your bunch of IDs, you can paste them, one per line, into the form and wait. The program running on our server will start to collect the posts using the Sina Weibo Open API and send you an e-mail that your job was queued. If the job is successful, you will get a second e-mail that tells you that all is ok, along a link to a zip file with your results in CSV format.

Try it and tell us if it works for you!

(I had the inspiration to write this service from the time when I was working in bioinformatics in the mid-2000s and where those sort of tasks, to find patterns in DNA or protein sequences, etc., normally took more time than a web user can wait for.)

While inspecting a potential hiccup to our Sina Weibo deleted posts monitoring system (see the ASL / see article explaining the method), we discovered the occurrence of a post that was seemingly deleted, but which still occasionally appeared on the API.

It was made as a repost made on March 2nd by David Bandurski of CMP, and was a repost of a Hu Shuli post commenting on the Wang Lijun incident on the same day. The post was archived, but can no longer be found on David’s timeline. Ms. Hu’s post was still alive on Weibo when I checked (and we have it archived, in case).

The strange thing was that this post was not marked as “permission denied” and was fully available when queried with the “statuses/show” function on the Weibo API (link — requires login). On the other hand, the user_timeline function, which lists the latest 200 posts made by a given user, reports that the post no longer existed.

Even more bizarre, still, when you searched on Weibo.com for David’s posts between the date range containing March 2nd, you’d see that the search page claims of 5 results, whereas only 4 are actually returned! This is shown in this following image:

For our monitoring tool, specifically the user_timeline function that casts the net for deleted posts, we have been using the first version (V1) of the API (because of rate limiting issues). V1 seems to me like the abandoned entrance for a data store that is more and more complex in its layers of visibility (and invisibility). These layers don’t seem supported with 100% fidelity on V1, and will return copies of user timeline that sometimes contained the “ghostly” post, and sometimes just won’t (probably depending on which physical server you end up accessing).

Not only there are “permission denied” or “weibo does not exist” posts, as described in our previous post about the methods behind the tool, but there are now posts wavering between a state of publication or non-publication (at least from the weibo.com website’s point of view, they were dead).

Anecdotally, I noticed that “permission denied” posts usually meant that their dependent posts (by people who repost) would also be marked as deleted with “permission denied” as well. In this case of the Hu Shuli post, the original post was alive and well, while the repost was deleted.

It’s the holy grail of any media researcher working on China: how do you quantify contents removal from social media services such as Sina Weibo? Here at JMSC, we’ve been developing tools to scour and assess social media of all sorts for the purpose of researching online media in Hong Kong and mainland China. (This is the same project that generated WeiboScope, for those tuned in.)

With the extensive archive of weibos we’ve accumulated so far and mechanisms underlying its retrieval, we were able to develop a routine that finds and marks deleted posts (method explained here).

The result is an archive of deleted posts (and the CMP’s Anti-Social List). Not only is it possible to find a large number (which is not exhaustive, we admit) of such posts within the day of their doom time, but also be able to presume with clear-cut evidence whether these deleted posts were removed by the user itself or simply deleted by system managers (we first noticed the difference in August 2011…).

As described in a post last week, the idea behind this archive is simple and straightforward to implement, once you’ve got the infrastructure.

A previous copy of the user timeline, containing all posts

A current copy of the user timeline, with a missing post

Both copies of a user timeline (post IDs extracted from the full JSON response) are obtained during two consecutive API calls, which may span a few minutes or several hours. The smaller this interval between pollings, the more precise would the routine be in finding the exact removal time (and the chances of missing something, smaller too).

A post is found to be deleted when you could see it in the previous version, but not the current one. Since we keep a copy of every post we see, we simply mark this post, and can then view them all in a custom webpage. Easy enough, right?

The previous two images show you exactly what, from a programmer’s point of view, Sina Weibo returns us for two different kinds of deleted posts. The former, with an API response “weibo does not exist”, identifies a post that was presumably deleted by the user. The latter, which returns “permission denied”, is presumably a post deleted by the system.

We don’t know the intention between both types of messages, but we can guess based on what their contents are generally. The first are generally made of spam-like posts that would be deleted on any online social network in the world. The second seem to provide more legitimate contents, including some made by so-called VIP users verified by Sina (we only check a 2500-odd sample of users, so can’t really infer on their representation).

This feature is indeed powerful, because it finally puts a number on post removal on Sina Weibo. (We computer science majors strongly dislike conclusions not based on numbers and data.)

It is however currently impossible to tell with certainty what gets deleted (versus what’s not), since our user sample is strongly biased towards public commentators, and perhaps because the number of posts found is still extremely small.

What it does give is an understanding of how post removal works, how much time it usually takes for something to be removed, and whether reposts of posts or just reposts but not original posts, get deleted (in fact, it happens). It’s a privileged peek, indeed, at what’s going on on the Chinese Internets, right here, right now.

Until we accumulate a substantial archive to do anything useful, our colleagues at China Media Project have started compiling and explaining deleted posts on their Anti-Social List.