Get redirected url from response object - treq (twisted)

-
treq follows page redirects (by default) like how a browser does. So, no extra work needed here. But at times, we might need to know from where actually we got the response from.

Unlike python-request library, the treq/twisted response object doesn't have the 'url' attribute which contains the redirected url. But fortunately response object (in Twisted > 13.1.0) have 'previousResponse' attributes, from which we get the redirected url as below.


#!/bin/env python 
 
__author__ = 'vignesh' 
 
import treq
from twisted.internet import defer, reactor


def get_page(url):
    print "Requesting URL : %s" % url
    d = treq.get(url=url)

    def callback(result):
        previous = result.previousResponse
        if previous:
            location = previous.headers.getRawHeaders("location")
            if location:
                response_from_url = location[0]
            else:
                response_from_url = url
        else:
            response_from_url = url

        print "Got response from %s" % response_from_url

    def errback(error):
        print error

    d.addCallbacks(callback, errback)
    d.addBoth(lambda x: reactor.stop())

    reactor.run()

if __name__ == "__main__":
    get_page("http://google.com")
 
 
Output:

Requesting URL : http://google.com
Got response from 'http://www.google.co.in/?gfe_rd=cr&ei=leGSVdrDB6fv8wfvn4G4BQ'

 
 

Comments

Post a Comment

Popular posts from this blog

Can a Coroutine is useful without Asynchronous IO ?

-
Coroutine is a way of doing concurrent programming without the need of Thread. For more information regarding coroutine, go to asyncoro  and  asyncio-task . Coroutine has numerous advantages when used with asynchronous IO. But in this post I am analyzing whether coroutine can be used without it and also checking whether coroutine can be used to replace threads in most scenarios as threads itself is not running in parallel due to GIL  in python. Below are the advantage and disadvantage of coroutines (without Asynchronous IO) when compared against threads. Advantage     1. No locking mechanism required (in most cases) as the next coroutine will run only after the current coroutine yields.       2.  No thread, so no context switch required between coroutine execution.     Disadvantage     1. If a coroutine gets blocked, it will block all other coroutines as well as there is...
Read more

Different way to search for files in Gnome

-
Image
Searching for files is one of the most frequently performed task that most of the user does. In Gnome, it can be done by any of the following four ways. Using Nautilus Search Using gnome-search-tool Using Tracker Desktop Search Using zeitgeist   Using Nautilus for searching.       It's more natural to use the file manager to search for files instead of using an external app, that's why i decided to starting with nautilus first. Nautilus do have a built in search functionality that works by using tracker as backend. But most of the user complains that the nautilus does not shows any search result most of the cases. This is because, search is case sensitive , and  search is done only in directives which are indexed by the tracker.        By default, tracker index recursively the user's document, pictures,  videos, Desktop, Music, Downloads directories and non recursively on user's home directory . ...
Read more

Effective Use Of For Loop in Bash Shell Scripting

-
"for loops" are effective than "while loop" when a loop is to be done for each element in a array or each word in a string. This "for loop" is different, more powerful and useful then the version of for loop available in c/c . Here i am here to show some interesting new ways to use them in our bash scripts. Example 1: Simple and basic data="Bash scripting is fun" for i in $data do     echo "$i" done Output: Bash scripting is fun Explanation:     The bash [ Interpretor of the script ] splits each word in the variable data and each split piece is stored in the variable i on each iteration. The for loop starts and ends with 'do' and 'done' respectively. Example 2: simple and direct for i in Bash scripting is fun do     echo "$i" done Output: Bash scripting is fun Explanation:     Same as before but instead of using variable, we directly used data in the for...
Read more