DZone Snippets is a public source code repository. Easily build up your personal collection of code snippets, categorize them with tags / keywords, and share them with the world

Snippets has posted 5883 posts at DZone. View Full User Profile

Custom HTTP/HTTPS GET/POST Queries In Ruby

10.05.2005
| 169990 views |
  • submit to reddit
        First, the scripts sends the GET query to read the website cookies (for session, etc.), and then it sends a POST query with the received cookies and custom POST parameters.

require 'net/http'
require 'net/https'

http = Net::HTTP.new('profil.wp.pl', 443)
http.use_ssl = true
path = '/login.html'

# GET request -> so the host can set his cookies
resp, data = http.get(path, nil)
cookie = resp.response['set-cookie']


# POST request -> logging in
data = 'serwis=wp.pl&url=profil.html&tryLogin=1&countTest=1&logowaniessl=1&login_username=blah&login_password=blah'
headers = {
  'Cookie' => cookie,
  'Referer' => 'http://profil.wp.pl/login.html',
  'Content-Type' => 'application/x-www-form-urlencoded'
}

resp, data = http.post(path, data, headers)


# Output on the screen -> we should get either a 302 redirect (after a successful login) or an error page
puts 'Code = ' + resp.code
puts 'Message = ' + resp.message
resp.each {|key, val| puts key + ' = ' + val}
puts data
    

Comments

Carla Brian replied on Sun, 2012/07/22 - 12:03am

This is really nice. This might be helpful in the future. i don't know more about this one. I want to learn more on this. - Mercy Ministries

Simab Si replied on Sun, 2012/05/20 - 7:35am

Nice to see the quality information presented in an easy and understanding manner. This is very nice to see this blog and it's really informative for the readers.
Thank you.

UK dissertations

Snippets Manager replied on Sat, 2009/06/06 - 8:53pm

Thanks for the example code. There are also Mechanize Ruby which maybe useful for these. Boost your website ranking with high quality edu text links . Read more on women's topics .

Snippets Manager replied on Mon, 2006/02/20 - 11:36am

Well, my example is now very old -- I had almost forgotten about it, since I wrote it about 3 years ago. But the reason why I URL-encoded the login and password is because all non-alphanumeric data should be URL encoded to ensure it is sent properly. For example, what if your password has a & in it? That could throw off the whole request. URL encoding has nothing to do with security. If you look at the URL for a Google Search request, next to q= you could -- on some browsers -- see your request URL encoded. It's the type of stuff like %24 for &, or %3D for = or %2F for /. Space is URL-encoded to + or %20, so + is actually %2B. That sort of thing. So since I can't predict the content of people's logins, or especially, their passwords, we should URL encode them to ensure they are sent correctly and our request is well-formed. Also, if people are doing web scraping or want to mirror as much as possible a request that works in a web browser, I highly recommend Charles Proxy from www.charlesproxy.com - written in fast Java code, it runs on Mac, Linux or Windows and works equally well for HTTP and HTTPS requests, regardless of browser. It even works with applications that don't support proxy servers, though the use of reverse proxying and either host files or some other form of DNS redirection. For instance, it can be configured to listen on localhost:81 and pass requests on to example.org, and as long as your hosts file says 127.0.0.1 example.org, your application will be none the wiser. It's also great for normal web development, as you can see a timeline of your requests, with or without bandwidth throttling, and inspect cookies, headers, and AJAX responses. It even has excellent support for Flash inspection. Plus you can create your own requests or craft them by editing existing ones. An amazing tool, and I'm constantly finding new ways to use it. Perfect for all web projects that requires code like this. (And no, I'm not getting paid by the developer to recommend it. It's just that good. And did I mention its Mac-compatibility?)

Snippets Manager replied on Mon, 2009/02/02 - 1:45pm

I am just wondering what purpose is served by url encoding the username and password, is this just for security? @grosser, your example is awesome man, thanks! ------------------------ Jesdon real estate license

Snippets Manager replied on Wed, 2008/03/19 - 5:38pm

I build a dead-simple post method from parts of your snippet, hope you can use it ;) http://pragmatig.wordpress.com/2009/01/23/simple-http-post-without-nethttp-complexity/

Snippets Manager replied on Thu, 2008/11/13 - 1:33am

DZone Snippets is a very good website. We can share our problems and views. The contents of the site is informative. ========================= Harry homes for sale by owner

Snippets Manager replied on Mon, 2006/02/20 - 11:36am

Ahh, I just spent the last ... oh I don't know how long - lets say three hours - trying to get this code to work. I finally did, but the problem was I needed to specify a user agent! So here's my code snippet in full - and get2/post2 is interchangable with get/post, the difference is that the 2 is safer, it won't toss any Net:: Errors. Also notice that I URL encoded the username and password. require "net/http" require "net/https" require "erb" require "singleton" PANEL = 'panel.dreamhost.com' PATH = '/index.cgi' USERAGENT = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1' class Panel include Singleton def Panel.login(webid, password) unless @headers.nil? raise ScriptError, "You should already be logged in!" end @http = Net::HTTP.new(PANEL, 443) @http.use_ssl = true # GET request -> so the host can set cookies resp, data = @http.get2(PATH, {'User-Agent' => USERAGENT}) cookie = resp.response['set-cookie'].split('; ')[0] # POST request -> logging in data = "Nscmd=Nlogin&username=#{ERB::Util.url_encode(webid)}&password=#{ERB::Util.url_encode(password)}&submit=Log%20In" @headers = { 'Cookie' => cookie, 'Referer' => 'https://'+PANEL+PATH, 'Content-Type' => 'application/x-www-form-urlencoded', 'User-Agent' => USERAGENT } puts "headers = " + @headers.inspect resp, data = @http.post2(PATH, data, @headers) # Output on the screen -> we should get either a 302 redirect (after a successful login) or an error page puts 'Code = ' + resp.code puts 'Message = ' + resp.message resp.each {|key, val| puts key + ' = ' + val} puts data end end If you curious, I'm scraping my own API for DreamHost's Control Panel, so I don't have to labourously open my web browser every time I need to add or change something. (Yeah, I'm that lazy.) Anyway, its unfinished, I'll be refactoring the code next - the Panel.login method was just for testing.