DZone Snippets is a public source code repository. Easily build up your personal collection of code snippets, categorize them with tags / keywords, and share them with the world

Snippets

  • submit to reddit

Recent Snippets

                    Here's my source code in ColdFusion for parsing 10x10. I can't imagine it being different in other languages.

----

<!--- build the wordsURL before getting the data --->
<cfset year = DateFormat(Now(),"YYYY")>
<cfset month = DateFormat(Now(),"MM")>
<cfset day = DateFormat(Now(),"DD")>
<cfset hour = TimeFormat(Now(),"HH")>
<cfset wordsURL = "http://tenbyten.org/Data/#year#/#month#/#day#/#hour#/words.txt">

<!--- connect to the url --->
<cfhttp method="get" url="#wordsURL#" timeout="60"></cfhttp>

<cfset wordlist = cfhttp.FileContent>

<!--- write the data into a text file --->
<cffile action="write" file="#ExpandPath('words.txt')#" output="#ToString(wordlist)#">

----

Reading the content is as simple as opening the file and traversing through it like so. I also added in a link to Google news. I've found it helpful to click on news bits in the past.

----

<cfhttp method="get" url="http://path/to/words.txt" timeout="60"></cfhttp>

<cfset wordlist = cfhttp.FileContent>

<ul><li>
<cfloop list="#wordlist#" delimiters="#chr(13)##chr(10)#" index="word">
  <cfoutput>
    <a href="http://news.google.com/news?q=#word#" target="_blank">#word#</a>
  </cfoutput>
</cfloop>
</li></ul>
                
                    This ruby script dumps all your del.icio.us bookmarks into a sqlite database. Requires sqlite3-ruby installed via rubygems. Don't forget to change the user and pass.
#!/usr/bin/env ruby
require 'rexml/document'
require 'net/http'
require 'net/https'
require 'rubygems'
require 'sqlite3'

# change credentials!
user = 'username'
pass = 'password'

# User-Agent: required for del.icio.us api
agent = 'del.icio.us backup v0.2'
schema = <<EOF
create table bookmarks (
    hash char(32) primary key,
    url varchar(1024),
    title varchar(1024),
    note varchar(2048),
    time timestamp
);
create table tags (hash char(32), tag varchar(1024));
create index ix_tags_hash on tags (hash);
create index ix_tags_tag on tags (tag);
EOF
insert_url = 'insert into bookmarks (hash, url, title, note, time) ' +
             'values (?, ?, ?, ?, ?);'
insert_tag = 'insert into tags (hash, tag) values (?, ?);'

http = Net::HTTP.new('api.del.icio.us', 443)
http.use_ssl = true
xml = http.start { |http|
    req = Net::HTTP::Get.new('/v1/posts/all', {'User-Agent' => agent})
    req.basic_auth(user, pass)
    http.request(req).body
}

db_name = ARGV[0] || Time.now.strftime("%Y-%m-%d.db")
SQLite3::Database.open(db_name).transaction { |db|
    db.execute_batch(schema)
    db.prepare(insert_url) { |url_stmt|
        db.prepare(insert_tag) { |tag_stmt|
            REXML::Document.new(xml).elements.each('posts/post') { |el|
                url_stmt.execute(el.attributes['hash'],
                    el.attributes['href'], el.attributes['description'],
                    el.attributes['extended'], el.attributes['time'])
                el.attributes['tag'].split(' ').each { |tag|
                    tag_stmt.execute(el.attributes['hash'], tag)
                }
            }
        }
    }
}
                
                    I try to dump my data out of a website. But it block my python's urllib bot. I tried other pure python libray but without success. So, I try a quicker route using IE automation.
For general automation you should try 'watsup'. For IE, there is PAMIE.
>>> from cPAMIE import PAMIE
>>> ie = PAMIE()
>>> ie.Navigate('www.google.com')   # go to google
>>> ie.SetTextBox('Python','q',0)       # search for python
>>> ie.ClickButton('btnG')               # Go!

>>> src = ie.pageText()                 # here it is
>>> cookie = ie.GetCookie()          # Ok, cookie is yummy

PAMIE is actually aimed for more of the testing work.
But in this case it serves me fine.
http://pamie.sourceforge.net/                
                    If you're having trouble with FastCGI (as I was), this Perl script can help you see if the problem is at the Apache end or the app end. In my case I'd simply not deleted the old Ruby sessions for my Rails app when switching from CGI to FCGI ;-) This script proved my Apache wasn't broken, at least.

#!/usr/bin/perl

use FCGI;

$cnt = 0;

while (FCGI::accept() >= 0)
{
   print ("Content-type: text/html\r\n\r\n");
   print ("<head>\n<title>FastCGI Demo Page (perl)</title>\n</head>\n");
   print  ("<h1>FastCGI Demo Page (perl)</h1>\n");
   print ("This is coming from a FastCGI server.\n<BR>\n");
   print ("Running on <EM>$ENV{USER}</EM> to <EM>$ENV{REMOTE_HOST}</EM>\n<BR>\n");
    $cnt++;
   print ("This is connection number $cnt\n");
}
                
                    I use YAML heavily to specify basic configuration options for my applications.  Additionally, I have bash scripts to download and parse currency and weather information, which all gets stored as YAML for ease of use.

This is how I parse the YAML into my applications:

require 'yaml'

class ApplicationController < ActionController::Base
  before_filter :configure_app

  # Your code here...

  def configure_app
    @config = YAML::load(File.open("#{RAILS_ROOT}/config/config.yml"))
  end
end
                
                    
select dist, count(dist) 
from (select round(pow(10,ceil(log(10,distance+0.0001)))) 
           as dist from cust\_with\_dist) as foo
group by dist

This gives
 dist | count
------+-------
    0 |   142
   10 |   235
  100 |   111
 1000 |     7
                
                    Here's a great wrapper class for include_once & require_once. Picked this one up from Pure PHP (http://www.pure-php.de/node/22).

<strong>Usage:</strong>
<?php
require("includeWrapper.class.php");
includeWrapper::includeOnce("test.php");
includeWrapper::includeOnce("test.php");
includeWrapper::includeOnce("test.php");
print_r(includeWrapper::getPaths());
?>

<strong>Output:</strong>
Array
(
[test.php] => 1
)

<strong>Source Code:</strong>
<?php
class includeWrapper{
    public static $paths = array();
    public static function includeOnce($path_file){
        if(!isset(self::$paths[$path_file])){
            include($path_file);
            self::$paths[$path_file] = true;
        }
    }
    public static function requireOnce($path_file){
        if(!isset(self::$paths[$path_file])){
            require($path_file);
            self::$paths[$path_file] = true;
        }
    }
    // just for testing
    public static function getPaths(){
        return self::$paths;
    }
}
?>
                
                    Found via the RoR wiki. Solution courtesy of Jakob S.

Assuming a schema like such:

create table node (
  id integer primary key,
  name varchar(32)
);

create table nodelink (
  source_id integer,
  destination_id integer
);

A Rails association like this would supposedly do the job:

has_and_belongs_to_many :nodes, :join_table => 'nodelink', :association_foreign_key => 'destination_id', :foreign_key => 'source_id'
                
                    You can simulate the MySQL group_concat aggregate function using the SQLite/Ruby create_aggregate method (documented here
http://docs.jamisbuck.org/read/chapter/17#page77 ), as follows:

require 'sqlite3'

db = SQLite3::Database.open("test.db")

db.create_aggregate("group_concat", 1) do
  step do |func, value|
    if String(func[:concat]).empty? then
      func[:concat] = String(value)
    else
      func[:concat] = String(func[:concat]) + " " + String(value)
    end
  end

  finalize do |func|
    func.result = func[:concat]
  end
end

db.execute "select grouper, group_concat(name) from tests group by grouper" do |row|
  p row
end

db.close

Test with a SQLite3 database containing the following:

CREATE TABLE tests (id integer primary key, name test, grouper text);
INSERT INTO "tests" VALUES(1, 'Foo', '1');
INSERT INTO "tests" VALUES(2, 'Bar', '1');
INSERT INTO "tests" VALUES(3, 'ABC', '2');
INSERT INTO "tests" VALUES(4, 'DEF', '2');
INSERT INTO "tests" VALUES(5, 'GHI', '2');
INSERT INTO "tests" VALUES(6, 123, '3');

Gives these results:

["1", "Foo Bar"]
["2", "ABC DEF GHI"]
["3", "123"]
                
                    Found this on the #rubyonrails irc log here http://www.loglibrary.com/show_page/view/62?Multiplier=60&Interval=18&StartTime=1115561823 - credit goes to <tufty>.

Problem: you want to get at the underlying database connection object, e.g. so you can do something like create_function or create_aggregate on a SQLite database.  You try ActiveRecord::Base.connection but that just gives you the Rails connection adapter, not the underlying database connection.

Solution:

db = ActiveRecord::Base.connection.instance_variable_get(:@connection)
db.create_aggregate("xxx", 1) do
etc.