Capturing output of terminal on Unix

Terminals do not have a great scroll length and even though you can increase the number of lines that can be scrolled up, it is always useful to have the output captured in a file so that you can search on it. Here are few ways

1. Redirection

cat foo.txt > bar.txt

Outout of foo.txt is captured in bar.txt

2. Redirection with "append"

ls >> bar.txt

This will append the output of "ls" to bar.txt ( Instead of overwriting it)

3. "tee"
Redirect and redirect with append will not display the content on console and hence if you want to see the content and also redirect it to a file , use "tee"

ls | tee bar.txt

this will display the contents of ls on terminal and also redirect it to bar.txt

4. output to multiple files

ls | tee a.txt b.txt c.txt

This will output contents of ls to a.txt, b.txt, c.txt

5. Outout all the terminal interaction to a file

script -f out.txt

Converting UTF-16 to UTF-8

I was running grep on an xml file (big one) and realized that it did not match the pattern I was expecting the the file. I then did a "head" from the file to see a pattern that is present in it. Grepping on that too returned nothing telling me that grep is somehow not working on this file.
"head" was returning some junk characters at the begining of the file - This idicated that the file does not have a normal text encoding and probably that is the reason why grep is not working on it.
Reading the XML header proved that, since it was "UTF-18", grep was not running over it. I had to convert the file to UTF-8 which is more text friendly if the content is only English.

iconv --from-code UTF-16 --to-code UTF-8 input_file.xml > output_file.xml

screen

when to I use screen :

* When "nohup" is just not enough
* When the power goes off and you are left in the middle of nowhere
* When I need to log on to 10 different terminals each day
* When I do not want to enter 10 different username passwords each day

"screen" is a terrific unix utility that allows you to start a terminal without having to worry about you job being interreputed mid-way. It is better than nohup in that respect nohup only allows unix commands to be started in a "nohup" fashion. If for instance you are working with an interactive program like sqlplus and your internet connection happen to break, you will loose your context and any transactions that you were working on.

It essentially allows you following :
* Multiple sessions in a single ssh window (screen multiplexing)
* Ability to "resume" a session in the state you left it - irrespective of power outage, internet connection termination etc

Here is how you use it :

1. Make sure it is installed on your system by typing "screen --help"
2. If it is not installed, you can install it by "yum install screen" on a fedora box.
3. start a screen session by typing "screen" and you should be in a screen session.
4. If you are disconnected or closed the terminal, type "screen -r -d" to reattach to the session

Here are some of the short-cuts available (All after CTRL+A)

c : creates a new
n : next screen window
p : previous screen window
a : toggle between two screen windows
[0-9] : Go to a specific screen window identified by a number
A : Set the name of a screen window
" : List all the existing screen
ESC : Scroll up or down on a screen

Man on the moon (well almost)

India becomes the 4th Nation to put a lunar probe on the moon ! I am proud to be the 4th country to do that(as against the alleged 3rd world country banter from Matthew Hayden who knows nothing more than being a sore loser)

pasting code in vim

:set paste

paste the code

:set nopaste

set pastetoggle=F3 in your vimrc file is an alternative.

Renaming Files in Linux

Type the following on commandline of a bash shell and change the logic as needed.

for i in `ls /home/ssinghal` ; do mv "$i" "$i.new" ; done

sudo without password

1. Log in to a unix box as root
2. Execute "visudo" to open sudo editor
3. Enter the following line
ALL=(ALL) NOPASSWD: ALL

Database Objects - Finding Dependencies

You come across tables in databases that are being populated by other pl/sql procs, triggers e.t.c what becomes difficult is to figure out which process is actually updating a table.

The following query will list objects that refer a table "TABLE_A" which belongs to a schema "SCHEMA_A"

select name,owner
from dba_dependencies
where referenced_name='TABLE_A' 
and referenced_owner='SCHEMA_A'

JQuery - Getting Started

// Makes an element visible which has the id 'mydiv'
$("#mydiv").show();
  
// Makes each odd tr element to have a blue backcolor
$("tr:odd").css("background-color", "blue");


// What if we want to make the header row have a different look ? Show Me
$("#results tr:first").css("background", "black");
$("#results tr:first").css("color", "white");
$("#results tr:first").css("font-weight", "bold");


// We also could have assigned a class name to the header tr element like this 
$("#results tr:first").addClass("header");


//All span that are inside a div element (may not be immediate child of the div) will be selected.
$("div span").css("border", "1px solid");

//Here is how we select the links that are descendent of span which are direct children of a div
$("div > span a")

//Here is how we can select a div with id mydiv 
$("div#mydiv")


// Here is how we select all links under div elements that have css class set to mycss
$("div.mycss a")

// if we want to find a link that points to http://www.google.com then we would write
$("a[href=http://www.google.com]")


// Modifications
$("body").prepend("");
$("div#mydialog").html("
Hi There!
");
$("#mytable").wrap("
");


// How to replace all hr elements (horizontal line) with a br element
$("
").replaceAll("hr");

Reference : http://www.shafqatahmed.com/2008/08/mybutton-font-f.html

tcpdump : capturing data

capture all traffic on port 80 for a specific machine

sudo tcpdump -A -s 0 -i eth0 dst or src host and port 80


Reference : http://code.google.com/support/bin/answer.py?answer=71567&topic=12022

Vim Movements - Some Common , Some Not so much

ESC + H Moves to the line at the top of the screen.
ESC + M Moves to the line in the middle of the screen.
ESC + L Moves to the line at the bottom of the screen.


[Ctrl-F] Move forwards one page.
[Ctrl-B] Move backwards one page.
[Ctrl-D] Move forwards by half a page.
[Ctrl-U] Move backwards by half a page.
[Ctrl-E] Display one more line at the bottom of the screen.
[Ctrl-Y] Display one more line at the top of the screen.

Reference : http://en.wikibooks.org/wiki/Learning_the_vi_editor/vi_Reference#Movement

Old Firefox Versions

http://releases.mozilla.org/pub/mozilla.org/firefox/releases/http://releases.mozilla.org/pub/mozilla.org/firefox/releases/

nmap - portscan

scan all the ports in a given range on a specific machine

nmap -p1-10000 127.0.0.1

The above command will scan host 127.0.0.1(localhost) to see if any of the ports from range 1 to 10000 is up.

Oracle - Enabling Parallelism

If your database server is running on multiple processor system. You can make all your processors conspire to run a query. You might want to do this when the query is known to take a long time and there is no way it would return in sane time. At the same time, be careful to run this on production systems as it may hamper performance of the database.

select /*+ parallel(x 4) */ count(1)
from huge_table
where huge_table.x = 'somevalue'

vim : ignoring white spaces in vimdiff

In your .vimrc file :
set diffopt+=iwhite


http://vim.wikia.com/wiki/Ignore_white_space_in_vimdiff

Appending to a register in vim

For example, you want to collect a sequence of lines into the a register. Yank the first line with:

"aY

Now move to the second line, and type:

"AY

Repeat this command for all lines. The a register now contains all those lines, in the order you yanked them.



Reference : Vim manual

Bind Parameters in a complex query

Oracle Data dictionary tip :

This is pretty useful when you are running a huge pl/sql code or a complex query and you need to know what bind parameters are being used for a query. helps figure out the iteration you are in amongst many other.

select name, position, datatype, 
       value_string, value_anydata 
from V$SQL_BIND_CAPTURE where sql_id 
IN 
 ( select a.sql_id 
   from v$sql a where sql_id in 
    ( select sql_id 
      from v$session 
      where osuser = 'ssinghal' 
      and machine = 'my_machine_name 
      and status = 'ACTIVE'
     )
 ); 

Search Replace in all Vim Buffers


:bufdo %s/pattern/substitution/ge | update



bufdo – apply the following command to all buffers – see :help :bufdo.

%s – search and replace the whole file

g – change all instances in a line

e – avoid an error when the pattern doesn’t exist in a buffer

update – write the file only if changes were made

source : http://vim.wikia.com/wiki/Search_and_replace_in_all_buffers

Getting Started with Rails 2.0 on ruby

create a rails project called student with mysql as database (default is sqllite)

rails -d mysql student

vim config/database.yml and edit username/password if required

create the database if not already created through backend
rake db:create:all

Generate a scaffold for a student
ruby script/generate scaffold studen name:string address:string

edit db/migrate/001_create_students.rb to add more tables or columns to existing table. This step can be skipped.

Create the Database tables
rake db:migrate

Run server
ruby scripts\server

Access the project through a browser
http://localhost:3000/students

tada

vim - stop autoindent while pasting text

On a putty terminal, when you copy-paste code from other sources, it results in a complete mess, due to vim trying to format it again. To avoid this, type

:set pastetoggle=

Now you can use to toggle between paste mode (and no paste mode).

When in paste-mode auto indent will be turned off. This is very useful when pasting text that's already indented.

source:
http://amix.dk/blog/viewEntry/19083

vim - going back in time

supported since vim 7

" Go back 1 minute
:eariler 1m

g+ and g- can be used for redo and undo ( u and CTRL + r still applies)

Long Operations in Oracle

It is nice to know what queries are taking time to finish. The following sql figures out the "long operations" currently being processed


SELECT username,time_remaining,sofar/totalwork,message,target_desc
FROM gv$session_longops where sofar<totalwork
ORDER BY message


Apache Web Service Framework

http://incubator.apache.org/cxf/

vim - recording keystrokes and reusing

http://www.vim.org/tips/tip.php?tip_id=144

The most useful feature that I find in VIM is the "recording" feature (:help recording). I have used this to automatically insert function headers, re-indent lines, and convert some 34 source files into HTML.

This feature is most useful when you want to do some repeated jobs, which you cant do easily using ".". You can set about writing a function, define a mapping, etc, but then these things might take time. By recording, you can try out and find the actual keystrokes that does the job.

To start recording, press "q" in normal mode followed by any of "0-9a-z". This will start recording the keystrokes to the register you choose. You can also see the word "recording" in the status(?) line. You can start the key sequences that you want to record. You can go to insert mode and type if you want.

To stop recording, press "q" in the normal mode.

To playback your keystrokes, press "@" followed by the character you choose. Pressing "@@" will repeat the same again.

subversion : clean non-versioned files from svn source tree

svn status|egrep "^\?"|awk '{print $2}'|xargs rm -rf

ruby - installing from source

1. Download ruby.tar.gz from rubyforge

2. unzip and untar the file

3. Navigate to extracted folder

4. following steps ( like any other source compile on linux)

./configure
make
make install


This will install ruby .

5. To install an extention (say to install openssl on linux)

go to ruby-source/ext/open ssl and execute following steps

ruby extconf.rb
make
make install

unix find - seach based on permissions

All the files in the current directory that does not have a write permission for user and group

find . -not -perm -ug=w

All files owned by 'ssinghal'

find . -owner ssinghal

Code Search Engines

TODO: Edit this post to give a description on special features of each search engine :)

http://www.google.com/codesearch

http://www.koders.com

http://www.krugle.org

http://www.codase.com/

using sun.* packages in java

From : http://java.sun.com/products/jdk/faq/faq-sun-packages.html

The classes that Sun includes with the Java 2 SDK, Standard Edition, fall into package groups java.*, javax.*, org.* and sun.*. All but the sun.* packages are a standard part of the Java platform and will be supported into the future. In general, packages such as sun.*, that are outside of the Java platform, can be different across OS platforms (Solaris, Windows, Linux, Macintosh, etc.) and can change at any time without notice with SDK versions (1.2, 1.2.1, 1.2.3, etc). Programs that contain direct calls to the sun.* packages are not 100% Pure Java. In other words:

The java.*, javax.* and org.* packages documented in the Java 2 Platform Standard Edition API Specification make up the official, supported, public interface.
If a Java program directly calls only API in these packages, it will operate on all Java-compatible platforms, regardless of the underlying OS platform.

The sun.* packages are not part of the supported, public interface.
A Java program that directly calls into sun.* packages is not guaranteed to work on all Java-compatible platforms. In fact, such a program is not guaranteed to work even in future versions on the same platform.

For these reasons, there is no documentation available for the sun.* classes. Platform-independence is one of the great advantages of developing in the Java programming language. Furthermore, Sun and our licensees of Java technology are committed to maintaining backward compatibility of the APIs for future versions of the Java platform. (Except for code that relies on serious bugs that we later fix.) This means that once your program is written, the class files will work in future releases.

Each company that implements the Java platform will do so in their own private way. The classes in sun.* are present in the SDK to support the Sun implementation of the Java platform: the sun.* classes are what make the Java platform classes work "under the covers" for the Sun Java 2 SDK. These classes will not in general be present on another vendor's Java platform. If your Java program asks for a class "sun.package.Foo" by name, it may fail with ClassNotFoundError, and you will have lost a major advantage of developing in Java.

Technically, nothing prevents your program from calling into sun.* by name. From one release to another, these classes may be removed, or they may be moved from one package to another, and it's fairly likely that their interface (method names and signatures) will change. (From the Sun point of view, since we are committed to maintaining the Java platform, we need to be able to change sun.* to refine and enhance the platform.) In this case, even if you are willing to run only on the Sun implementation, you run the risk of a new version of the implementation breaking your program.

In general, writing java programs that rely on sun.* is risky: they are not portable, and are not supported.

Regular Expressions Across Multiple Lines

I was looking at this huge java file which had hundreds of SQL statements and thousands of java code interleaved like a spaghetti ! What I really needed was to analyze all the SQL statements in the file since they were all related to a single application which we were supposed to modify. I needed a mechanism to extract all the SQL statements from the file for sake of brevity.

Simple regular expressions using vim or sed did not work since they all deal with a single line, but each one of them provide a mechanism to match regular expression over multiple lines.

sed - the following command will extract all lines between a 'select' and the first ';' following that. Hence you would have printed all the sql queries from a file

sed -ne '/^select/,/;/p' SqlIntensiveFile.java

vim - using the 'g' command can achieve the same. For e.g. the following will delete all the sql queries from the file.

%g/select/,/;/d


Formatting Oracle Timestamp

create table temp (a timestamp)

insert into temp values(sysdate)

select to_char(a,'MM/DD/YYYY HH24:MI:SS:FF3') from temp

http://www.databasejournal.com/features/oracle/article.php/2234501

Oracle Instance Configuration Parameters

To find out the parameters that oracle is using for configuring an instance, logon to that instance using sqlplus and execute the following command.

sql> show parameters

This will display list of all oracle configuration parameters with their current values

Block Select in Vim

Go to the visual mode using "ctrl + v" instead of "v", then use the motion arrow keys *hjkl* to select the block you want.

Perform any copy/delete/edit function you would normally perform using vim ( y , d etc)!

GET POST PUT

When a browser requests a normal page from a server, it uses the "GET" method. This is the standard way to get back information from a server. The information itself may come from a static page, a CGI program, a server-side include page or any other source handled by the server. By definition it is safe for a browser to obtain a page by GET as many times as it likes - it will never cause any permanent action on the server (such as entering a product order).

To perform a permanent action on the server, the "POST" method is used. This method must be handled by a program or script, and the browser should not re-request a POST page without getting the user to confirm it. This POST method is used when a script or program requires a lot of form data input or when the request makes the server perform a real action such as entering an order.

The "PUT" method is similar to the POST method in that it can cause information to be updated on the server. The difference is that the POST method is normally handed a script which is explicitly named by the resource (that is, something that already exists), while a PUT request could be directed at a resource which does not (yet) exist. Another difference is that the POST method can be used in response to a form, while the PUT method can only contain a single data item. The PUT method is suited for publishing pages.

Reference :
http://www.apacheweek.com/features/put