Tuesday, August 28, 2012

New Terminal Tab in Mac OS X

I haven't really used this for anything, but it seems that interfacing with the GUI through Command Line is a worthy skill to learn.

osascript -e 'tell application "Terminal" to activate' -e 'tell application "System Events" to tell process "Terminal" to keystroke "t" using command down'

In Mac OS X, it will open a new Terminal tab. More specifically, it will send the key combination "Command-T" to the application called "Terminal". I wonder if Terminal needs to be open for this to work.

Anywho, I think I'll come back to this later! And thanks to these commenting dudes:

http://stackoverflow.com/questions/7171725/open-new-terminal-tab-from-command-line-mac-os-x

Wednesday, August 22, 2012

bc threading

bash allows integer arithmetic, but no floating-point arithmetic. It is not uncommon to want to perform floating-point arithmetic on quick 'n dirty command-line scripts. I needed this a few days ago, and I found the handy bc command, which evaluates arbitrary-precision arithmetic and returns it as a neat shell string:

http://www.linuxquestions.org/questions/programming-9/decimal-numbers-in-bash-script-variables-380924/

Later on, I realized that the scripts that included the bc command were taking just as long or longer to run on a few hundred lines of text than a speech-recognition engine took to decode a half-hour audio file (yes, that's a lot. My script was sloooooooow).

I realized then that bc is a full-on interpreter, and that the overhead of starting it up and shutting it down multiple times per text line was awfully time-consuming.

So I googled around a little bit more, and I found a pretty neat way to deal with this problem: Open up bc on a separate process, and communicate with it through FIFOs! It's not straightforward, but it's certainly ingenious (and oh-just-ALMOST-so-very-clean). Most importantly, it speeds up my process a THOUSAND-FOLD!! Or by a lot, anyway.

It's not a one-line procedure - it requires preemptive initialization and some clean-up afterwards. Basically,
  1. Create two temporary FIFO files, one for input, one for output.
  2. Assign numbered streams to the FIFOs.
  3. Start up a background bc process that inputs from one stream, and outputs to the other. Perform the following two steps as many times as required:
  4. Send bc commands to the input stream.
  5. Read the bc result from the output stream.
  6. When you're done playing with bc, send it a "quit" command.
  7. Close both streams.
And you're done!! It'll be soo much faster! Smart guy who wrote this for all of us to usefully utilize in such situations:

Thanks dude!

As he mentions, this nifty little trick can be used with any other interpreters, like perl, php, or python.

For future reference and to avoid future link breakage, I'm copy-pasting his code here:

#!/bin/bash
##
# Make some fifos we can connect to.
##
mkfifo /tmp/bc.in
mkfifo /tmp/bc.out
##
# Attach file descriptors to those fifos.
##
exec 7<>/tmp/bc.in
exec 8<>/tmp/bc.out

##
# Start a single bc process in background and connect our fds to
# it's stdin, stdout, stderr we can use to do all of our math. We can
# do this because bc is an interpreter, with it's own shell.  Using 
# these same methods we can do inline perl/php/python, etc in 
# bash.
##
bc 0<&7 1>&8 2>&8 &

while read numerator denominator
do
  echo "scale=5; $numerator / $denominator" >&7
  read quotient <&8
  # bc errors have ':' in them somewhere.
  if [ "$quotient" != "${quotient//:/}" ; then
      echo "Error: $quotient"
  else
      echo "Answer of $numerator / $denominator is $quotient."
  fi
done

##
# Shutdown our bc process
##
echo "quit" >&7
##
# Close down our file descriptors.
##
exec 7>&-
exec 7<&-
exec 8>&-
exec 8<&-

Monday, August 13, 2012

vi/bash Miscellanea

Today I discovered and exercised a bunch of great things about vi and bash. To celebrate, I will write the most memorable ones here. First vi:
  1. :% - Indicates to execute a command across the range of all lines. It has helped me greatly in my quick file-wide substitutions.
  2. Recording commands: Press q, then any other letter that will identify the custom command, like a. Then perform the exact sequence of commands, insertions, deletions, modifications you want to record, then press q again. Now you have a recorded command!! Press @a (or whichever letter you chose as the command identifier) to execute this sequence of commands again. Or do it many times with N@a!
  3. You can cut/copy/paste not only lines but words and characters in vi. For example, cut the next 7 characters with 7d[RIGHTARROW]. Or copy the current word with yw. And if you have less than entire lines selected, pasting stays in the same line (I used to only cut/copy/paste entire lines in vi).
  4. Redo with Ctrl-r (for those hasty undos!)
  5. Awesome little "up-arrow"-style previous-command recoverers. To see the last vi command you executed: ":[UP-ARROW]". (And navigate with up-arrow and down-arrow, like in bash or in MATLAB). To see the last search string you looked for: "/[UP-ARROW]". To see a list of possible undo's in your file, execute ":undolist". (I haven't actually tried that last one, but it seems too useful to not write).
And now for bash:

  1. I found a way to exercise that the length of a string variable can be obtained through ${#f}, and that the length of an array can be obtained through ${#f[@]}.
  2. I also exercised with bash arrays: define -a arrVariable=($(ls -all)). Notice the parentheses around whatever you're trying to initialize the array with. I believe the string is split and nicely put into the array using the default IFS (internal file separator) delimiter.
  3. I also found an awesome little tool called bc that makes up for the lack of non-integer arithmetic in bash. Basically, this command $(bc <<< "scale=9; $x/5.7") outputs the result of whatever x was divided by 5.7 with 9-significant-digit precision (I think). It's not exactly a built-in solution, but I think it complements bash's capabilities very appropriately. It really hits the spot. I see it like an adjunct floating-point co-processor for bash. It's great! (Notice the <<<, it'll be explained soon).
  4. I also discovered you can identify streams in this manner: 3 < filename. I'm still not entirely certain how input streams are handled, but at least now I know that I can identify them neatly inside bash scripts.
  5. Also, I learned about the triple redirector operator!! (<<<). It's great! For example, before when I wanted to do something like:
while read line
do
    blah
    blah
    moreBlah
done < filename

filename HAD to be an actual filename - it could not be the ad-hoc result of a quick command obtained on the fly. So sometimes weird hacks had to be made to adapt a quick command result into a file stream. But today I learned that if you replace the last line by 

done <<< $(command here, such as ls -all)

it will now work! I often wondered how to bridge that gap between file streams and ad-hoc input streams. Well, this helps a lot to make that happen!

And... that's it. I'm gonna go home because it's past 11PM and I'm still in the office. Bye...

Newlining

As a budding vi, bash, sed, and general Unix-command-line power user, I'm often pleased at the magical powers that these awesome tools provide. There are a few annoyances inherent to this more minimal environment, though, and one of them has been newline manipulation.

The newline is this mystical, elusive, not-always-obvious-on-terminal, cross-platform-incompatible character that permeates a darn large majority of text files out there. It is a natural and almost global exception case for terminal display and for most text-based manipulation commands. As such, I often find myself wanting to collapse or insert newlines, unable to do so in command line, and unelegantly reverting to TextEdit or other GUIs temporarily just to get the job done. I've always trusted that command line provides appropriate tools to deal with newlines, but I have not set out to learn them just yet.

But today I learned one pretty nifty way of dealing with it in vi. I was trying to convert this command I created ad-hoc'ly:


declare -a f=$(cat robots2002_segs.txt); n=9; echo ${#f}; while (( n < ${#f[@]} )); do echo ${f[$n]}; echo ${f[ (( $n + 3 )) ]}; n=$((n+4)); done

into a nicely delineated:

declare -a f=$(cat robots2002_segs.txt)
n=9
echo ${#f}
while (( n < ${#f[@]} ))
        do echo ${f[$n]}
        echo ${f[ (( $n + 3 )) ]}
        n=$((n+4))
done


Seems easy enough, right? Replace the "; " strings with a newline, and I'm (almost) done! But alas, how do I tell vi's substitute (:s) command that I want to represent the newline character, when the key I use to represent it, [ENTER], only tells vi to execute the command?

Well, today I found out! Command down here:

:%s/; /CtrlVCtrlM/g

Break-down:

  • : begins the vi command
  • % tells it to apply this command on all lines of the file (unnecessary in this example)
  • s stands for substitute-command
  • ; / text to be replaced
  • CtrlvCtrlM The magic! Vi's representation for a newline, as far as I can tell.
  • /g Do it as many times as you can.


And done! And then I spent 20 minutes writing the blog post, when doing the whole thing manually would've taken me no more than 20 seconds. Oh, the joys of obsessive optimization!

Credits: (Thanks Mr. Boothe!)
http://www.computing.net/answers/unix/vi-ex-substitute-with-newline/6485.html

Sunday, August 12, 2012

File splitting

Have you ever wanted to email a large file and discovered that it was JUST a bit larger than the maximum allowed file size? Have you ever wanted to copy a 6GB file but only had a 2GB USB flash drive to do it? Have you ever wanted to transfer a 4.7GB DVD image file through your free 2GB Dropbox account, and even considered paying the $5 monthly rate to get the overkill of 50GB limit?

I have. Well, not the Dropbox thing - I got 16GB free because I invited everyone I could when it started :). Even so, one time I couldn't move a 20GB file through.

And just now I'm asking a non-techie to send me a file several GB in size, but her Dropbox account is not large enough. Mine is, but I'm not giving her my password. Oh no! What to do?

So I googled a bit, and sure enough, Unix can solve that problem too! How, you may ask? With split!!

As described by man split:


SYNOPSIS
     split [-a suffix_length] [-b byte_count[k|m]] [-l line_count] [-p pattern] [file [name]]

DESCRIPTION
     The split utility reads the given file and breaks it up into files of 1000 lines each.  If file is a single dash (`-') or absent, split reads from the standard input.


So, say you have a file called IMG_4625.JPG that is around 3MB large:

3009327 Aug 12 00:05 IMG_4625.JPG

and for some reason you want to split it into files only 500K each. You can do:

split -b 500k IMG_4625.JPG 


This will create the following files:


512000 Aug 12 00:09 xaa
512000 Aug 12 00:09 xab
512000 Aug 12 00:09 xac
512000 Aug 12 00:09 xad
512000 Aug 12 00:09 xae
449327 Aug 12 00:09 xaf

When they are in this form, you can transfer them however you want. To put them back together, you just say:

cat x* > IMG_4625.JPG

And voilĂ , you have your file back! Easy, huh? Extra options are available through the man file.

Credits (thank you!):
http://stackoverflow.com/questions/2016894/easy-way-to-split-a-large-text-file

Thursday, August 2, 2012

Command Line Copy/Paste

At times I find myself grabbing and copying some text from a GUI editor, and wanting to save it all into a file somewhere for command line access and manipulation (e.g. log files). My usual approach used to be one of the following (numbers in parentheses are the number of keystrokes used):
  1. Open Spotlight(2); Open TextEdit(2+O(1)+1); [PASTE](2); [SAVE](2); type in the new filename(N); Select WITH THE MOUSE to save it as a plain text file and NOT as an RTF as is TextEdit's freaking default setting, and click SAVE(3 mouse seeks, 3 clicks!); Close TextEdit(2 or 4, I close the window first with Command-W before Command-Q to avoid the file re-appearing when I open TextEdit the next time); Move this file to where I want it through Terminal. (O(N)). At least 15 keystrokes AND 3 mouse seek+clicks - it's a freaking long process for a task so punctual.
  2. type "vi <filename.txt>" in Terminal(4+N); Press "i" to enter INSERT mode(1); [PASTE] the contents(2); ESC, Save and Exit with ":wq"(6). 13+N keystrokes in total, much better considering the mouse is never needed.
The second option sounds way better, although sometimes when I copy some text many MB long, vi dutifully attempts to load it all up into memory and display it for me on the screen. This can take a while, plus for especially lengthy texts, it encourages dreaded pagination to kick in.

So I Googled around for a bit and found two shiny little gems that will redeem my inter-Terminal-GUI interfaces (at least on my Mac OS X):
  • pbcopy < inputStream
  • pbpaste > outputStream
As straightforward as they should be. 8+N keystrokes, and no further user interface required. Would be absolutely perfect to automate GUI tasks, if I knew how to handle GUI's from the command line in detail.

Credits where credit's due: