Monday, April 23, 2007

Some useful things to remember about the gcc compiler :MIT tips

BACKGROUND: THE COMPILATION PROCESS:



Before getting into the options, a little background. When you
compile a program as we've been doing, "gcc" checks the source code
for errors and creates a binary object file of that code (if no errors
exist). It then calls the linker to link your code's object file with
other pre-compiled object files residing in libraries. These linked
object binaries are saved as your newly compiled program. The options
to "gcc" dictate the way in which this process is performed. For example,
you could tell "gcc" to just create the object file and skip the
linking, such as when developing large programs or building your own
libraries -- how to do this is outside the scope of this class.


All of the options of the gcc command are explained in the
"man" pages. Type "man gcc" at the Athena prompt and
the (huge) list of options will be displayed.



OPTIONS IMPORTANT IN 10.001:



The gcc options that are important for 10.001 are as follows:



    -Wall


    -ansi


    -pedantic


    -I{directory_name}


    -L{directory_name}


    -l{library}


    -o{file_name}



where


    {directory_name} would be replaced by an Athena directory name,


    {library} denotes a library file as explained below, and


    {file_name} would be replaced by a valid Unix filename.



Examples of the options "-I", and "-L" are given below in the
description of "gcnr." The description gives an example of "-l", too.
I explain "-o" and "-l" here.



-Wall
tells the compiler to implement 'all' Warning options.
Warnings are diagnostic messages that report constructions which
are not inherently erroneous but which are risky or suggest
there may have been an error. Very useful for debugging code.
-ansi
tells the compiler to implement the ANSI language option.
This turns off certain "features" of GCC which are incompatible
with the ANSI standard.
-pedantic
used in conjunction with -ansi, this tells the compiler to
be adhere strictly to the ANSI standard, rejecting any code which
is not compliant.
-o
tells the compiler to save the compiled
program under the name . So, typing
"gcc myfile.c -o myfile.x" will take the source code
of file "myfile.c" and create program "myfile.x" rather
than the default program "a.out".


-l
tells the linker to search a standard list of directories
for the library, which is actually a file named
"lib.a". The linker then uses this file as if
it had been specified precisely by name.
The directories searched include several standard system
directories plus any that you specify with "-L".


EXAMPLES FROM gcnr:



An example of using the above options is the little file named "gcnr",
which we use to compile programs that use Numerical Recipes in C
functions. The file "gcnr" is a one-line Unix script that issues the
following command:



    gcc -Wall -ansi -pedantic -I/mit/recipes/src/recipes_c-ansi/include
    -L/mit/recipes/`machtype`lib $* -lrecipes_c-ansi -lm |& grep -v nrutil


(Note that the above is a long command and may word wrap in your browser.)
The numbers in this list mark each part of the command:


  1. /mit/cygnus/`machtype`bin/gcc
  2. -I/mit/recipes/src/recipes_c-ansi/include
  3. -L/mit/recipes/`machtype`lib
  4. $*
  5. -lrecipes_c-ansi
  6. -lm
  7. |& grep -v nrutil

where
  1. Tells Unix where to find the gcc program (`machtype`
    is needed to make sure that the correct version of gcc
    is used for the machine type you are using).

  2. Tells gcc that it should look in the Num. Rec. in C
    include directory to find nr.h and nrutil.h when it
    encounters #include "nr.h" and #include "nrutil.h" in your code.

  3. Tells gcc to look in the Num. Rec. in C library directory to
    find the pre-compiled binaries for all Num. Rec. in C
    functions used in your code.

  4. Is the Unix script notation for the command line argument to
    the script. In other words, when you type "gcnr myfile.c",
    "myfile.c" will be substituted in place of "$*".

  5. Specifies the name of the library of pre-compiled binaries for
    all Num. Rec. in C functions. It resides in the directory mentioned
    under (4). This option tells the linker to link in the
    Num. Rec. in C functions when building your program.

  6. Tells the linker to link in the math library. The name of the math
    library (i.e., file of pre-compiled binaries for all math functions)
    is "libm.a". (See explanation of "-l" above.)

  7. Pushes the output through a parsing program called grep. This looks
    through the output for lines containing "nrutil" and ignores them.
    This removes a number of error messages that do not prevent the
    executable from functioning.


CONCLUSION:


There are MANY options to gcc which can be used to make programming
easier and faster. These are only a few of the more common ones. As
mentioned above, the man pages for gcc list them all.



Powered by ScribeFire.

Google Print Hack





Hacking Google Print || kuro5hin.org




Introduction

Many people are interested in how Google works, and Google is mostly interested in keeping it a secret. I'm going to tell you a few things I've learned about Google by playing around with their software . It's not terribly advanced, but I think it's interesting nonetheless. The first thing I will cover is Google's cookie, and then I will explain how I used this information to exploit Google Print.

Google's Cookie

Most web browsers allow small text files, called cookies, to be stored on behalf of web servers ... this allows a persistent state to be associated with a user. After a cookie is created, it will be sent back to the web server every time you request a page (but only when you request a page from the server that originally requested the cookie). For example, when you set your SafeSearch preferences on the Google web site it stores your choice in the Google cookie. Then, whenever you request a page from Google it can see what you set your preference to earlier and use it without having to ask you again. If you delete your cookie, you'll just get a new cookie the next time you visit ... but you'll have to set your preference again. Pretty useful, huh?

Google does some more interesting things with its cookie, though. Some of them are hard to figure out. The first thing to notice is that your cookie will store some preferences locally, like SafeSearch, because Google probably doesn't care if you see that information (it won't bother you to see that they are storing your preferences). Otherwise, they probably have a server side system that uses the following characteristics of the cookie to store more .. ahem .. personalized information about you.

Here is an example of a Google cookie:

GPREF=ID=26b2149fe108b391:TM=1109736400:LM=1109736400:S=pbbDWyL8tVmJrILc

You can see that after "GPREF=", there are name-value pairs ID, TM, LM, and S (separated by colons). In this case, our ID is 26b2149fe108b391. This is a (hopefully) unique ID, and it is most likely generated randomly. Google probably doesn't worry about "collisions" (two users getting the same ID) because this is a 16-digit hexadecimal number, and there are 16^16 = 18446744073709551616 = 18.44674 x 10^18 possible IDs that could be assigned. Even if everyone on the planet used Google, the chance of collision would be very low. Google's cookie has an expiration date of January 17, 2038. Essentially, unless you purposely clear your cookies, format your hard drive, etc. this means it will be with you for a very long time.

The TM value is a timestamp of the moment (to the second) that Google generated your cookie. Here it is 1109736400, measured in seconds since January 1, 1970, or March 1, 2005 at 10:06:40 PM (CST).

LM seems unimportant because it is a timestamp of when the user last changed their preferences. Many other name-value pairs can appear, but the only others that I have seen represent more preferences. Having the unique ID means they are most definitely storing *something* on the server side, but don't worry it's probably only analyzed in aggregate unless you are one of Sergey's ex-girlfriends :-p.

Now, S is the most interesting value in the cookie. Some have hypothesized that it is a checksum of some sort. It could be a hash, for instance. In my experience, the signature only varies with different ID and/or TM values. Thus, Google is assured that THEY generated the cookie at a given time by doing a simple calculation of the hash. But relying on a pure hash would be security through obscurity, i.e. Google would basically be relying on the secrecy of the hash function. Instead, I think that Google probably uses a digital signature algorithm of some kind to generate it. So, maybe S stands for signature. It appears that the signature is 16 characters long, case-sensitive, and alpha-numeric only, giving (10+26+26)^16 possibilities or roughly the equivalent of a 93-bit hash (not incredibly strong by today's standards, but definitely a good chunk of hash). I tried my luck at guessing a hash function and mapping parts to base 62 numbers, but I just don't think that they are stupid enough to do it that way. Sucks for me, because I'm no Bruce Schneier when it comes to cryptography. My instinct is that an attack against the signature would be futile.

Now, the payoff. Well, after another explanation that is :) What are some reasons that Google needs to know you by an ID and when your cookie was created?

Google Print

Google Print URLs are of the form:

http://print.google.com/print?id=VvBRboW2icUC&pg=1&sig=hoLj_9Ot12vG6mSjZ vK547vbP3E

Anything look familiar here? Another signature! Maybe this one is generated in a similar manner (then again, maybe not ... they are probably different teams). The ID in the URL points to the book that you are viewing, and PG points to the page number. Now click the "Next Page" arrow. You'll get a URL like:

http://print.google.com/print?id=VvBRboW2icUC&lpg=1&pg=2&sig=gBBbI6T 0FzHxgVeJJQKQqmZ_MNk

The signature changes when you change pages, and LPG points to the page you started from! Eventually, you will not be able to advance through the pages any more. Google wants to limit you on the amount of pages you can scroll just so you can't read an entire book for free (that would make the publishers very unhappy, and here's my sad face for it :(). Try removing LPG and going to the resulting URL. You'll get a "page not found". So, apparently, the signature depends on the page you entered on, the page you are at, and the book you are viewing. This allows Google to impose their "page lookahead/lookbehind" limit.

You may see a search box on the side of the Google Print page to search within the book. Funny enough, you can use this to search for page numbers and skip through parts of the book. However, it will eventually hit a hard limit based on your unique ID ... i.e. you've viewed too many pages overall in this book; nothing to see here, please move along. Google probably already knows you can skip around pages like this because the search box doesn't appear unless your cookie is 24 hours old or more! Try it, if you have the search box now delete your cookie and refresh the page. The box will disappear! If you saved some of the URLs for the search results of the search-in-book feature, they will also not work! Wait 24 hours or so and try again. Now it works. Here's where the timestamped cookie comes to play. This way, if a user hits the hard limit they cannot clear their cookie and come back instantaneously to leech more pages.

So recently I wrote some software to grab and store up a bunch of cookies, keep them for more than 24 hours, and then automate searching for pages by this method. If I wanted to view page 100, the software would search for it and attempt to extract the image with a regular expression. If that doesn't work, it will search for page 99 and extract the "next page" link to get to page 100. It will continue doing this for page 101, 98, and 102 until it finds the correct page. Whenever a cookie would hit the hard limit, I'd replace it with a new cookie from the queue. By grabbing the "next" and "previous" links automatically in this "inductive" fashion and using the search for skipping, I could view an entire book on Google Print with one click every time. I later modified the software to spit out a PDF of the book. I used simple components like GoogleCookie (cookie with accessible properties), GoogleCookieOven (queue with "baking time", i.e. it only pops when the head of the queue is old enough to get the ability to search), and GoogleCookieBaker (thread that keeps the oven full of baking cookies by querying Google for new ones when the number drops below a certain threshold). Theoretically, if you set the cookie limit to a high enough number, the new cookies being fed in will have aged enough by the time you need them. This is a lot simpler than breaking an unknown digital signature algorithm, but of course that solution *would* be a lot more elegant. Oh well.

I sent a link to this software (web-based) to Google, and I actually got a response! I think they actually changed up some things because the software broke now and then, but it seemed to work consistently after many algorithm improvements. 




Powered by ScribeFire.