Tuesday, December 04, 2007

Inside Google's Intranet


Have you wondered ever what do employees of the world's greatest enterprise stare at in the morning when they’ve arrived at the office? They might be looking at Moma, the name for the Google intranet. The meaning of the name of “Moma” is a mystery(it reminds you of the momma of yours - the mother company),but Moma’s mission is prominently displayed on its footer: “Organize Google’s information and make it accessible and useful to Googlers.” A “Googler,” as you may know, is what Google employees call themselves (they have other nicknames for specific roles; a noogler is a new Google employee, and so on ...)


Fig :Inside Moma's pages

"MOMA, Google's intranet, was designed by and for engineers and for the first couple of years, its home page was devoid of any aesthetic enhancements that didn't serve to provide information essential to the operation of Google. It was dense and messy and full of numbers that were hard to parse for the uninitiated, but high in nutritional value for the data hungry.

MOMA displayed latency times, popular search terms, traffic stats for Google-owned properties and, at the center of it all, a large graph with colored lines labeled with the names of Muppet characters.I am taking the accounts from another blog on the structure of the site for those interested.


The login dialog on Moma's homepage reads, “Many internal apps. One login page.”

On the top of the homepage, you’ll find the logo reading “Moma - Inside Google.” Next to it is a search box allowing you to find information from Moma in general, information on specific Google employees, information on availability of meeting rooms, building maps and more. You can choose to include secure content or not via a checkbox. Another checkbox offers you to use “Moma NEXT"(<--go to the link for a news piece for more info) for a more experimental variant of search results.

To the top right, there’s an option to switch to iMoma, an iGoogle-style tool prepared by the company which allows further customization of the intranet start page. This way, employees may be able to select their own news and service widgets of interest to be displayed when they log-in.

The actual content of the homepage in the picture is split up into 4 columns. To the left, there’s a “My Office” section, with information for employees and a way to choose your own office for more relevant links. It’s followed by the sections “Survival Kit” and “My shortcuts.” In the middle columns, news gadgets are headlined “Welcome to Google!,” “Communications,” “HR” (human resources), “Company Info” and “Internal Google news,” all in common soft shades of Google base colors. The right column is listing Google teams.

Ex-employee Doug Edwards mentioned how he came to take for granted everything was available on the intranet, “from the status of products in development to the number of employees at any point in the company’s history.If you are concerned with coworkers, there is also an employee search part on the system which gives you relevant information without compromising the confidentiality.

How employees access the system

A Google employee can log-in to the intranet from within the office, or with a so-called Virtual Private Network (VPN) connection. This connection comes pre-installed on laptops Google hands out, and can be reached via a desktop icon. A Google employee is required to authenticate their sign-in with account credentials(The talk of the town is that the authentication is based on LDAP Protocol)

From within a Google building, an employee may likely reach the intranet via the address corp.google.com. We previously found out Google additionally uses many sub-domains in their intranet, like album.corp.google.com, agency.corp.google.com, alien.corp.google.com, karma.corp.google.com, periscope.corp.google.com, pineapple.corp.google.com. You may also likely just enter e.g. “m” (which maps to “http://m” which is “http://m.corp.google.com”) to be taken to a service like your Gmail-powered email account.

Externally, like from a laptop at a conference – or if you’re one of the employees mainly working from home, as there are some – employees can access the VPN servers located on sites like Mountain View or Dublin, Ireland, with different hostnames each like man....ext.google.com or de....ext.google.com (we depleted part of the hostname).If you work in a team for a product, you may also get a prototypical version of the service. Special links to debug windows are offered to developers as well, one of them being opened in the screenshot. Google employees also get to see previews of completely unreleased tools, such as wiki service JotSpot (which is being integrated into Google Apps), or Platypus, the internal Gdrive client for file-sharing.For code reviews, Google created Mondrian, a “Perforce backend with some custom Google wrappers on top - once again it is based on creative commons : the sweetest of all licences.

Others , Welcome

But, Google doesn’t just use their own tools. For instance,many Google employees prefer social network Facebook to their own production, Orkut (e.g. some Google employees considered Orkut too spammy, or too buggy in the past).

If a Google employee encounters trouble with any Google tool, they can call their internal support hotline named “Tech Stop.” The hotline promises 24-hour availability. Numbers like +1 877... (last part depleted) are partly toll-free and partly with toll, and accessible from all over the world. Internally, a Google employee may also simply press 3-HELP (3-4357). Tech Stop centers aren’t just located in the US, but also in places like Hyderabad, India.


Big thanks to the article :What the Google Intranet Looks Like by Philipp Lenssen & Tony Ruscoe.Images reproduced from Wikimedia Commons - the free repository.Information in this article is in no way violating the confidentiality policies of any Enterprise.

Tuesday, October 09, 2007

One who doesn't understand the GPL

"This is the absolute reproduction of the famous column by Bennett McElwee on the Technology Blog "Semicolon" on the new venture of Microsoft to Open path and its real motives and flaws..All rights reserved by the author.We want to inform you that we are not against any trade practices of any corporation.Only for informative purposes of Open Source advocates"

Microsoft’s Shared Source Initiativ.e website contains some pretty "outrageous" falsehoods about free software licenses and the GNU General Public License (GPL). The Shared Source Initiative (SSI) allows certain third parties access to some of the Windows source code. It’s dismaying, but not entirely surprising, that the SSI website shows such a lack of understanding about the GPL.

I don’t mean to interpret this as Microsoft’s official position statement on the GPL, but it has been on their website for quite some time, on the pages for the well-publicised SSI. Also, there is no author’s byline, implying that the piece reflects Microsoft policy rather than the opinions of a Microsoft employee.

Microsoft’s Shared Source Initiative: Licensing Overview is the main offender. In a paragraph on free software licenses (including the GPL), they say

These licenses often are described as “viral,” because they purport to extend their own terms to, or to “infect,” software other than the licensed program, and even to software not yet in existence.

This is silly; commercial software licenses also “infect” “software not yet in existence”, but you don’t hear Microsoft bemoaning this fact. If I write a program incorporating a part of the Linux kernel, I am not allowed to distribute it under a commercial license now or in the future. But the same is true for a program incorporating a part of the Windows kernel. At least with the Linux-based code, I am allowed to release it under the GPL; as for the Windows-based program, I can’t release it at all.

This page goes on to a specific discussion of the GNU General Public License (GPL), and attempts to discuss three features of the license. Impressively, it gets all three wrong.

The GPL permits unlimited free use, modification, and redistribution of software and its source code, but imposes three key restrictions on every licensee:

  • If the licensee redistributes any code licensed under the GPL, it must guarantee availability of the code for the entire work for unlimited replication by anyone requesting it.

This is not true. GPL section 3 says that in general, the distributor has only to either distribute the source code with the program, or include a written offer to make the source code available for a period of three years. And it does not have to guarantee this to “anyone requesting it”, but only to licensees who have obtained their license from the distributor.

  • If the licensee redistributes GPL code, it may not charge a licensing fee or royalty, but may charge only for distribution costs.

Not true. This applies to source code only. The licensee is free to charge any fee for executable code.

  • If the licensee includes any GPL code in another program, the entire program becomes subject to the terms of the GPL.

Wrong again. This is Microsoft’s interpretation of the “viral” GPL, but they have it backwards. Here’s what the GPL actually says.

2. b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License.

4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. […]

This seems quite clear. Here’s an example of how it works.

  1. You write a program that includes (knowingly or otherwise) someone else’s GPL code.
  2. You distribute this program under a commercial license.
  3. Because you have failed to comply with clauses 2b and 4 of the GPL, your GPL rights are terminated. In other words, you now have no right to distribute the (copyrighted) GPL software.
  4. The copyright holder of the GPL code can sue you for copyright infringement.

The GPL license has not “infected” your program at all. All that has happened is that you have failed to comply with the terms of the GPL, so your license has been terminated. Exactly the same would happen if you distributed a program that incorporated source code from, say, Microsoft Word. Again, you may distribute your GPL code-based program under the terms of the GPL; you can’t legally distribute your Word-based program at all.

Microsoft may have been confused on this point by GPL section 6:

6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients’ exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License.

So if the licensee includes any GPL code in another program and distributes it to a third party, then the third party automatically gets a GPL license for the original GPL code (”the Program”), not the entire distributed program. In our example from above, the third party would automatically get a GPL license for the GPL code incorporated into your program, not your program itself.

Microsoft say that they do not oppose the GPL for use by individual developers, but “our concern is based on the health of the overall software ecosystem and the effect of the GPL on the process of sustained innovation.” They recommend relatively unrestricted BSD-style licenses instead, especially by researchers, because “use of unrestrictive open-source licenses by the research community preserves the ability of the private sector to develop basic research into useful and affordable software products for individual and business consumers.”

In other words, use of unrestrictive open-source licenses by the research community preserves the ability of Microsoft to make money from others’ research. Fair enough, but raising concerns about “the health of the overall software ecosystem” seems a little overblown. If Microsoft are unable to use somebody else’s research to develop products, then they’ll just have to do their own research — which they already do anyway. The same goes for other commercial software developers. This should actually increase innovation by forcing commercial developers to come up with their own original software, rather than simply build on somebody else’s.

(Rather than use the GPL, presumably Microsoft would prefer researchers to patent all their new software, so that Microsoft and other commercial developers would be free to pay license fees to the patent holders — or buy the patents outright. They don’t mention this in the SSI site, for some reason.)

At the end of the page, Microsoft say that developers should be free to choose licensing terms, but sensibly encourage caution: “The decision to use or produce open-source or free software, however, should be based on an informed understanding of the licenses that govern such software.” It looks as if Microsoft could do with some informed understanding themselves.

Wednesday, September 12, 2007

Subdomain Configuration in DNS

I am posting this as a quick tip for those who want to add a subdomain of some form in a DNS server having domains configured in it.This is usually handy for those system administrators who have to frequently add subdomains as part of hosting services.This is being narrated as a real experience of mine as recently i had to work over it.

The requirement was to add a subdomain (h***dmin.a**s.net) in the form ..net to the DNS gateway at the organisation.The domain has already been existing.I had prior knowledge of adding a domain in the "named.conf" file which is the main configuration file for the server.

The entry is given as following steps:

Step 1: In /etc/named.conf, find the domain file corresponding to the domain which we wish to configure the subdomain. Example for here, the domain in arys.net and the file is :
"/var/named/slaves/sec.arys.net" as evident from its entry in 'named.conf'


Step 2: Open the domain file and give the subdomain as a C or A entry.

This can be either as :

h***admin CNAME www (C Entry)

or

h***admin A

Friday, August 10, 2007

"March Linux"- Simple, Lean and Fast Distro

"March Linux" aims to be a Simple, Lean and Fast GNU/Linux distribution for the desktop which urges the users to learn linux instead of holding their hands at every step!

At the same time, it is usable and fully functional with almost every software that one might need being available. It is heavily based on Arch Linux








Some Statistics

  • The default desktop in March Linux is Fluxbox
  • The current version is 1.0 BETA, to start off with.
  • Currently, March is available only as a Live CD (built using the Linux Live scripts).
  • No installation option is available, as of now.
You can do the following right out of the box :
  • Listen to music and watch movies in a variety of formats, record sounds
  • View pictures and perform graphical editing
  • Connect to and browse internet, check emails, download via torrents, perform ftp transfers
  • Play a number of cool games, including a few DOS based ones
  • View and edit document files, create spreadsheets, view pdf-s, perform calculations, take notes
  • Do small programming in a number of languages with an IDE
  • …… and many more!!
March is designed to run amply well on a 128 mb RAM machine, PII or later (or equivalent). A 160 mb RAM is recommended for best performance! The apps it includes are mostly LnF (Lean and Fast), and the default desktop (fluxbox) is a minimalistic and lightweight one, which just doesn't get in your way!

Some Wisdom

We can say that March Linux is well suited for anyone who wants to use Linux! Even a relatively newbie can begin to "learn the ropes" on this distro, provided he/she has the will and resilience to learn!

On the other hand, if you are a windows fan-boy, and prefer being served everything on a platter to you rather than using your brains even to the least extent, then March Linux is NOT for you! March Linux tries its best to make learning linux as easy & enjoyable as possible, but really the onus is on you to actually benefit from it!

Links

Hear to some user comments at Arch Linux Forums.

Have a first hand look at the different sides of this rocking distro at March Linux Homesite



Unless stated otherwise Content of these links are licensed under Creative Commons Attribution-Share Alike 2.5 License.

Monday, July 16, 2007

Quick-Tip: Permission Denied error on secondary DNS server

I like to consider myself a newcomer in DNS/BIND,and it goes to show that you can learn something new everyday. In between my work,i was asked to setup my secondary DNS servers for both 'ar*s.net' and 'calpin*****.net' and other personal domains of the company in the way that I always have.

Scenario:

There was a gateway machine in CentOS which was acting also as the Primary DNS of the company(Set with both Public and Private IP's).The requirement was to set up a secondary DNS in another machine which will recieve updated of all domains from the primary so that it can act as a fallback server.

But I noticed that the secondary was consistently getting the following error:

transfer of 'ar*s'/IN' from #53: failed while receiving responses: permission denied

From this error I assumed that my master server was not setup correctly to allow transfers from the secondary. This is normally done with the following configuration option in the /etc/named.conf file:

allow-transfer { ; };

After I had checked this configuration on the master to make sure it was there and that I had not done something like typed the IP address in error. On the surface everything seemed to be perfectly setup, but I was still getting the error.

Eventually I realized that the error was not a permission denied error from the remote master server, but from the local DNS server. The error turned out to be a file permission error in the default layout of BIND on a CentOS system.

Around the time of Fedora Core 3(Similary in CentOS 4) the default configuration for BIND is setup to chroot the daemon into it's own filesystem space to help avoid and contain any security breaches. This is a great feature.

Option 1

To fix your permission denied error on your secondary or slave DNS server all you need to do is change the permissions of your data directory to include group write permissions. On my system that directory is set as /var/named/sec. You can do this with a simple:

# chmod 775 /var/named/sec

or
# chmod g+w /var/named/

It should be noted that you will only run into this error on a secondary or slave DNS server if you have the secondary store it's slave information in a file. This happens when a slave is configured like this:

zone "example.com" IN {
type slave;
file "secondary-example.com";
masters { 192.168.0.1; };
};

Option 2:

Another option, that in many ways is more correct on a Fedora Core system, is to store your secondary zone files in the slaves/ directory. This directory is in /var/named/chroot/var/named/ and already has the proper permissions for you. So instead of file "secondary-example.com"; you would simply say: file "slaves/secondary-example.com";.

Error Checking:

For checking the errors during the tryout of these options, just restart named daemon like this:

#/etc/init.d/named restart

(note: the shutdown of named my fail, since it may not be running after the initial install)

4. Let's look in the /var/log/messages file and see if the named daemon started without error. Use the command:

#less /var/log/messages

Hopefully this shows you how to resolve this particular error. These suggestions have been tested on Fedora Core and CentOS alike.

Here are some other good DNS guides:

http://ldp.hughesjr.com/HOWTO/DNS-HOWTO.html
http://linux.maruhn.com/sec/dns-howto.html

DNS Rerource Directory : http://www.dns.net/dnsrd/



Powered by ScribeFire.

Thursday, June 28, 2007

Search missing Packages With apt-file On Debian and Ubuntu

This article describes how you can search for missing packages with apt-file on Debian and Ubuntu systems. apt-file allows you to search for a file name, and it gives back the name(s) of the package(s) containing that file so that you can install the appropriate package.


NOTE:Tried this on a Ubuntu 7.04(feisty fawn)distribution in my Intel 1.6GHz machine.

*I do not issue any guarantee that same will work for you in Debian Sarge.But worth a try !

*Try out as root user as no linux distro allows another poweruser as root...:)

You all know this: you try to compile some software from the sources, and suddenly the compilation stops because it complains that some file is missing of which you have never heard before.I had a new Ubuntu PC edition running fine until i stumbled upon such messages during some installs and upgrades:

(The example is just for demonstration.Some messages masked)

root@ajith:~/avant-window-navigator-0.1.1# make
make all-recursive
make[1]: Entering directory `/root/avant-window-navigator-0.1.1'
Making all in src
make[2]: Entering directory `/root/avant-window-navigator-0.1.1/src'
if gcc -DHAVE_CONFIG_H -I. -I. -I.. -DORBIT2=1 -pthread -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/gtk-2.0 -I/usr/lib/gtk-2.0/include -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/pango-1.0 -I/usr/include/libwnck-1.0 -I/usr/include/gconf/2 -I/usr/include/orbit-2.0 -DDATADIR=\""/usr/local/share"\" -DGNOMELOCALEDIR=\""/usr/local/share/locale"\" -g -O2 -Wall -pedantic -std=c99 -fno-strict-aliasing -fmessage-length=0 -D_FORTIFY_SOURCE=2 -MT main.o -MD -MP -MF ".deps/main.Tpo" -c -o main.o main.c; \
then mv -f ".deps/main.Tpo" ".deps/main.Po"; else rm -f ".deps/main.Tpo"; exit 1; fi
main.c:21:21: error: gtk/gtk.h: No such file or directory
In file included from main.c:25:
awn-gconf.h:69: error: expected â)â before â*â token
In file included from main.c:26:
awn-bar.h:43: error: expected specifier-qualifier-list before âGtkWindowâ
awn-bar.h:45: warning: struct has no members
awn-bar.h:48: error: expected specifier-qualifier-list before âGtkWindowClassâ
awn-bar.h:49: warning: struct has no members
awn-window.h:45: warning: struct has no members
awn-window.h:48: error: expected specifier-qualifier-list before âGtkWindowClassâ
awn-window.h:49: warning: struct has no members
awn-window.h:55: error: expected â=â, â,â, â;â, âasmâ or â__attribute__â before â*â token
In file included from main.c:28:
awn-app.h:30:29: error: libwnck/libwnck.h: No such file or directory
In file included from main.c:28:
awn-app.h:60: error: expected specifier-qualifier-list before âWnckWindowâ
awn-app.h:80: warning: struct has no members
awn-app.h:83: error: expected â)â before â*â token
In file included from main.c:29:
awn-win-manager.h:35: error: expected â=â, â,â, â;â, âasmâ or â__attribute__â before â*â token
main.c:32: error: expected â)â before â*â token
main.c: In function âmainâ:
main.c:48: error: âGtkWidgetâ undeclared (first use in this function)
main.c:48: error: (Each undeclared identifier is reported only once
main.c:48: error: for each function it appears in.)
main.c:48: error: âwinâ undeclared (first use in this function)
main.c:49: error: âbarâ undeclared (first use in this function)
main.c:50: error: âboxâ undeclared (first use in this function)
main.c:51: error: âwinmanâ undeclared (first use in this function)
main.c:52: error: âlabâ undeclared (first use in this function)
main.c:54: warning: implicit declaration of function âgtk_initâ
main.c:56: warning: implicit declaration of function âawn_bar_newâ
main.c:58: warning: implicit declaration of function âawn_window_newâ
main.c:59: warning: implicit declaration of function âgtk_window_set_policyâ
main.c:59: warning: implicit declaration of function âGTK_WINDOWâ
main.c:76: warning: implicit declaration of function âgtk_window_set_transient_forâ
main.c:79: warning: implicit declaration of function âgtk_mainâ
make[2]: *** [main.o] Error 1
make[2]: Leaving directory `/root/avant-window-navigator-0.1.1/src'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/avant-window-navigator-0.1.1'
make: *** [all] Error 2

So how do you know which (obviously missing) package contains that file so that you can install the package and try the compilation again? For Debian based systems (like Ubuntu) the solution is apt-file. apt-file lets you search for file names and gives back the package(s) containing that file.

2) Install apt-file

apt-file is installed as follows:

#apt-get install apt-file

After the installation, we must update apt-file's package database like this:

#apt-file update

You must also do this whenever you modify /etc/apt/sources.list.

3) Search For Packages

Now let's search for the package containing the file gtk/gtk.h. You could search like this:

#apt-file search gtk.h

but this would give back lots of packages, so let's narrow down the search by running

#apt-file search gtk/gtk.h

The output looks like this:

#root@ajith:~/avant-window-navigator-0.1.1#apt-file search gtk/gtk.h

libgfcui-dev: usr/include/gfc-2.0/gfc/gtk/gtk.hh
libgtk+2.0-directfb-dev: usr/include/directfb/gtk-2.0/gtk/gtk.h
libgtk1.2-dev: usr/include/gtk-1.2/gtk/gtk.h
libgtk2.0-dev: usr/include/gtk-2.0/gtk/gtk.h
libgtk2.0-doc: usr/share/doc/libgtk2.0-doc/gtk/gtk.html

root@ajith:~/avant-window-navigator-0.1.1#

As you see there are three packages containing gtk/gtk.h (one contains gtk/gtk.hh, another one gtk/gtk.html which is not what we are looking for), and it's now up to you to install the right one. If you are unsure, you can install all three packages, or you install one after the other and check after each one if the compilation error still occurs. In our case the right package is libgtk2.0-dev, so we install that one:

#apt-get install libgtk2.0-dev

Now we try the compilation again:

root@ajith:~/avant-window-navigator-0.1.1# make
make all-recursive
make[1]: Entering directory `/root/avant-window-navigator-0.1.1'
Making all in src
make[2]: Entering directory `/root/avant-window-navigator-0.1.1/src'
if gcc -DHAVE_CONFIG_H -I. -I. -I.. -DORBIT2=1 -pthread -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/gtk-2.0 -I/usr/lib/gtk-2.0/include -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/pango-1.0 -I/usr/include/libwnck-1.0 -I/usr/include/gconf/2 -I/usr/include/orbit-2.0 -DDATADIR=\""/usr/local/share"\" -DGNOMELOCALEDIR=\""/usr/local/share/locale"\" -g -O2 -Wall -pedantic -std=c99 -fno-strict-aliasing -fmessage-length=0 -D_FORTIFY_SOURCE=2 -MT main.o -MD -MP -MF ".deps/main.Tpo" -c -o main.o main.c; \
then mv -f ".deps/main.Tpo" ".deps/main.Po"; else rm -f ".deps/main.Tpo"; exit 1; fi
In file included from /usr/include/gtk-2.0/gtk/gtk.h:170,
from main.c:21:
/usr/include/gtk-2.0/gtk/gtktextbuffer.h:52: warning: ISO C restricts enumerator values to range of âintâ
/usr/include/gtk-2.0/gtk/gtktextbuffer.h:53: warning: ISO C restricts enumerator values to range of âintâ
/usr/include/gtk-2.0/gtk/gtktextbuffer.h:55: warning: ISO C restricts enumerator values to range of âintâ
In file included from main.c:28:
awn-app.h:30:29: error: libwnck/libwnck.h: No such file or directory
In file included from main.c:28:
awn-app.h:60: error: expected specifier-qualifier-list before âWnckWindowâ
awn-app.h:80: warning: struct has no members
awn-app.h:83: error: expected â)â before â*â token
main.c: In function âmainâ:
main.c:52: warning: unused variable âlabâ
make[2]: *** [main.o] Error 1
make[2]: Leaving directory `/root/avant-window-navigator-0.1.1/src'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/avant-window-navigator-0.1.1'
make: *** [all] Error 2
root@falko-desktop:~/avant-window-navigator-0.1.1#

As you see, the initial error is now gone, but still another file is missing, libwnck/libwnck.h, so let's search for the missing package:

#apt-file search libwnck/libwnck.h

This time we're lucky, there's only one package containing that file:

root@ajith:~/avant-window-navigator-0.1.1# apt-file search libwnck/libwnck.h
libwnck-dev: usr/include/libwnck-1.0/libwnck/libwnck.h
root@falko-desktop:~/avant-window-navigator-0.1.1#

So let's install the missing package:

#apt-get install libwnck-dev

I think this handy post can help you from a lot of headaches sporting a debian distribution.


Thanks to article reference by Falko Timme ()at Howtoforge.com.Ubuntu and Canonical are registered trademarks of Canonical Ltd

Saturday, June 16, 2007

Apple's Smartphone - iPhone





















T
he long-awaited Apple iPhone, which hits store shelves on June 29, marks Apple's formal entry into the cell phone world. Steve Jobs announced the iPhone at Macworld 2007 to a frenzied reception and the handset has continued to attract wide interest since then.



iPhone combines three amazing products — a revolutionary mobile phone, a widescreen iPod with touch controls, and a breakthrough Internet communications device with desktop-class email, web browsing, maps, and searching — into one small and lightweight handheld device(also rumoured that the latest addition to gadget would be youtube). iPhone also introduces an entirely new user interface based on a large multi-touch display and pioneering new software, letting you control everything with just your fingers.



Features:



* A full iPod MP3 player with a wide screen, the iPhone also packs in a version of OS X




* True push e-mail via Yahoo Mail (plus compatibility with your office e-mail)



* A 2-megapixel cameraA 3.5-inch display with higher resolution than any current iPod.



* Only one physical controller and only a touch screen for a keypad



* Storage: 4 or 8 GB Flash memory



* Quad band GSM (GSM 850, GSM 900, GSM 1800, GSM 1900)



* WiFi (802.11b/802.11g), EDGE and Bluetooth 2.0 with EDR





Go thru the official gallery at Apple home



Apple received FCC approval for the iPhone on May 17, 2007.
The iPhone will be available in 4GB and 8GB configurations for $499 and $599, respectively, and will be carried by AT&T starting on June 29, 2007 on the US.



The logo apple and all gadgets connected with it are Copyright © 2007 Apple Inc. All rights reserved.Images used for informative purposes.

Monday, June 11, 2007

Adding a hard drive to Linux



I am chalking out a way to add a drive to a linux system.Please note that it is sporting the RedHat flavour of linux and can work with both Enterprise and Fedora versions.The entire process may contain a hardware and software configuration parts.



The physical Stuff



To add the disk drive to the system,first shut down the system, power it off and attach a new drive.For IDE drives, be sure to set the drive as master or slave as appropriate in the BIOS settings.For SCSI drives, you must select an unused SCSI id for the new device and ensure proper termination of SCSI bus to which the drive will be attached.When the system is powered up, watch the output from the kernel during its initialization.If you dont see any references to the new drive, check /var/log/mesg once the system has booted.If the drive doesn't show up there, try restarting the system and checking the system BIOS.The drive may not be recognized there.



The command stuff



Once recognized,run fsck or one of its variants to create the partitions you need.If the partition is going to be a swap partition, change the partition id type to 0x82



Once the partition table on disk is modified, it may be necessary to also update the in-memory copy of the partition table.Use the partprobe command to do this.



Use mkfs(for help #man mkfs) to create filesystems on each of your new,non-swap, partitions.Swap partitions are marked with mkswap.Take into consideration the intend to use the filesystem labels to mount the filesystem later.Specify them using the -L option.Another way to label a disk is the e2label command.



Related issues



Create any needed mount points in your current filesystem hierarchy.Keep in mind that directories used as mount points need not be empty,but any files in the directory are temporarily unavailable when a filesystem is mounted on that mount point.



Add entries for the new filesystem to /etc/fstab.Check these entries with mount manually before you reboot.This will not only make filesystem management simpler, but will call the system initialization scripts to mount, check and provide information to other utilities like the dump



Checklist of files are:



1) /etc/fstab

2) /etc/mtab

3)/proc/partitions



Also refer these links as additional reference:



http://www.skullbox.net/newsda.php

http://www.yolinux.com/TUTORIALS/LinuxTutorialAdditionalHardDrive.html





The articles on filesystems management are on open domain and for reference



Powered by ScribeFire.



Tuesday, May 29, 2007

A Backup Strategy for UNIX

The canonical standard for doing backups on UNIX systems is tar (short for tape archive). tar converts directories of files into a single tar archive file which you would then spit to an attached tape drive. To be up-to-date, you need to do this on a regular basis - perhaps daily. So each day, you'd tar up all your important files and send them off to the tape drive. To recover your files, you'd need to read the whole tar archive file back from the tape and then extract the specific files or directories that you want from the tar archive. This works fine if you're backing up to a tape drive, because you don't care about network bandwidth. If you're backing up over the network to another machine's hard drive, then each daily archive file has to be sent over the network to your backup machine. Even if you don't change any files from one day to the next, the entire set of files gets archived and sent across the network. Bye bye precious bandwidth!



Before going thru the practical part, have a good reference on the linux backup strategy from the following article at linux journal :



http://www.linuxjournal.com/article/1208



The tools of the game

rsync is a remote synchronization tool. The first time, rsync sends all the data over the network to your backup machine. Just like tar. The benefit comes the next time you backup. Instead of sending all the files again, rsync only transfers files that have been changed. If no files were changed, no files get transferred. And when you want to recover data, you transfer just the specific files that you want back to your machine (using rsync or scp or telnet or whatever).

Note that rsync also works better than an incremental backup strategy using tar. You can use tar to do a full backup weekly and incremental backups daily. Incremental backups backup just files which have changed since yesterday's backup. This will improve bandwidth usage, but makes recovery more complex. You have to extract the latest full backup, then extract any more recent incremental backups over your full backup and then extract the specific files that you need. On the other hand, the backup produced with rsync is always up-to-date (as of the last time you ran rsync).

There are lots of backup tools that use rsync as their workhorse and add features on top of it.



Now let me list out some tools that come handy in preparing a full backup strategy for a whole network.Do make a good reference for each tools before using it over the network.The checklist for such a system is as follows:



i)An archiver program like Tar(In manual backups)

ii)Secure shell access(SSH)between machines

iii)rsync server

iv)
cron - a background process scheduler for scheduling the backup processes.



Now i am giving the backup strategy as an example which i prepared for the Nila services in IIITMK.



A backup strategy for IIITMK services

rsync copies only the diffs of files that have actually changed, compressed and through ssh if you want to for security. Only actual changed pieces of files are transferred, rather than the whole file. This makes updates faster, especially over slower links like modems. FTP would transfer the entire file, even if only one byte changed.

We have set up the rsync in each server whose files are to be backed up.It runs as a service in these servers and get invoked through command.Then, it is configured to run on a specific schedule (once every month) and added into the crontab as a script.An example script is shown below :

(Crontab Script for invoking – rsync.sh)

#!/bin/sh

RSYNC=/usr/bin/rsync

SSH=/usr/bin/ssh

KEY=/home/binu/cron/thishost-rsync-key

RUSER=root

RHOST=

RPATH=/backup/BACKUP

LPATH=/BACKUP3

$RSYNC -az -re "$SSH -i $KEY" $RUSER@$RHOST:$RPATH $LPATH

As soon as the cron script is run, rsync is invoked along with a validate script.This script checks the connection attempt and allow only if it is a valid rsync connect.It will not allow any other simultaneous connection between the backed up server and backup machine.The script is outlined as below:

(Script for validating connections – validate.sh)

#!/bin/sh

case "$SSH_ORIGINAL_COMMAND" in *\&*)

echo "Rejected"

;;

*\(*)

echo "Rejected"

;;

*\{*)

echo "Rejected"

;;

*\;*)

echo "Rejected"

;;

*\<*)

echo "Rejected";;

*\`*)

echo "Rejected"

;;

rsync\ --server*)

$SSH_ORIGINAL_COMMAND

;;

*)

echo "Rejected"

;;

esac.

The shell could be connected using rsync to the backup server only if the keys are present thus offering a secure connection.

Backing up to Tapes

After backing up to the Backup server, the backups are transferred to Tape Storage in SUN Solaris server.The primary aim is to rotate the backup periodically and to save diskspace so that we cannot continue dumping to the Solaris disk.For this, the tape library is used(LTO).The tape Library is connected to the Webserver 1. It can host a total of 24 Tape Catridges. The native/compressed capacity of each tape is 100/200GB. Hence it has a total storage capacity of 2.4/4.8 TB. You have load the concerned tape cartridge in the LTO before taking the backup. Make sure the correct Tape cartridge is loaded as it overwrites data. You can use the same cartridge for taking the backup periodically.

We can now go to the process.After the earlier automated backup, the backup is now stored in Backup server with IP : .Now, we have to manually transfer these files to /data directory of Solaris server having IP .This done using SCP.After this, the files are transferred to Tape using the command:



# ufsdump –0uf /dev/rmt0 IP:

On other side,we also manually backup the contents of Solaris servers using ufsdump command.For example, to backup contents of /data2 (containing edugrid and compchem web content), the command is as follows:

# ufsdump –0uf /dev/rmt0 /data2

NOTE :The ufsdump command is issued in the shell of Node-A (Webserver) so as to back it up to the tape.Also, recent backup of all Portals are kept in /dns/BACKUP directory. It can be transferred to a tape cartridge if necessary. Keep the backup in desired directories for a period of time and do rotate the same thus saving disk space

The article on 'Backup strategy' by used for informative purposes.Copyrights lies with the author and Linux Journal Magazine.

Tuesday, April 24, 2007

The Computational Chemistry Cluster of IIITMK

The Computational Chemistry Portal project in IIITM-K help to work in frontier areas in chemistry through interaction with world-class experts and web-accessed resources in order to build Quality education in Chemistry interactively from basic to graduates,post graduates and research levels.The specific aim of this portal is that it has to enable computation in chemistry through grid access to open source software and high end computers spread accross universities and institutions.The use of the Web-enabled software WebMO is a good step ahead in this endeavour.



This project aims at making use of the computational infrastructure of the IIITM-K especially the high-end servers like Linux and Solaris.As IIITM-K has a 24x7 datacenter facilities and high bandwith availability, this
portal has found it to be a good choice to host it there.Moreover, the project here are strongly based on service-oriented principles and ready availability.

































Figure 1 : A typical HPC Stack




IIITMK has a c
oherent Servers Farm comprised of seven high-end servers and a CD-stack server with total capacity approaching a terabyte. It provides a wide variety of Intranet and Web space services. This server farm is accessible from anywhere facilitated by policy-based access mechanisms. Most of the services are accessible through 'My Desktop' and the dot NET Enterprise Servers and Linux Servers.



Let me chalk out the computational resources employed in this portal under IIITM-K.The Computational Chemistry Portal,its associated services like Blog, the analytical chemistry portal,etc are hosted under high-speed SUN Solaris servers which make the web part.The backend of this portal is a set of High-performance Cluster(HPC) machines which provide aggregate computational speed of 5 individual machines.



Apart from this, there are the high-performance machines for separate computations which are operated through console or shell.There are also many open source visualization and analytical tools or softwares as part of the project and easily downloadable from the portal.The portal conducts workshops, hand-on labs, etc from time to time and undertake research projects, training ,etc side-by-side as its activities.



Computational Resources : Statistics



  • Solaris Webserver (SUNFire v120) with Storedge Storage Server(in Terabytes) in cluster

  • Computations Server with 2.4 GHz and 2GB RAM for shell computations

  • Computational Cluster(HPC) with 5 nodes and having each 2 GHz processor and 256 MB RAM fully operational with WebMO for web-enabled computations

  • Independant workstations for users with 2.4 GHz and Windows preinstalled
  • Computational Packages like: GAMESS,NWChem,WebMO(licensed),QMView,Molden,Ghemical,gromacs,dgbas and lots of other open source and free softwares for Structure Computations,Molecular Dynamic Computations and Visualizations.



Overview of the Clu
ster































Figure 2: Cluster Schematic Diagram



We are having a Beowulf Cluster for Computational chemistry.It has 5 nodes in total - A head node and 4 compute nodes which assist in the partitioning of the work.We have some parallel jobs in computational chemistry submitted to the machine. The cluster makes the computations 30 times faster than the computation in a single machine with greater RAM and processor speed. In fact, it consists of 4 machines with Intel 2.93*2(dual core) processors and 512 ram each. The NFS mount has greatly reduced RAM consumption acting as swap. Also,it is available for the 4 parallely connected system.Also we are planning for using SUN grid engine integrated to the cluster to provide grid services.



Cluster software’s



The following software’s are installed:

1. GAMESS in Parallel

2. NWChem in Parallel

3. Tinker in sequential

4. WebMO on Parallel installed in head node.



The applications are installed in /export/apps/

(Only root has access to modify the apps directories and files)

The cluster can be monitored through the url http://192.168.1.12/ganglia locally.



WebMO is yet to be programmed and configured for automatic host detection.
Currently GAMESS and NWChem are hard coded for parallel execution.Cluster Front-end Webpage can be accessed at : http://192.168.1.12 ,locally



Cluster Configuration



Compute clusters – cluster.iiitmk.ac.in (Head node)

Compute-0-0.local

Compute-0-1.local

Compute-0-2.local

Compute-pvfs-0 (Compute nodes)



Compulsory services on nodes:

nfs, sshd, postfix, gmond, nfslock, network, gmetad, iptables, mysqld, httpd





Partitioning scheme



/                          Label =/           5.8GB(/dev/hda1)

/state/partition     /state/partition  1 67GB

swap                                           1GB



'/state/partition1' is mounted on ‘/export’ on the head node and /export is exported to other nodes via NFS. Whenever you create a user its home directory is created on “/export/home/username”. When user logs in, it is mounted on /home/username.



On compute nodes, a cluster. local:/export/home/username is mounted as /home/chemistry. This is available on each node.Applications that are compiled custom are placed in ‘/export/apps’ on the head node and exported as /share/apps on compute nodes.



deMon,dgbas,fftw,g03,gamess,gromacs,gv,NWChem,tinker



A directory called ‘scr’ is created on /state/partition and permissions are changed to user chemistry as owner. This /scr partition is not shared (Only the head node partition, /state/partition is shared).

You can execute same command on all nodes by issuing simply once in the head node in this manner:



# cluster-fork “command name”



e.g.: # cluster-fork “df –h”



(Next - Backup Scheme of Computational Clusters)

Friday, April 20, 2007

The Google Architecture

I was in through various aspects of google technology for past 1 month and i thought it is good to alert you to peek into the underlying unique technology of the Internet Search engine giant.Unlike common high traffic servers' architecture, the Google cluster architecture is based on strong software and lots of PC's. Some say more then 15,000 PC's are taking part of Google's phenomenon.

I am referring the Quazen web article which is a fine imprint of the technology.Let's take a look at this wonderful search engine intestine.

Google's architecture provides reliability in the Google's servers and Pc's environment at the software level, by replicating services across many different machines. Google is also proud of its own failures detecting mechanism which handles different threats and malfunctioning in its web .

The mechanism : When a user enters a query to Google the user’s browser first performs a domain name system (DNS) lookup to map www.google.com to a particular IP address. To provide sufficient capacity to handle query traffic, the Google service is being spread to multiple clusters distributed worldwide.

Each cluster has around a few thousand machines, and the geographically distributed setup protects Google against disaster at the data centers (like those arising from earthquakes and large scale power failures).

A DNS-based load-balancing system selects a cluster by the user’s geographic location to each physical cluster. The load-balancing system minimizes back and forward trips for the user's request.

The user’s browser then sends HTTP request to one of these clusters, and thereafter, the processing of that query is entirely local to that cluster. A hardware-based load balancer in each cluster monitors the available set of Google Web servers (GWSs) and performs local load balancing of requests across a set of them. After receiving a query, a GWS machine coordinates the query execution and formats the results into HTML response to the user’s browser. Query execution consists of two major phases.

In the first phase, the index servers consult an inverted index that maps each query word to a matching list of documents (the hit list). The index servers then determine a set of relevant documents by intersecting the hit lists of the individual query words, and they compute a relevance score for each document.

This relevance score determines the order of results on the output page. The search process is challenging because of the large amount of data: The raw documents comprise several tens of terabytes of uncompressed data, and the inverted index resulting from this raw data is itself many terabytes of data. Fortunately, the search is highly parallelizable by dividing the index into pieces (Index shards), each having a randomly chosen subset of documents from the full index. A pool of machines serves requests for each shard, and the overall index cluster contains one pool for each shard. Each request chooses a machine within a pool using an intermediate load balancer which means - each query goes to one machine (or a subset of machines) assigned to each shard. If a shard’s replica goes down, the load balancer will avoid using it for queries, and other components of our cluster management system will try to revive it or eventually replace it with another machine.

During the downtime, the system capacity is reduced in proportion to the total fraction of capacity that this machine represented. However, service remains uninterrupted, and all parts of the index remain available. The final result of this first phase of query execution is an ordered list of document identifiers (docids). The second phase involves taking this list of docids and computing the actual title and uniform resource locator of these documents, along with a query-specific document summary.

Document servers (docservers) handle this job, fetching each document from disk to extract the title and the keyword-in-context snippet. As with the index lookup phase, the strategy is to partition the processing of all documents by randomly distributing documents into smaller shards, having multiple server replicas responsible for handling each shard, and routing requests through a load balancer.



It would be worthwhile at this moment to refer some of the research papers on different aspects of this tehnology,

Luiz Barroso, Jeffrey Dean and Urs Hoelzle

Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Mike

Burrows, Tushar Chandra, Andrew Fikes, and Robert E. Gruber

Sanjay Ghemawat, Howard Gobioff, and Shun Tak-Leung

Mike Burrows

Chapter from the book 'The Google Legacy'(PDF - right click and save target)



Reference : Article on Quazen Web on 'Google Cluster Technology' and Google labs (http://labs.google.com) for Research Papers on Google Technology

Sunday, April 15, 2007

Problem of hosts blocked in MySQL because of 'max_connect_errors' variable

Problem : "Host 'host_name' is blocked" error in mysql version 4.0 and later



This is a situation encountered in remote database connections to a typical mysql server (versions greater than 4.0).The mysql can be either installed in linux or windows platform.But such an error is thrown up when a user is trying to connect using sqlyog to that machine through the network.Some more factors that you should check is given first.



Checklist:



  1. Make sure that firewall is not running in blocked state in the client or the server machine.This can block the connection attempts.
  2. Make sure that no more clients other than the allowed no: of clients are trying to connect to the server at a given point of time.


If you get a 'Too many connections' error when you try to connect to the mysqld server, this means that all available connections are in use by other clients.



The number of connections allowed is controlled by the max_connections system variable. Its default value is 100. If you need to support more connections, you should restart mysqld with a larger value for this variable.


mysqld actually allows max_connections+1 clients to connect. The extra connection is reserved for use by accounts that have the SUPER privilege. By granting the SUPER privilege to administrators and not to normal users (who should not need it), an administrator can connect to the server and use SHOW PROCESSLIST to diagnose problems even if the maximum number of unprivileged clients are connected. See See reference in MySQL site

The maximum number of connections MySQL can support depends on the quality of the thread library on a given platform. Linux or Solaris should be able to support 500-1000 simultaneous connections, depending on how much RAM you have and what your clients are doing. Static Linux binaries provided by MySQL AB can support up to 4000 connections.

Now we can come to the error message which is the heart of the problem.

Error :Host 'host_name' is blocked.

If you get the following error, it means that mysqld has received many connect requests from the host 'host_name' that have been interrupted in the middle:

The actual error listing goes like this.

Host 'host_name' is blocked because of many connection errors.Unblock with 'mysqladmin flush-hosts'

The number of interrupted connect requests allowed is determined by the value of the max_connect_errors system variable. After max_connect_errors failed requests, mysqld assumes that something is wrong (for example, that someone is trying to break in), and blocks the host from further connections until you execute a mysqladmin flush-hosts command or issue a FLUSH HOSTS statement

By default, mysqld blocks a host after 10 connection errors. You can adjust the value by starting the server like this:
shell> mysqld_safe --max_connect_errors=10000 & 

If you get this error message for a given host, you should first verify that there isn't anything wrong with TCP/IP connections from that host. If you are having network problems, it does you no good to increase the value of the 'max_connect_errors' variable.

It is also handy to issue a flush hosts command.Refer the following link for the syntax:

http://dev.mysql.com/doc/refman/5.0/en/flush.html

After these steps,reconnect using 'Sqlyog'.Wishing good days.

Source Documentation : MySQL Developer Zone

Wednesday, April 04, 2007

Scribefire - Fire up your blogging

























Scribefire - Preview



ScribeFire - earlier called 'Performancing' ,is a blog editing tool which can integrate with your firefox browser and lets you easily post to your blog.It opens a split panel that provides a full of text-editing tools, complete with rich/source editing tabs, a live preview, a post history and more. You can easily drag and drop text, images and links to the ScribeFire pane, making it super-easy to reference other sites and info within your blog entries.

The latest version, 1.4, promises improved support for Blogger accounts and better file uploading. It also supports the new WordPress API.You can hook it up to your personal blog and fire your words whenever you find convenient !!!

In addition to Blogger and WordPress, ScribeFire works with Jeeran,LiveJournal, TypePad and Windows Live Spaces. This killer extension costs nothing; it requires Firefox 1.5 or later.
Powered by ScribeFire.