Planet Sun
January 07, 2009
Product: Solaris 10 Operating System OpenSolaris
Multiple security vulnerabilities in the Flash Player plugin distributed with Solaris may allow a remote unprivileged user the ability to execute arbitrary code with the privileges of a local user on the system while loading a malicious SWF file with the affected plugin. In addition, the Flash Player plugin may allow a remote user to bypass the Security Sandbox Model, modify the clipboard with a URL, allow cross-site scripting attacks, inject arbitrary web script or HTML, obtain sensitive data, conduct DNS rebinding and hijack the camera or microphone while loading a malicious SWF file with the affected plugin.
Additional information regarding these issues are available at:
State: Resolved
First released: 06-Jan-2009
- Debian, Philosophy, and People
Where angels fear to tread. "So if even the sixth and eighth commandments admit to exceptions, why is it that some Debian developers approach the first clause of the Debian Social Contract with a take-no-prisoners, no-exceptions policy?" - Police set to step up hacking of home PCs
Oh for goodness sake, surely someone has both a clue and a conscience in British politics? Most upsetting phrase? "step up". They are already doing it. - Spinning the war on the UK's sex trade
Excellent look at how interest groups and politicians spin issues to create bad legislation that serves their campaigning ends. The same process goes for anything involving the word "drug" as well. - Gaza conflict: Who is a civilian?
"When I use a word," Humpty Dumpty said, in a rather scornful tone, "it means just what I choose it to mean - neither more nor less." - Open source: a different approach to developing software
From the customer viewpoint, a lot has been made of the cost benefits of free software. Simon Phipps, chief open source officer and evangelist at Sun, denies that this is the main attraction. "It is not about getting free stuff. If you hear a CIO say he is going to reduce his costs by not having to pay for software licences any more, then he has the wrong idea. Open source is about having control over what you pay for, what you hire for, and what you do not pay for."
When I was a kid, I could play with running water almost indefinitely.
Running outside I mean, over the local topography of dirt and
stones.
The idea is to guide it and shape it and send it where you want it to go, and
enjoy the slow progress of the leading tendrils around the obstacles
inexorably seeking less potential energy, and then the urgent following flow,
probably discharged into a new local-minimum pool that needs Somewhere To
Go.
Well, we have a
snow problem. Which
will be ameliorated, we hope, by warmer temperatures and the best part of 10cm
of rain over the next couple of days.
But only if we give the run-off Somewhere To Go.
In our case, there’s a storm drain at the downhill end of the block, five
houses away. The water needs to go there. In between are large
shoveled-up heaps
of what was once fresh white snow, also the occasional hopelessly-buried
automobile.
Thus we have invested some hours, with a
loose confederation of neighbors, in fashioning a small snow-free spillway
along the gutter to the drain to confer an escape route on the
melt-off.
Of course, strategically-placed ditches are required in the snowpack to
afford drainage for odd combinations of rain-lashed shovel-heaps and nearby
local minima. When you connect one of these to the mainstream, there
is both a rush of both chilly water and aesthetic pleasure.
I had to go out in the cold rain at midnight to admire once again
the smooth melting-snow-flow extending a hundred yards from our uphill
neighbors’ down to the roar of the corner drain.
遅くなりましたが、明けましておめでとうございます。今年もよろしくお願いします!
さて、いきなりですが、Sun Tech Days の Globalization ブースでも紹介していた CTI (Community Translation Editor) を紹介します。
これは、まだ開発段階のものですが、完成した暁には、サンで持っている翻訳データベースからマッチするものを最大限に利用して、タグや更新部分がどこかということをまったく気にすることなく、翻訳作業に集中できるというものです。これまでオープンソースではなかなか対応できなかった SGML も、これなら大丈夫!.... のはず !!
社内にプロジェクトチームがあって、日本語とチェコで開発しているのですが、そのチームがフィードバックを欲しがっています。ブログは、こちらです。わたしも興味があるので、コメントいっぱい出しています。ブログの日本語化もしてみました。ベータ版がありますので、みなさんも是非触ってみてください。そして、こんな機能が欲しい、ってコメントしてくださいね。わたしのブログでも、CTI のブログでも。
English:
It's a little late, but let me send you the new year greeting! How have you been in the winter break ? I have been with my parents and brother/sisters and talked a lot over wonderful sake and wine...
.
Today, let me introduce CTI, Community Translation Interface (CTI). This is still under the development, but once it completes, we do not have to worry about the complexity of SGML structure/tags nor the difference from the past release, and leverage the max of the existing Sun translation database. Isn't it fascinating ?
We have a project team in Prague and they want to get more feedback. I am communicating with them and sent lots of requests already, but not enough
I also joined a authoring member of their blog, and added Japanese translation. Please take a look of this blog, try CTI, and send your comments to us!
When
to use Solaris vs. Linux: Operating system comparison: "So what's
the difference between OpenSolaris and Linux? First and foremost –
Linux distributions will use the Linux kernel and OpenSolaris
distributions will use the Solaris kernel. This important to note,
because while Red Hat Enterprise Linux is built entirely from open
source Linux, Solaris is (or was until recently) built entirely from
closed/propriety engineering efforts by Sun Microsystems engineers." --
Ken Milberg, Search Enterprise Linux
"Built entirely from closed/propriety engineering efforts by Sun
Microsystems engineers?" Sorry. Not quite.
The OpenSolaris kernel is over 40,000 files now, and there are
thousands of files from BSD and thousands of files from AT&T and
many of those of files are core to the system. And, of course, there
are many other bits that come from other open source communities. Also,
since we started opening in January of 2005 (that's four years ago, not
"recently" as suggested above) the OpenSolaris community has been
contributing. We have about three hundred contributions integrated from
a couple of hundred contributors in two dozen countries and also contributions to projects and other distributions as well.
Now, given the
obvious infrastructure constraints this nascent community has been
working under, that relatively modest contribution level could be seen
as a pretty good start. It's something to learn from and build on with
the opening of more infrastructure and the creation of new package
repositories in an effort to increase contributions. That's pretty much
what we are doing. Are
we behind on some of this? Yep. Are we satisfied with where we are?
Nope. Are we "entirely" closed? Nope. In fact, that assertion is
entirely wrong. Is all of
this well known in the community? Absolutely. So, although it's true
the majority of code in the kernel has been written by Sun engineers,
we've never stated otherwise. It's simply the reality of where the
project is at the moment and how it evolved. But to understand where
the project is going you need more context, and that perspective is
not offered in the quote above. Nor is it present in the paragraph that
follows the quote and concludes the article. I'm not saying the article is all bad. There are some positive things in there about OpenSolaris, but the overall impression you are left with is that this is all a half hearted attempt to jump on some bandwagon. And, well, I just disagree.
Maybe Blu-Ray technology is utterly irrelevant, and maybe Java's role there is irrelevant too.
パスワードの最大文字数に関してメモ代わりにまとめておきます。
Solaris 10 ではデフォルトでは8文字までになっています。
これは、crypt_unix という旧来のアルゴリズムがデフォルトになっているためです。
9文字以上のパスワードを利用するためにはこのアルゴリズムを変更する必要があります。
変更するファイルは
/etc/security/policy.conf
になります。
下記はその1部を抜粋したものになります。
この CRYPT_DEFAULT を 1 や 2a などに変更することで、
デフォルトのアルゴリズムが変更されます。
Solaris 10でデフォルトで用意されているのはCRYPT_ALGORITHMS_ALLOWで定義されている
1,2a,md5 で、
1はMD5, 2a は Blowfish になります。
#
CRYPT_ALGORITHMS_ALLOW=1,2a,md5
#
CRYPT_DEFAULT=__unix__
当然ですが、この設定を変更後、passwd コマンド等でパスワードを変更しなければ意味はありません。^-^;
Sun Java Directory Server を利用している場合も
userPassword フィールドに crypt で格納している場合、
同様の設定で対応できます。
The comments on the
Digg post on "Shouting in the Data
Centre" [
Youtube |
this Blog ]
disappointed me. I am not a great user of Digg and very few of my submissions
have taken off. It is one of the feeds I subscribe to using Google Reader which
is my first choice feed reader today. It seems that I am obviously not
interested in the same stuff as most of its users, but to find the majority of
comments about the provenance of the Digg takes self reference to the point of
absurdity. It reminded me of a very recent a post 'openpeel', called '5 Ways to fix Digg', and it
also reminds me of Simon Phipps' comment,
"When you invent a system, you invent the system that
games it!".
Its a shame, but I suppose that the social software designers will have to
become cleverer. It's clearly a fact that a 'karma' systems attracts people to
contribute to the 'wisdom of crowds', but also trying to measure the influence,
popularity or even innovativeness/leadership of contributors often leads to
anti-social, even destructive behaviour.
I wonder if digg has jumped the shark as its user community has grown beyond
an expertise focus and its designers loose the arms race with the gamers. Is
there an alternative? I have considered for a while the use of 'clubs',
where feed consumers, i.e. me and you, qualify the contributors to our feeds, or
membership is gated. I use del.icio.us to keep my bookmarks and thus act as
the original source of my contributions to finding interesting news. These thus
become available through RSS, and then those I really think are interesting to
others, I use
Google Reader shares to share them. In the past I have used Slynkr, and have been using Digg to act as an entry point to my friend feed. The Google Share is a cute
feature as the Google Reader makes my google friends' shares available to me. I use this to read other people's shared articles. The google shares I
post may become my Digg replacement, but there's now no weighting or rating and
my community is pretty small, since it is based on google talk/chat friends,
which is not my first choice chat protocol.
The Google Share/Talk synergy is another interesting example of leveraging closed communities, and
functional synergy by the software authors. Retaining the choice of internet participants against this new "lock in" could be open source's next
big problem to solve.
tags: technology socialsoftware discovery
digg "google reader" youtube
l
Product: Solaris 10 Operating System OpenSolaris
A security vulnerability in the NFS version 4 client within Solaris may allow a local unprivileged user to panic the system. This is a type of Denial of Service (DoS).
State: Resolved
First released: 05-Jan-2009
Product: Solaris 10 Operating System OpenSolaris
A security vulnerability in the Solaris name service cache daemon (nscd(1M)) may, under certain conditions, allow local unprivileged users to gain access to unauthorized information and gain elevated privileges.
Sun acknowledges with thanks, Mike Gerdts for bringing this issue to our attention.
State: Resolved
First released: 18-Dec-2008
Product: Sun Java System Access Manager 6 2005Q1 Sun Java System Access Manager 7.1 Sun Java System Identity Server 6.1 Sun Java System Identity Server 6.2 Sun Java System Access Manager 7 2005Q4
The Sun Java System Access Manager may not securely process XSLT stylesheets which are contained inside XSLT Transforms in XML Signatures.
A remote user who is able to create such an XML Signature which is viewed locally with Access Manager may be able to execute arbitrary code with the privileges of the Access Manager application. Access Manager is run by a web container application, such as the Sun Java System Application Server, and thus the privileges of Access Manager are the same as the configured web container application.
Sun acknowledges with thanks, Brad Hill of iSEC Partners for bringing this issue to our attention.
State: Resolved
First released: 26-Jun-2008
Glenn Brunnette pointed this Youtube Video out to me
which struck me as rather cool in that it demonstrates the awesome advantage
of the FISHworks analytics i.e. the management software that comes with Sun's
Unified Storage systems. Its such a great way of seeing the power of the
software I decided to bookmark it on del.icio.us and digg it, [here], I glad to
see I am not the first. I was, however, sad to see that the digg conversation
was so trivial, amusingly focused on the effects of shouting at computers,
which we've all done, and less so about the track record of the person who
submitted the story to digg. Has Digg jumped the shark?
tags: technology storage sunw
fishworks video youtube
Finally I decide to install OpenSolaris on the bare metal, and probably use it as a nightly build machine.
- Create a USB installer using usbcopy
- Boot from this USB disk and install
- Reboot, disable network/physical:nwam, enable multicast and network/physical:default, call sys-unconfig
- Reconfigure the machine
- Reboot again
I hadn't enabled/disabled the services first time when I run sys-unconfig, and the machine cannot reboot complaining avahi-bridge-dsd cannot start. Fortunately I can login to single user mode and do that again.
I'm learning how to give more privileges to my NIS user account now.
I am very grateful for Zipcar today. Seven bucks an hour is a friggin deal.
気がつくと、年が変わってもう一週間も経ってしまっていました・・・今年もよろしくお願いいたします。
もしかしたら気づいている方もおられるのではないかと思いますが、JavaFXの公式ホームページだった https://openjfx.dev.java.net/ が、少し前から out-of-date 化しています。「これどうするの?」と最近サイトの管理人に尋ねてみたのですが、とりあえず当分ほっとくことになりそうです。(^^; JavaFXがオープンソースになった暁には再利用するかもしれません。
これまでこのサイトには日本語ページが存在していて櫻庭祐一さんがボランティアで面倒を見てくださっていたのですが、使用停止に伴い、そちらもなくなってしまいました。櫻庭さん、これまでどうもありがとうございました!!!!!
(そしてもちろん、ドキュメントの和訳を提供してくださっていた日本Javaユーザグループにも感謝申し上げます。)
今後JavaFXの公式ページとしては、http://javafx.com を使っていただけますよう、お願いいたします。
> 特に記事をお書きになる方々
残念ながら http://javafx.com は翻訳されていないのですが、サンプル集 のところで見られるコード群は十分参考になるのではないかと思います。
新年伊始,趁着假期,把几篇英文的JavaFX博文翻译了一下,以后争取中英两种语言一起发吧,可以参考这里:
JavaFX技术 学习 编程 演示 教程 例子程序范例 示范
javaFX guy的博客 教程 例子示范游戏 学习 编程
JavaFX技术 交流 学习
I've lived in Southern California all of my life.
The Pasadena
Tournament of Roses Parade has been on my list of local things to experience for a long time. This year we finally got a chance to go and see the parade. We headed up to my in-laws house New Year's Eve and spent the night. They live about 30 minutes away from Pasadena. My sister-in-law, Lisa works as a registrar at the
Norton Simon Museum, which is right on the corner of Orange Grove Boulevard and Colorado Boulevard. She was able to get us free tickets and VIP parking at the museum. We sat in the front row right across from the KTLA TV broadcast booth. If you are going to watch the parade, this is the way to do it!
We arrived at around 7am and really didn't get stuck in too much traffic on the way in. Our VIP parking pass got us through a couple of road blocks quickly. The streets were packed with people before the parade started. It took multiple passes by the police on motorcycles to clear out the parade route. The parade officially started at 8am, but the first 15 minutes is an opening show that takes place just around the corner from where we were located. Once the parade started it is a
steady stream of equestrians, marching bands and floats. I took lots of pictures. About an hour into the parade the B-2 Bomber made a flyover.
After the parade ended, it took us about an hour to fight our way through the traffic back on to the freeway. We headed back to my in-laws house and watched USC beat Penn State in the Rose Bowl game. Go Trojans! A great way to start off 2009.
Back to work after a nice 2 weeks off for the year end holidays. Since I've been neglectful of this blog I'll bring you up to date. Right before the break we had to take our dog Rocket to the vet. He had been drinking water constantly (and peeing everywhere). He was diagnosed with canine diabetes. We had a rough weekend while we tried to decide what to do with him. Our hectic lifestyle is not very conducive to twice daily insulin injections and regular blood sugar checks. We tearfully put him down on Monday 12/22/08. He was almost 10 years old and we bought him from a neighbor who breeds Bichon Frises', so we had him from when he was weaned.

Rocket with Alex.

Marissa and Rocket.
Our other dog, Dakota, has been really missing his little buddy.
NetBeans 6.5 用に
NetBeans jVi プラグインも 1.2.1.alpha1 というリリースが出ています。
昨年末いくつか新しいリリースが出ていたものがあるのでメモしておきます。以前に紹介した Flex プラグインは 1.0 Beta が出ています。以前のは 6.1 用でしたがこれは 6.5 用のプラグインです。
Life is too short for Sendmail!
Background
No Sendmail wizards!
|
This article is about upgrading ancient
Sendmail
email server software with
Postfix.
About a year ago I
upgraded my OpenSUSE-based web/file/dns/ntp/mail server to OpenSolaris.
That has worked out for the most part—especially having the ZFS filesystem. The part I missed though was having to go back to using Sendmail instead of Postfix for email.
Postfix was written by
Wietse Venema
as a Sendmail replacement.
Venema has also written
TCP Wrappers and SATAN security software.
The uninitiated may wonder is Sendmail evil, and why?
Well, here's a few reasons off the top of my head:
-
Configuration is difficult. Here's two samples from the sendmail.cf configuration file:
V10/Sun
Cwlocalhost
Fw/etc/mail/local-host-names
CP.
DS'smtp-server.san.rr.com'
CO -AT- % !
C..
C[[
C{ResOk}OKR
# . . .
|
SHdrFromSMTP
R$+ $: $>PseudoToReal $1 sender/recipient common
R:; <@> $@ list:; special case
R$* <@> $* $@ $1 <@> $2 pass null host through
R< @ $* > $* $@ < @ $1 > $2 pass route-addr through
R$* $: $>MasqSMTP $1 qualify unqual'ed names
R$+ $: $>MasqHdr $1 do masquerading
SMasqRelay
R$+ $: $>MasqSMTP $1
R$+ $: $>MasqHdr $1
Msmtp, P=[IPC], F=mDFMuX, S=EnvFromSMTP/HdrFromSMTP, R=EnvToSMTP, E=\r\
n, L=990,
T=DNS/RFC822/SMTP,
A=TCP $h
Mesmtp, P=[IPC], F=mDFMuXa, S=EnvFromSMTP/HdrFromSMTP, R=EnvToSMTP, E=\r
\n, L=990,
# . . .
|
In fairness, this complexity can be partly-hidden with m4 macros.
Here's an example:
VERSIONID(`@(#)sendmail.mc 1.11 (Sun) 06/21/04')
OSTYPE(`solaris8')dnl
DOMAIN(`solaris-antispam')dnl
MASQUERADE_AS(`drydog.com')dnl
FEATURE(`dnsbl', `sbl.spamhaus.org',`"550 Mail from " $`'&{client_addr} " refused - see http://www.spamhaus.org/sbl/"')dnl
define(`PROCMAIL_MAILER_PATH',`/opt/sfw/bin/procmail')dnl
FEATURE(local_procmail)dnl
MAILER(`local')dnl
MAILER(`smtp')dnl
MAILER(`procmail')dnl
LOCAL_NET_CONFIG
R$* < @ $* .$m. > $* $#esmtp $@ $2.$m $: $1 < @ $2.$m. > $3
|
However, the above is very position oriented.
Add a line in the wrong location and sendmail breaks.
Furthermore, when things go wrong you still have to debug the original
sendmail.cf file, not the slightly-more-friendly sendmail.mc macro file.
-
Error messages are cryptic and require searching Google and Google Groups
for hints for what they mean.
For example, try figuring out what's wrong when you get these cryptic
error messages:
550 5.1.2 <nameremoved@illinois-DOT-edu>... Host unknown (Name server: 127.0.0.1.com: host not found)
|
-
The Sendmail (or "bat" book), the "Bible" for Sendmail users, is 1308 pages long!
-
Sendmail has poor ("add-on") integration with useful filters such
as
SpamAssassin,
Procmail,
SSL certificates,
and various Blackhole lists.
-
People who favor Sendmail seem to like gratutious complexity.
Sendmail experts tend to call themselves Wizards—as if
complex, buggy, mysterious, hard-to-use software is acceptable!
-
Finally, life is short. Spending time on Sendmail is like dying
a little bit sooner.
Installation
Postfix is available from a few sources.
I used the one from Blastwave.
Blastwave packages tend to be solid, but they also tend to install
a lot of library packages that duplicate what's already in Solaris.
I won't repeat the steps to setup Blastwave.
Instead carefully follow the steps at
http://www.blastwave.org/howto.html
When complete, type these commands to add
Procmail,
along with
SpamAssassin
(useful mail filters):
pkg-get spamassassin
pkg-get procmail
pkg-get postfix
|
Other alternative Postfix ports are available.
The port by Dogan is popular, but I haven't tried it. See here for a list
of Postfix ports:
http://www.postfix.org/packages.html
You can also download and compile the source from
http://www.postfix.org/download.html
If you know of other Postfix ports to Solaris, leave a comment.
Customization
After Postfix is installed, you only need to type "postfix start"
to start it (after disabling sendmail).
This can be automated with init.d scripts (see below) or
SMF (SMF steps are left as a exercise for the user).
Besides setting up init.d scripts, I setup some links
(for /etc/postfix), a spool directory at /var/spool/postfix,
and copy over the aliases file from Sendmail to Postfix:
# Disable sendmail (works for S10 or higher)
svcadm disable svc:/network/smtp:sendmail
# Change spool directory to /var/spool/postfix
mkdir /var/spool/postfix
chgrp postdrop /var/spool/postfix
cd /opt/csw/var/spool/postfix; find . -depth | cpio -pdm /var/spool/postfix
# Setup init.d startup links (or use SMF):
ln -s /opt/csw/sbin/postfix /etc/init.d
cd /etc
for i in rc0.d/K36postfix rc1.d/K36postfix rc2.d/S88postfix rcS.d/K36postfix
do
ln -s ../init.d/postfix $i
done
# Setup /etc/postfix link to configuration directory:
cd /etc; ln -s opt/csw/postfix
# Setup link for SpamAssassin spamc for easy access:
ln -s /usr/csw/bin/spamc /usr/bin
# Copy aliases from sendmail:
cp /etc/mail/aliases /etc/postfix
/opt/csw/bin/newaliases
|
Here's the changes I make to the Postfix /etc/postfix/main.cf
configuration file (in bold).
The main.cf file has lots of comments to explain what these lines mean.
The host and domain name stuff default to reasonable values,
but I set them anyway, just to be sure they're correct.
Other changes are for spam control.
queue_directory = /var/spool/postfix
command_directory = /opt/csw/sbin
daemon_directory = /opt/csw/libexec/postfix
mail_owner = postfix
myhostname = tahoma.drydog.com
mydomain = drydog.com
myorigin = $mydomain
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain
unknown_local_recipient_reject_code = 550
mynetworks_style = subnet
alias_maps = dbm:/etc/opt/csw/postfix/aliases
alias_database = dbm:/etc/opt/csw/postfix/aliases
mail_spool_directory = /var/mail
mailbox_command = /opt/sfw/bin/procmail
debug_peer_level = 2
debugger_command =
PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin
xxgdb $daemon_directory/$process_name $process_id & sleep 5
sendmail_path = /opt/csw/sbin/sendmail
newaliases_path = /opt/csw/bin/newaliases
mailq_path = /opt/csw/bin/mailq
setgid_group = postdrop
html_directory = /opt/csw/share/doc/postfix/html
manpage_directory = /opt/csw/share/man
sample_directory = /opt/csw/share/doc/postfix/samples
readme_directory = /opt/csw/share/doc/postfix/README_FILES
maps_rbl_domains = sbl.spamhaus.org
smtpd_helo_required = yes
smtpd_recipient_restrictions = permit_mynetworks,
reject_maps_rbl,
reject_unauth_destination,
reject_non_fqdn_sender,
reject_unknown_sender_domain,
reject_non_fqdn_recipient,
reject_unknown_recipient_domain,
reject_unauth_pipelining,
reject_unknown_client,
reject_invalid_hostname,
reject_non_fqdn_hostname
|
Finally, I start postfix and test it by sending and receiving
myself emails to myself locally and remotely (e.g., from gmail or Yahoo).
Errors and messages are logged to /var/log/syslog by default.
/etc/init.d/postfix start
/etc/init.d/postfix reload
|
When customizing Postfix, or software in general,
I recommend changing just one thing at a time and verifying it works.
Also, save old copies of the configuration files you modify— especially main.cf
Type postfix reload after modifying main.cf to have Postfix re-read the current version of main.cf.
For more information on Postfix use and configuration,
see postfix.org's
documentation
and
mailing lists.
RFE 6386192 concerns adding Postfix to Solaris.
2008 was a busy year for me since I spent most of my free
time co-authoring a book on OpenSolaris; the
OpenSolaris Bible.
Having never written a book before, this was a new experience for me.
Nick originally had
the idea for writing a book on OpenSolaris and he'd already published
Professional C++ with Wiley,
so he had an agent and a relationship with a publisher. In December 2007 he contacted
me about being a co-author and after thinking it through, I agreed. I had
always thought writing a book was something I wanted to do, so I was
excited to give this a try. Luckily,
Dave agreed to be the
third author on the book, so we had our writing team in place. After
some early discussions, Wiley decided our material fit best into their
"Bible" series, hence the title.
In early January 2008 the three of us worked on the outline and decided which chapters
each of us would write. We actually started writing in early
February of 2008. Given the publishing schedule we had with Wiley, we had
to complete each chapter in about 3 weeks, so there wasn't a lot of time to
waste. Also, because this project was not part of our normal work for
Sun, we had to ensure that we only worked on the book on our own time, that is evenings and
weekends. In the end it turned out that we each wrote
exactly a third of the book, based on the page counts.
Since the book came out at around 1000 pages, with approximately
950 pages of written material, not counting front matter or the index,
we each wrote over 300 pages of content. Over the course of the project we were
also fortunate that many
of our friends and colleagues who work on OpenSolaris were willing to review
our early work and provide much useful feedback.
We finished the first draft at the end of August 2008 and worked on the revisions
to each chapter through early December 2008. Of course the
OpenSolaris 2008.11
release came out right at the end of our revision process, so we had to scramble
to be sure that everything in the book was up-to-date with respect to the new
release.
From a personal perspective, this was a particularly difficult year because we
also moved to a "new" house in April of 2008. Our new house is actually about
85 years old and hadn't been very well maintained for a while, so it needs some
work. The first week we moved in, we had the boiler go out, the sewer back up
into the basement, the toilet and the shower wouldn't stop running, the
electrical work for our office took longer than expected, our DSL wasn't hooked
up right, and about a million other things all seemed to go wrong. Somehow we
managed to cope with all of that, keep working for our real jobs, plus I was able
to finish my chapters for the book on schedule. I'm pretty sure
Sarah
wasn't expecting anything like this when I talked to her about working on the book
the previous December.
Needless to say, we're looking forward to a less hectic 2009.
If you are at all interested in OpenSolaris, then I hope you'll find something in our
book that is worthwhile, even if you already know a lot about the OS. The book is
targeted primarily at end-users and system administrators. It has
a lot of breadth and we tried to include a balanced mix of introductory material as well as advanced
techniques. Here's the table of contents so you can get a feel for whats in the book.
I. Introduction to OpenSolaris.
1. What Is OpenSolaris?
2. Installing OpenSolaris.
3. OpenSolaris Crash Course.
II. Using OpenSolaris
4. The Desktop.
5. Printers and Peripherals.
6. Software Management.
III. OpenSolaris File Systems, Networking, and Security.
7. Disks, Local File Systems, and the Volume Manager.
8. ZFS.
9. Networking.
10. Network File Systems and Directory Services.
11. Security.
IV. OpenSolaris Reliability, Availability, and Serviceability.
12. Fault Management.
13. Service Management.
14. Monitoring and Observability.
15. DTrace.
16. Clustering for High Availability.
V. OpenSolaris Virtualization.
17. Virtualization Overview.
18. Resource Management.
19. Zones.
20. xVM Hypervisor.
21. Logical Domains (LDoms).
22. VirtualBox.
VI. Developing and Deploying on OpenSolaris.
23. Deploying a Web Stack on OpenSolaris.
24. Developing on OpenSolaris.
If this looks interesting, you can pre-order a copy from Amazon
here. It comes out early next month, February 2009, and
I'm excited to hear peoples reaction once they've actually had a chance to look
it over.
Ahead of a full product review of
VirtualBox 2.1, eWEEK's Cameron
Sturdevant has published a slideshow on the offering. Check it out
here.
January 06, 2009
Back when posting your musings to your coworkers over office email was an affront to accepted standards of corporate behavior, it was a lot more appealing. Once the guys who signed your paycheck started encouraging you to blog, it lost some of its charm. After all what's the appeal of being naughty if it's no longer naughty?
A friend of mine asked me to dig this up from the pre-blogging archives. It's from the .com bust of 2000, it's about a startup that went under, it's got more than a few inside jokes, but in many ways it's apropos to recent events. I figured, why not?.
A SLICE OF THE PIE
- by Rick Ramsey and Don McLean, inspired by Jim
Engquist and commissioned by Jack Phillips, with help
from Michael Barton and Laura Ramsey.
A long, long time ago I can still remember how that
big sum used to make me smile.
And I knew if I had my chance
That I could make my broker dance
And maybe we'd be happy for a while.
But the CFO he made me shiver.
With each postponement he delivered,
Bad news on my desktop.
Announcements made my heart stop.
I can't remember if I cried
When I read about the market slide,
But something touched me deep inside,
The day, the IPO, died.
So...
CHORUS:
Bye, bye Mister Millionaire Guy!
Worked my ass off for the billions but the billions ran dry.
Them banker boys are making me cry
Singin' this may be the day that you vest,
But this won't be the day that you rest.
Did you buy that load of hype?
And did you decide to work all night?
If a VP asked you to.
Now do you believe in rent control?
Can foodstamps save your mortal soul?
And can you teach me how to spend real slow?
Well I know you're hoping for the best
Cuz I saw you wearin' your green vest.
We all sure paid our dues,
But now we're singing the blues!
I was a lonely coder chasin bucks
With a Porsche traded for my pickup truck,
But I knew I was out of luck,
The day, the IPO, died.
I started singin...
CHORUS:
Bye, bye Mister Millionaire Guy!
Worked my ass off for the billions but the billions ran dry.
Them banker boys are making me cry
Singin' this may be the day that you vest,
But this won't be the day that you rest.
Now for four years we've been counting down
To the day when we might leave this town,
But that's not how it's gonna be.
Now we're forced to work for pork and beans
Instead of looking like James Dean
Dreaming of how it could have been.
While the market was looking nice
The stock lost its inflated price.
The S-1 was withdrawn.
Our suitors were all gone.
All those books we read by Moore
All those investors beating down our door
So we sing dirges all the more
Because, the IPO, died.
We were singin'...
CHORUS:
Bye, bye Mister Millionaire Guy!
Worked my ass off for the billions but the billions ran dry.
Them banker boys are making me cry
Singin' this may be the day that you vest,
But this won't be the day that you rest.
Reorg, reorg, in a summer downpour
Could we have just one VP more?
Fifty five and rising fast.
All these promotions are such a blast.
Do you think it helps if you arrive last?
Or should you learn to drive real fast?
Now the Fortune press was sweet perfume!
And the analysts played an upbeat tune.
We all got up to dance.
Oh but we never got the chance!
When marketing tried to take the field,
Engineering just refused to yield
Do you recall what was revealed,
the day, the IPO, died?
We started singin'...
CHORUS:
Bye, bye Mister Millionaire Guy!
Worked my ass off for the billions but the billions ran dry.
Them banker boys are making me cry
Singin' this may be the day that you vest,
But this won't be the day that you rest.
Oh and though we are all in first place,
Procrastination lost that race.
With no time left to start again.
So come on, hack be nimble, hack be quick.
Hack it fast for the latest fix.
Because hacking is the devil's only friend.
As I saw the market turn the page
My hands were clinched in fists of rage,
No angel born in the Valley
Could reproduce that market rally!
And as the flames climbed high into the night,
No one told us it would be alright.
I saw Satan laughing with delight
The day, the IPO, died.
He was singin'...
CHORUS:
Bye, bye Mister Millionaire Guy!
Worked my ass off for the billions but the billions ran dry.
Them banker boys are making me cry
Singin' this may be the day that you vest,
But this won't be the day that you rest.
I met a guy who shines my shoes
And I asked him for some market news
But he just frowned and shined away.
I went down to the sacred store
Where I'd heard the pitches years before
But the man there said my placement wouldn't pay.
And in the house the children screamed
The wife she cried, and alone I dreamed.
But not a word was spoken,
Our hopes they all were broken.
And the three beings we admire most
Meeker, E-Trade, and the Holy Ghost
They stole the champagne for the toast
The day, the IPO, died.
And they were singin'...
CHORUS:
Bye, bye Mister Millionaire Guy!
Worked my ass off for the billions but the billions ran dry.
Them executive boys are making us cry
Singin' this won't be the day that we rest ...
Back in Colorado. Sun. Hills. Gravel crunching under my feet. No roots, bugs, or humidity. Achilles tendon inflammation down. Time to get back into running. Ummm....weight is 230. As in Lbs. And that's holding my breath. I hate to think what I'd weigh if I exhaled. I DIE of SHAME!
I started out a little too fast in December, and wound up exhausting myself on the Falcon Loop. So I rested and started up again, slowly.
I'm currently running 4 days a week inside the Park (Perry Park). Two light days (about 30 minutes of running plus 15 minutes of walking) and two regular days (about 45-50 minutes of running plus some walking). One long day every other weekend.
Did I mention that about 10 years ago, when I used to run with "the boys" in Colorado I was disappointed because I couldn't find an official classification for my weight? 180 lbs was considered the Clydesdale class. 190 lbs was Super Clydesdale. Or maybe I made that up. In any case, there was no class for 200 lbs.
On light days I walk up the ravine and run down Cheyenne Road back to the house. I take Buster with me when I can, since I'm trying to teach him that other people get upset when he objectifies their pets as chewy snacks.
Yesterday the ravine had 5 inches of fresh snow on a combination of hardpack and bare earth. Slippery as hell. Wind was blowing like a bastard, too. The relative plane of Cheyenne Road was a relief. That road would be awesome on cross country skis. Maybe I'll get some for the winter of 2010. Too damn broke to afford cross country skis in 2009.
So what do you think they'd call a class for people over 220 lbs or -Gawd forbid- 230 lbs? Maybe I should consult with the Coca Cola Bottling Company.
I tried a little more of the POST method that Jerry Jackson recommended. Jerry used to entertain himself at lunch by tossing the caber. Now he runs. I'm not sure he performs feats of strength any more, but whatever you do, don't let him grab you by the throat. At first it was kinda jarring, but I slowly learned to cushion the impact. Gonna keep experimenting with it. Jerry Jackson looks like he's running on air.
Also doing 6 s l o w pullups, now. If I weren't such a FAT bastard, I might do 7 or 8. Goal is 10. Ten slow pullups in one set. Hard going. But my pullup bar in the garage ROCKS. I can reach it if I stand on the tips of my tippie-toes.
Cloud Computing has turned out to be the hottest trend for 2008.
Google searches
for the term skyrocketed (with Bangalore, India and San Jose, CA
leading the trend!).
Several interesting announcements helped:
IBM,
Google/Salesforce
deal,
Yahoo/Intel/HP
alliance,
VMware, ,
Microsoft's
Azure, and Sun all announced major new activities.
Despite
some
significant glitches which didnt go unnoticed, all indications are
that this trend is here to stay. Or, more accurately,
grow
in 2009, judging from these 8 IT analysts who all had Cloud Computing
in their Top 5 predictions for 2009.
Now, thats saying
something!
Inspired by Constantin's comment on USB sticks wearing out Matthias's blog entry about an eco-friendly home server, I tried to find out more about how and how often the ZFS uberblock is written.
Using DTrace, it's not that difficult:
We start by finding out which DTrace probes exist for the uberblock:
$ dtrace -l | grep -i uberblock
31726 fbt zfs vdev_uberblock_compare entry
31727 fbt zfs vdev_uberblock_compare return
31728 fbt zfs vdev_uberblock_load_done entry
31729 fbt zfs vdev_uberblock_load_done return
31730 fbt zfs vdev_uberblock_sync_done entry
31731 fbt zfs vdev_uberblock_sync_done return
31732 fbt zfs vdev_uberblock_sync entry
31733 fbt zfs vdev_uberblock_sync return
34304 fbt zfs vdev_uberblock_sync_list entry
34305 fbt zfs vdev_uberblock_sync_list return
34404 fbt zfs uberblock_update entry
34405 fbt zfs uberblock_update return
34408 fbt zfs uberblock_verify entry
34409 fbt zfs uberblock_verify return
34416 fbt zfs vdev_uberblock_load entry
34417 fbt zfs vdev_uberblock_load return
So there are two probes on uberblock_update: fbt:zfs:uberblock_update:entry and fbt:zfs:uberblock_update:return!
Now we can find out more about it by searching the OpenSolaris sources: When searching for definition of uberblock_update in project onnv, we find one hit for line 49 in file uberblock.c, and when clicking on it, we see:
Now, when searching again for the definitions of the first two arguments (args[0| and args[1|) of uberblock_update (which is uberblock and vdev), we get:
For uberblock, the following hits are shown:

When clicking on the link on the definition of struct uberblock (around line 53 in file uberblock_impl.h), we get:
For the members of struct vdev, it's not that easy. First, we get a long hit list when searching for the definition of vdev in the source browser. But if we search for "struct vdev" in that list, using the browser's search function, we get:

When clicking on the definition of struct vdev (around line 108 in file vdev_impl.h), we can see all the members of this structure.
Here are all the links, plus one more for struct blkprt (a member of struct uberblock), again in one place:
Now we are prepared to access the data via DTrace, by printing the arguments and members as in the following example:
printf ("%d %d %d", args[0]->ub_timestamp, args[1]->vdev_id, args[2]);
So a sample final DTrace script to print out as much information in the event of an uberblock_update as we can, and also print out any relevant I/O (hoping that from showing both at the same time, we can see where and how often the uberblocks are written):
io:genunix:default_physio:start,
io:genunix:bdev_strategy:start,
io:genunix:biodone:done
{
printf ("%d %s %d %d", timestamp, execname,
args[0]->b_blkno, args[0]->b_bcount);
}
fbt:zfs:uberblock_update:entry
{
printf ("%d %s, %d, %d, %d, %d", timestamp, execname,
pid, args[0]->ub_rootbp.blk_prop, args[1]->vdev_asize, args[2]);
}
The lines for showing the I/O are derived from DTrace scripts for I/O analysis in the DTrace Toolkit.
Although I was unable to print out members of struct vdev (the second argument to uberblock_update() ) with the fbt:zfs:uberblock_update:entry probe (I also tried fbt:zfs:uberblock_update:return but had other problems with that one), the results when running this script, using
$ dtrace -s zfs-uberblock-report-02.d
, are quite interesting. Here's an extract (long lines shortened):
0 33280 uberblock_update:entry 102523281435514 sched, 0, 922..345, 0, 21005
0 5510 bdev_strategy:start 102523490757174 sched 282 1024
0 5510 bdev_strategy:start 102523490840779 sched 794 1024
0 5510 bdev_strategy:start 102523490873844 sched 18493722 1024
0 5510 bdev_strategy:start 102523490903928 sched 18494234 1024
0 5498 biodone:done 102523491215729 sched 282 1024
0 5498 biodone:done 102523491576878 sched 794 1024
0 5498 biodone:done 102523491873015 sched 18493722 1024
0 5498 biodone:done 102523492232464 sched 18494234 1024
...
0 33280 uberblock_update:entry 102553280316974 sched, 0, 922..345, 0, 21006
0 5510 bdev_strategy:start 102553910907205 sched 284 1024
0 5510 bdev_strategy:start 102553910989248 sched 796 1024
0 5510 bdev_strategy:start 102553911022603 sched 18493724 1024
0 5510 bdev_strategy:start 102553911052733 sched 18494236 1024
0 5498 biodone:done 102553911344640 sched 284 1024
0 5498 biodone:done 102553911623733 sched 796 1024
0 5498 biodone:done 102553911981236 sched 18493724 1024
0 5498 biodone:done 102553912250614 sched 18494236 1024
...
0 33280 uberblock_update:entry 102583279275573 sched, 0, 922..345, 0, 21007
0 5510 bdev_strategy:start 102583540376459 sched 286 1024
0 5510 bdev_strategy:start 102583540459265 sched 798 1024
0 5510 bdev_strategy:start 102583540492968 sched 18493726 1024
0 5510 bdev_strategy:start 102583540522840 sched 18494238 1024
0 5498 biodone:done 102583540814677 sched 286 1024
0 5498 biodone:done 102583541091636 sched 798 1024
0 5498 biodone:done 102583541406962 sched 18493726 1024
0 5498 biodone:done 102583541743494 sched 18494238 1024
Using the following (n)awk one-liners:
$ nawk '/uberblock/{print}}' zfs-ub-report-02.d.out
$ nawk '/uberblock/{a=0}{a++;if ((a==2)){print}}' zfs-ub-report-02.d.out
$ nawk '/uberblock/{a=0}{a++;if ((a>=1)&&(a<=5)){print}}' zfs-ub-report-02.d.out
, we can print:
- only the uberblock_update lines, or
- just the next line after the line that matches the uberblock_update entry, or
- all 4 lines after that entry, including the entry itself.
When running the script for a while and capturing its output, we can later analyze at which block number the first block after uberblock_update() is written, and we can see that the numbers are always even, the lowest number is 256 and the highest number is 510, with a block size of 1024. Those block numbers always go from 256, 258, 260, and so forth, until they reach 510. Then, they start with 256 again. So every (510-256)/2 = 127th iteration, the first block is overwritten again. The same is true for blocks 768...1022, 18493696...18493950 and 18494208...18494462 (the third and fourth block ranges should be different for different zpool sizes).
Now that we understand how and in which order the uberblocks are written, we are prepared to examine after how many days the uberblock area of a USB stick without wear leveling would probably be worn out. More on that and how we can use zdb for that, in my next blog entry.
Some more links on this topic:
Seit etwas mehr als einem Jahr gibt es eine kleine und bescheidene OpenSolaris User Group in Berlin (BROSUG), die sich monatlich zu Vorträgen und einer angeregten Diskussion trifft. Als Lokalität nutzen wir den Newthinking Store in der berliner Tucholskystr. 46, den wir dankenswerter Weise kostenlos nutzen dürfen. Vielen Dank dafür.
Am 7.1.2009 28.1.2009 wird Ulrich Gräf von Sun über ZFS berichten. Mehr Details zu dem Vortrag und zum Programm der BROSUG siehe auf dieser Projektseite.
In München und Hamburg sind nun ebenfalls OpenSolaris User Gruppen im Entstehen.
- Hier gibt es die Seite der Münchener Gruppe.
- Jörg hat hier seine Gedanken zu der Hamburger Initiative aufgeschrieben.
The last couple of weeks before the holidays I worked on an interesting project. It involved assembling pretty much everything Sun offers for HPC into a single coherent demo and throwing in Amazon EC2 to boot. This post will explain what I did and how I did it. Let's start at the beginning.
One of the new offerings from Sun is the Sun HPC Software. Beneath the excessively generic name is a complete, integrated stack of HPC software components. Currently there are two editions: the Sun HPC Software, Linux Edition (aka Project Giraffe) and the Sun HPC Software, Solaris Developer Edition. (A Sun HPC Software, Solaris Edition and Sun HPC Software, OpenSolaris Edition will be following shortly.) The Linux edition is exactly what the name implies. It's a full stack of open source HPC tools bundled into a Centos image, ready to push out to your cluster. The Solaris developer edition is a slightly different animal. It is targeted at developers interested in writing HPC applications for Solaris. The Solaris developer edition is a virtual machine image (available for VMware and Virtual Box) that includes Solaris 10 and a pre-installed suite of Sun's HPC products, including Sun Grid Engine, Sun HPC ClusterTools, Sun Studio, and Sun Visualization, all integrated together.
For this demo, I used the Solaris developer edition. The end goal was to produce a version of the virtual machine image that was capable of automatically borrowing resources from a local pool or from the cloud in order to test or deploy developed HPC applications. Inside the developer edition virtual machine, there are already two Zones that act as virtual execution nodes for testing applications. That's a nice start, but what about testing on real machines or a larger number of machines? That's where the resource borrowing comes in. In the end, I had a VM image that was capable of automatically borrowing and releasing resources first from a local pool and later from the cloud, on demand.
The first step was to get the developer edition running as-is. Sounded simple enough. The first wrinkle was that I was doing this demo on a Mac. The regular VMware Player is not available for Mac, so I had to download an eval copy of VMware Fusion. Once I had Fusion installed, I was able to bring up the developer edition VM without a hitch.
Step 2 was to get the VM networked. The network configuration for the developer edition beta 1 is such that the global and non-global Zones can see each other, but nobody can get into or out of the VM. Getting the networking working was probably the hardest part of the demo, and honestly, I can't tell you how I finally did it. Per the suggestion of the pop-up dialogs from VMware, I installed the VMware Tools in the VM's Solaris instance. That changed the name of the primary interface from pcn0 to vmxnet0, but didn't actually help. Solaris was still unable to plumb the interface. After twiddling the VM's network settings several times and doing several reconfiguration boots, I eventually ended up with a working vmxnet1 interface (and a dead pcn0 and vmxnet0). As usual in such adventures, I'd swear that the last thing I did before it started working should not have had any appreciable effect. Oh, well. It worked, and I wasn't interested in understanding why.
Now that I had a functional network interface, the next step was to reinstall the Sun Grid Engine product. The VM comes with a preinstalled instance, but this demo requires features not enabled in a default installation, like what the VM provides. I left the original cell (default) intact and installed a new cell (hpc) with the -jmx and -csp options. -jmx enables the Java thread in the qmaster that serves up the JGDI API over JMX. I needed JGDI so that the demo GUI that I was building could receive event updates from the qmaster about job and host changes. With Sun Grid Engine 6.2, I was unable to successfully connect to the JMX server unless I installed the qmaster with certificate-based security, hence the -csp option. After the installation was complete, I then had to do the usual CSP certificate juggling, plus a new wrinkle. In order to connect to the JMX server, I also had to create a keystore for the connecting user with $SGE_ROOT/util/sgeCA/sge_ca -ks <user>. There's a quirk to the sge_ca -ks command, though. By default, it fails, explaining that it can't find the certificates. The reason is that the path to the certificates is hard-coded in the sge_ca script to a ridiculous default value. To change it to the correct value, I had to use the -calocaltop switch. After the certificates were squared away, I installed execution daemons in both Zones. At least that part was easy.
The next thing I did was to create some more Zones. Yes, I know this demo was supposed to be using real machines from a local pool and the cloud. Because it's a demo on a laptop, the "local machines" had to be equally portable. Because of firewall issues, I also wanted to have a backup for the cloud. In an effort to be clever, I moved the file systems for the two existing Zones onto their own ZFS volumes. I wanted to create the new Zones as cloned snapshots of the old Zones. Unfortunately, it turns out that even though the man page for zfs(1M) says that it's possible, the version of Solaris installed in the VM is the last version on which it isn't possible. After chasing my tail a bit, I decided to just do it the old fashioned way instead of trying to force the new fangled way to work.
Now that I had six non-global Zones running, the next step was to get Service Domain Manager installed. It is neither installed nor included in the developer edition VM, so I had to scp it over from my desktop. Technically, I could probably have managed to download it directly from the VM, but I had already downloaded it to my desktop before I started. For the Service Domain Manager installation, I followed Chansup's blog rather than the documentation. Chansup's blog posts detail exactly what steps to follow without the distraction of all the other possibilities that the docs explain. Following the steps in the blog, I was able to get the Service Domain Manager master and agents installed with little difficulty. The hardest part is that the sdmadm command has extremely complicated syntax, and it took a while before I could execute a command without having the docs or blog in front of me as a reference. To prove that the installation worked, I manually forced Service Domain Manager to add one of the new Zones to the existing Sun Grid Engine cluster, and much to my shock and wonderment, it worked.
The last step of VM (re)configuration was to configure the Service Domain Manager with a local spare pool and a cloud spare pool and a set of policies to govern when resources should be moved around. This step proved about as tricky as I expected. As one of the original architects and developers of the product, I had a good idea of what I wanted to do and how to make it happen, but the syntax and the details were still problematic. The syntax was the first hurdle. The docs have issues with both understandability and accuracy, and Chansup's blog was too narrowly focused for my purposes. After I poked around a bit, I figured out how to do what I wanted, but actually doing it was the next challenge. What I wanted to do was create two MaxPendingJobsSLO's...
We interrupt your regularly scheduled blog post to bring you a public service announcement. Please, for your own well being and the well being of others who might use your software, test all of your code contributions thoroughly on all supported platforms, and have them reviewed by an experienced member of the development team before committing, especially if you're working on the Firefox source base. This point in the blog post is the last time I saved my text before completing the post. Before I could save it, Firefox segfaulted causing me to loose a significant amount of work. What follows is a downtrodden, half-hearted attempt to complete the post again. We now return you to your regularly scheduled blog post.
What I wanted to do was create two MaxPendingJobsSLO's for the Sun Grid Engine instance. The first would post a moderate need (50) when the pending job list was more than 6 jobs long. The second would post a high need (99) when the pending job list was more than 12 jobs long. I also wanted to have a local spare pool with a low (20) PermanentRequestSLO and a low FixedUsageSLO, and a cloud spare pool with a moderate (60) PermanentRequestSLO and a moderate FixedUsageSLO. The idea was that when the Sun Grid Engine cluster was idle, all the resources would stay where they were. When the pending job list was longer than 6 jobs, resources would be taken from the local spare pool. When the pending job list was longer than 12 jobs, additional resource would be taken from the cloud spare pool. When the pending job list grew shorter, the resources would be returned to their spare pools. In theory. (The philosophy of setting up Service Domain Manager SLOs is a full topic unto itself and will have to wait for another blog post.)
The first problem I ran into was that Service Domain Manager does not allow a spare pool to have a FixedUsageSLO. An issue has been filed for the problem, but that didn't help me set up the demo. The result was that I had no way to force Service Domain Manager to take the local spare pool resources before the cloud spare pool resources. The best I could do was set the averageSlotsPerHost value for the SLO for the MaxPendingJobsSLO's to a high number so that Service Domain Manager only would take hosts one at a time, rather than one from each spare pool simultaneously.
The nest problem was quite unexpected. With the SLOs in place, I submitted an array job with 100 tasks. I waited. Nothing happened. I waited some more. Still nothing happened. I turns out that the MaxPendingJobsSLO only counts whole jobs, not job tasks like DRMAA would. The work-around was easy. I just had to be sure the demo submitted enough individual jobs instead of relying on array tasks.
The last problem was one that I had been expecting. After a long pending job list had caused Service Domain Manager to assign all the available resources to the cluster, when the pending job list went to zero, the borrowed resources didn't always end up where they started. Service Domain Manager does not track the origin of resources. Fortunately, the issue is resolved by an easy idiom. I created a source property for every resource, and I set the value of the property to either "cloud", "spare", or "sge". I then set up the spare pools' PermanentRequestSLO's to only request resources with appropriate source settings. I also added a MinResourceSLO for the cluster that wants at least 2 resources that didn't come from a spare pool, just to be complete.
With the SLOs in place, the configuration actually did what it was supposed to. When the cluster had enough pending jobs, hosts were borrowed first from the local spare pool and then from the cloud. When the pending jobs were processed, the resources went back to the appropriate spare pools. To make the configuration more demo-friendly, I changed the sloUpdateInterval for the Sun Grid Engine instance to a few seconds (from the default of a few minutes). I also changed the quantity for the spare pools' PermanentRequestSLO's to 1 so that they would only reclaim their resources one at a time, rather than all at once. With those last changes made, I was ready to move on to the UI.
The idea of the demo was to present a clear graphical representation of what was going on with Sun Grid Engine and Service Domain Manager. From past experience building a similar demo for SuperComputing, I knew that JavaFX™ Script was the best tool for the job. (OK. It's not the best tool for the job in a general sense, but I'm a long-time Java™ geek, I don't know Flash, and I didn't have any budget to buy tools. Under those constraints, it was the best I could do.) Before I could get to building the UI, though, I first needed a JGDI shim to talk to the qmaster. Richard kindly provided me with some JGDI sample code, and from there it was pretty easy. The hardest part was figuring out what the events actually meant. In the end, my shim registered for job add events (to recognize job submissions), task modified events (to recognize job tasks being scheduled), and job deleted events (to recognize job completions). It also registered for host added and deleted events to recognize when Service Domain Manager reassigned a host.
With the shim working smoothly, I turned to the actual UI. Given the complexity of the animations that I wanted to do, it was shockingly simple to achieve with JavaFX Script, especially considering that there was not yet a graphical tool equivalent to Matisse for Swing. Every bit of it was hand-coded, but it still was fast, easy, and came out looking great. In the end, the whole UI, counting the shim, was about 1500 lines of code, and about 500 lines of that was the shim. (JGDI is rather verbose, especially when establishing a connection to the qmaster.)
And with that, I ran out of time. The next step would have been to actually populate the cloud spare pool with machines provisioned from the cloud. Torsten graciously provided me a Solaris AMI that included Sun Grid Engine and Service Domain Manager. The plan was to pre-provision two hosts to populate the pool and then create a script that would provision an additional host each time the cloud pool dropped below two hosts and release a host every time it grew larger than two hosts. Now that the demo has been presented, the pressure is off, and other things are higher priority. I do plan, however, to eventually come back and put the last piece of the puzzle in place.
Below is a video of the demo, showing how jobs can be submitted from the Sun Studio IDE, and how Sun Grid Engine and Service Domain Manager work together with the local spare pool and the cloud to handle the workload. The job that is being submitted is a short script that submits eight sleeper jobs. Because the MaxPendingJobsSLO ignores array tasks, I needed to submit a bunch of individual jobs, but I didn't want to have to click the submit button multiple times in the demo.
Filming the video turned out to be an interesting challenge unto itself. I did the screencap using Snapz Pro on the Mac. It has no problem with JavaFX Script or with VMware VMs, but it apparently can't film JavaFX Script running inside a VMware VM. I ended up having to twiddle the UI a bit so that I could run it directly on the Mac. That's why in the demo, when I switch from Sun Studio to the UI, I swap Mac desktops instead of Solaris workspaces. The voice over and zooming effects are courtesy of Final Cut, by the way.
An
Open Government Workshop to be held at MIT on January 15th, will address the role of Digital Identity as a key enabler for effective interaction between citizens and government leaders.
This workshop is being organized by Dazza Greenwood of
Civics.com on behalf of the
MIT eCitizen Architecture Program, the
MIT Media Lab SmartCities Group and the
eCitizen Foundation.
The Digital Identity part of the workshop is being directed by Bruce Bakis of Mitre Corporation, Team Leader of the
Safeguarding Digital Identity research project for the
Institute for Information Infrastructure Protection. In an invitation to Identity Management community, Bruce stated:
"Several goals in the Obama-Biden technology agenda articulated at change.gov fit right into our Digital Identity wheelhouse. Two of these really hit our sweet spot: Create a Transparent and Connected Democracy, and Lower Health Care Costs by Investing in Electronic Information Technology Systems.
"So, here’s what we’re doing: holding several virtual events and one “real” one to compile and present to the Obama-Biden administration a prioritized list of issues, problems and questions. During the “real” event we will hold three interrelated discussions:
- The use of Digital Identity as a key enabler (for the other two agenda items and so much more)
- How to Create a Transparent and Connected Democracy that’s open, effective, privacy preserving and secure;
- How to Lower Health Care Costs by Investing in Electronic Information Technology Systems."
You can register
here for participation in the January 15th event. You can participate in formulation of the dialogue that will occur at the main event by using this
online forum to submit and rank questions to be addressed during the event.
Technorati Tags:
Identity,
Identity Management,
Digital Identity,
eGovernment
I've been playing with VirtualBox a lot in the last month; I'm hoping to switch from VMware Workstation on my Ubuntu box and VMware Fusion on my Mac Book Pro. Don't get me wrong; I've been using VMware for years, but now that there's a free solution that seems to work well, I thought I'd try it. Plus, VirtualBox works on Solaris and OpenSolaris, so I can use it on my home fileserver to run Windows-based services on the same box. Nice.
One of the great things about these virtual machines is that your whole environment is just a single file on disk. Want to do play around with your operating system? No problem; just make a backup copy of the .vdi file (or in VMware land, your .vmdk file), which is your OS disk. I like to create a VM, install my standard Windows XP OS and applications, then make a copy of that image so that later, when the OS gets corrupted by viruses and what not, I can go back to a nice, safe working state.
There's an easy way to make copies of your virtual machine in VirtualBox. If your virtual machine's hard disk is called something like "WindowsXP.vdi", just type "VBoxManage clonevdi WindowsXP.vdi copyOfWinXP.vdi" and VirtualBox will create a nice copy of your Windows image, saving it in a file called "copyOfWinXP.vdi".
Except that it doesn't work with VirtualBox 2.1.0. There's a bug,
bug #2813, that prevents the cloning from working exactly right. The result is that you'll have problems booting up your virtual machine. The bug is known and a fix is already checked into the source tree for a future version, but until then, here's the workaround:
- copy your original VDI file using normal file copy mechanisms (e.g., "cp WindowsXP.vdi copyOfWinXP.vdi")
- type "VBoxManage internalcommands sethduuid copyOfWinXP.vdi"
You can now safely use this new VDI file in VirtualBox. Launch VirtualBox, open the Virtual Media Manager, and tell it to add this "copyOfWinXP.vdi" (or whatever you're calling your image). Then when you create a new virtual machine within VirtualBox, you can tell that virtual machine to use this new VDI, no problem.
Powered by ScribeFire.
Twitter is a
micro-blogging service that allows its users to send and read other
users's updates which are text-based posts upto 140 characters in
length. This
Tip
Of
The
Day will show how to
add Twitter feeds to your blog but first some facts about Twitter:
The
State
of Twittersphere Q4 2008 was released recently (generated
using
Tweet Grader)
and shows:
- Total of 4-5 million users
- 70% users joined in 2008, 20% joined in past 60 days, 5-10
new accounts/day (so mostly noobies)
- Only 5% have 250 or more followers
- Only 0.8% have 1000 or more followers
- 35% tweeters have 10 or fewer followers
- 9% tweeters follow no one at all
Here are
Tweet
Grader statistics generated for
@arungupta:
Lets add twitter feeds to your blog:
- Go to twitter.com/widgets
and select Other as shown below:

It allows you to create Flash and HTML widgets for any web page.
Twitter feeds can be directly added to some other popular social
networking websites by selecting their option.
- Click on "Continue" and choose between Flash or HTML widget
as shown below:

- Click on "Continue" and feel free to customize as shown
below:

"Number of updates" is the number of tweets that will be shown and
title can be changed as well.
- For blogs.sun.com, this obtained code fragment can be
easily added to "_sideColumn" template.
Here are some other nifty tools to manage your Twitter account:
- Twitterfeed:
Twitter your blog entries
- Twitterank:
Page rank for Tweeters (85.01 as of this writing, live here)
- Twinfluence:
Combined influence of twitterers and followers (as of
Jan 2)
- TwitterCounter:
Widget that shows twitterer follower
- TwitterFox:
Firefox extension that notifies of tweets (really sweet and lots
of other related plugins)
- SocialToo:
Manage online interactions of Twitter (surveys, auto-follow/unfollow,
Direct Messages to new followers)
- TweetStats:
Trends and analysis as shown below:

Live stats for @arungupta
are available here.
- TweetLater:
Schedule tweets for a particular time/day and some other features
similar to SocialToo (auto-follow/unfollow and DM to new followers)
- Easy Tweets:
Schedule tweets in future, automatically post RSS feeds and other
similar features
There
are
numerous
other
tools
available
to
manage your Twitter account! What is your favorite ?
Please leave suggestions on other TOTD (
Tip
Of
The
Day) that
you'd like to see.
An archive of all the tips is available
here.
Technorati: twitter
microblogging
blogs bsc milestogo
Are the tigers jumping out of the fish's mouth or is the fish about to swalow them?
A completely random thought: is there a name for the motion of a
carousel horse?
One of those would be informative and impish:
- Donusoidal
- Torusoidal
[For you overly serious types
- Yes, I know that both of those
answers are incorrect because 'donut' and 'torus' refer
to 3-dimensional surfaces. My question and answers are not meant
to be geometrically correct. However, if there really is a term describing
the combination of sinusoidal and
circular motions of a carousel horse, I would like to know what it is.]
One of the top 10 features of Egypt is, of course,
the shisha (more specifically, the contents thereof). Here is me last night outside a cafe in Cairo with Hamada, Amr, and Khaled, applying the default shisha usecase:
The discussion obviously turned to the need for tight integration between shisha and NetBeans IDE. Initial brainstorms around powering NetBeans IDE via a shisha were briefly assessed and then rejected as brilliant but impractical. Then, as if out of thin air, the concept of a NetBeans plugin for shishas was born. Here is a first implementation—shisha cursor:
package org.netbeans.shisha;
import java.awt.Cursor;
import java.awt.Image;
import java.awt.Point;
import java.awt.Toolkit;
import javax.swing.ImageIcon;
import javax.swing.JFrame;
import javax.swing.SwingUtilities;
import org.openide.modules.ModuleInstall;
import org.openide.windows.WindowManager;
public class Installer extends ModuleInstall {
@Override
public void restored() {
SwingUtilities.invokeLater(new Runnable() {
public void run() {
//Create the cursor:
Toolkit tk = Toolkit.getDefaultToolkit();
ImageIcon shishaImageIcon = new javax.swing.ImageIcon(getClass().getResource("/org/netbeans/shisha/shisha.png"));
Image shishaImage = shishaImageIcon.getImage();
Cursor shishaCursor= tk.createCustomCursor(shishaImage, new Point(10,10), "Shisha");
//Use the cursor in the main window:
JFrame frame = (JFrame) WindowManager.getDefault().getMainWindow();
frame.setCursor(shishaCursor);
}
});
}
}
And this is how the source structure looks, together with the shisha cursor installed into the IDE:
Even tighter integration can be achieved by setting the shisha cursor in the editor too, by changing the caret to use the same icon. Further steps could involve extending the Options window with a selection of different shishas from which the user can select their preferred model. In this case, unfortunately, the content thereof is irrelevant.
I thought it would be useful to post the LDoms 1.1 README.
It contains useful information for
documentation and required software, especially required System Firmware for supported servers.
This is the Logical Domains (LDoms) 1.1 README included in the zip download file:
# Copyright 2008 Sun Microsystems, Inc. All rights reserved.
# Use is subject to license terms.
#
Documentation
=============
Please see the Logical Domains Admin Guide for instructions on
how to install & use LDoms.
Documentation available on http://docs.sun.com/app/docs/prod/ldoms
Logical Domains (LDoms) 1.1 Administration Guide 820-4913
Logical Domains (LDoms) 1.1 Release Notes 820-4914
Logical Domains (LDoms) Manager 1.1 Man Page Guide 820-4915
Logical Domains (LDoms) Management Information Base (MIB) 1.0.1
Administration Guide 820-2319
Logical Domains (LDoms) Management Information Base (MIB) 1.0.1
Release Notes 820-2320
Please see the Libvirt Admin Guide for instructions on how to
install & use Libvirt.
Libvirt for LDoms 1.0.1 Administration Guide 820-3838
Libvirt for LDoms 1.0.1 Release Notes 820-3839
This download also contains the SUNWjldm.v package, which installs
the localized Japanese man page for the ldm(1M) command.
Japanese locales supported: ja, ja_JP.eucJP, ja_JP.PCK, ja_JP.UTF-8
This package is found in the Product/Japanese directory.
Software
========
In order to avail of all the features in LDoms 1.1, the Operating System on all
domains should be equivalent to Solaris 10 10/08 at a minimum. This can either
be a fresh/upgrade install of Solaris 10 10/08 or any of Solaris 10 5/08,
Solaris 10 8/07 or Solaris 10 11/06 with 137137-09 applied.
In addition, the patches listed below are required. It lists what types of
domain each patch is required to be applied to but the patches can be applied
to all domains.
Patch Control Service/IO Guest
Domain Domain Domain
139562-02 (multiple LDoms drivers) X X X
139570-02 (nxge driver) X X X
139508-01 (niumx driver) X X X
139502-01 (picl plugin) X
139458-01 (aggr driver) X X
Following is a matrix of required software to enable all the
Logical Domains 1.1 features:
Supported Servers System Firmware Solaris OS
Sun UltraSPARC T2 Plus-based 7.2 One of the configs above
Sun UltraSPARC T2-based 7.2 One of the configs above
Sun UltraSPARC T1-based 6.7 One of the configs above
It is possible to run the Logical Domains 1.1 software along with previous
revisions of the other software components. For example, you could have
differing versions of the Solaris OS on the various domains in a machine.
It is recommended to have all domains running Solaris 10 10/08 OS plus
patches listed above, but an alternate upgrade strategy could be to upgrade
the control and service domains to Solaris 10 10/08 OS plus patches list above
and to continue running the guest domains at the existing patch level.
Following is a matrix of the minimum version of software required.
The minimum software versions are platform specific and depend on the
requirements of the CPU in the machine. The minimum Solaris OS version for a
given CPU type applies to all domain types (control, service, I/O, and guest).
Supported Servers System Firmware Solaris OS
Sun UltraSPARC T2 Plus-based 7.1.x Solaris 10 8/07(1)
Sun UltraSPARC T2-based 7.0.x Solaris 10 8/07
Sun UltraSPARC T1-based 6.5.x Solaris 10 11/06(2)
(1) Must also use Patch ID 127111-08 at a minimum
(2) Must also use Patch IDs 124921-02, 125043-01, and KU 118833-36 at a minimum
Sun System firmware available on http://sunsolve.sun.com
The Libvirt and MIB software for LDoms 1.0.1 also supports LDoms 1.1.
In addition to the software packages for Libvirt and virt-install,
the source is included in this download.
The Libvirt source can be found in the Product/Libvirt-source directory.
The virt-install source and the binaries are the same.
Here are the default locations for the virt-install files:
/usr/lib/python2.4/vendor-packages/urlgrabber/__init__.py
/usr/lib/python2.4/vendor-packages/urlgrabber/byterange.py
/usr/lib/python2.4/vendor-packages/urlgrabber/grabber.py
/usr/lib/python2.4/vendor-packages/urlgrabber/keepalive.py
/usr/lib/python2.4/vendor-packages/urlgrabber/mirror.py
/usr/lib/python2.4/vendor-packages/urlgrabber/progress.py
/usr/lib/python2.4/vendor-packages/urlgrabber/sslfactory.py
/usr/lib/python2.4/vendor-packages/virtinst/DistroManager.py
/usr/lib/python2.4/vendor-packages/virtinst/FullVirtGuest.py
/usr/lib/python2.4/vendor-packages/virtinst/Guest.py
/usr/lib/python2.4/vendor-packages/virtinst/Makefile
/usr/lib/python2.4/vendor-packages/virtinst/ParaVirtGuest.py
/usr/lib/python2.4/vendor-packages/virtinst/__init__.py
/usr/lib/python2.4/vendor-packages/virtinst/util.py
/usr/sbin/virt-install
Flag-Day for configurations
---------------------------
Existing LDoms 1.0 configurations do not work in LDoms 1.1 software.
See the "Upgrading to LDoms 1.1 Software" section in the Logical Domains
(LDoms) 1.1 Administration Guide to migrate existing configurations.
LDoms 1.1 was released just before Christmas and its available for
download here!
LDoms 1.1 greatly expands the capabilities and performance of Sun servers with CoolThreads technology (UltraSPARC T1, UltraSPARC T2, and UltraSPARC T2 Plus systems).
LDoms 1.1 adds new features, such as:
- Warm and Cold Migration
- Network NIU Hybrid IO
- VLAN Support for Virtual Network Interface and Virtual Switch
- Public XML Interface and XMPP Connection with the Domain Manager
- Virtual IO Dynamic Reconfiguration
- Virtual Disk Multipathing and Failover
- Virtual Switch Support for Link Aggregated Interfaces
- iostat(1M) Support in Guest Domains
- And dozens of other product improvements
I recommend reading Alex Chartre's LDoms 1.1 post for more specifics on the new features and for usage examples.
Included in the download are optional LDoms System Management Tools:
- Ldoms Management Information Base (MIB)
- libvirt for LDoms
- Logical Domains (LDoms) 1.1 Administration Guide (html)(pdf)
- Logical Domains (LDoms) 1.1 Release Notes (html)(pdf)
- Logical Domains (LDoms) Man Page Guide (pdf)
- Logical Domains (LDoms) Management Information Base (MIB) 1.0.1 Administration Guide (html)(pdf)
- Logical Domains (LDoms) Management Information Base (MIB) 1.0.1 Release Notes (html)(pdf)
- Libvirt for LDoms 1.0.1 Release Notes (html)(pdf)
- Libvirt for LDoms 1.0.1 Administration Guide (html)(pdf)
See the README and/or Release Notes for required Software and patches
The key components needed to utilize the LDoms technology are:
- Sun servers with CoolThreads technology (UltraSPARC T1, UltraSPARC T2, and UltraSPARC T2 Plus systems) with required system firmware.
- Solaris 10 OS
- Logical Domains Manager software that is used to create and manage logical domains on CoolThreads servers.
Support
Developer Blogs
Bruno Bonfils, aka asyd, longtime denizen of #opensso, has put together a VMware instance with OpenSSO and EJBCA. In Bruno's words:
The image was made to demonstrate an application protected by opensso. The application is divided in three parts, the first one is available for everyone (non authenticated users). The second part, the secure area, is available only for users authenticated in OpenSSO, and members of group employee. And finally, only users authenticated by certificates and member of group employee can access to the very secure area.
Watch for a series of articles at Sun Developer Network (feed) describing the integration in more depth; in the meantime, you can go download the instance and have a play - instructions are included...
There are so many events in life which one can observe
with little reaction or response. We can write
them off easily as someone else's reality, things we
can't change, and things that simply don't matter....
whatever the reason.
And then, there are some observations which do impact
us. Things that color how we see the world & life.
Things that change our opinions. Things that cause
us to drop whatever we're doing and radically shift
gears.
Recently a friend & work colleague suffered a heart
attack. He is doing really well now and is making
changes to his lifestyle.
Surprisingly, his crisis had an impact on me
personally.
It reminded me that each of our lives are finite
in length. It reminded me to look at my own life
and, as one Sun VP, Diann Olden, would say, "Inspect
what you expect"... assessing if my life path
is going in the right direction, in the direction I want.
And with that, over the holiday season, I realised that
all is about 98.5% good... starting with a
superbly wonderful marriage and happy homelife....
a very satisfying 33 year career in the computer
industry, the co-authoring with friend & Sun colleague
Chris Drake of
Panic!, a
highly successful technical book,
impressive successes in my "hobby" of horse training
& dressage competition, travel in 16 countries around
the world (mostly for customer site visits), excellent friends & families, both
here and abroad, and a wonderful home.
But, for the past few years, one more great big target
has been shining in the distance, tempting me to aim
for it. But, due to other committments, I
haven't even considered it. Until now.
Thanks to my friend's own brush with death, I can now
actually envision going after one more goal before
my own heart stops beating... Someday...
Someday soon.
Here's to 2009! May it be an interesting year!
I see attack politics and attack marketing as pretty much the same thing. Or, a distinction without much of a difference, anyway. Politicians generally attack enemies who threaten their getting elected or getting some policy implemented. If you aren't a threat, though, you are basically ignored in that system. And if you are a little guy trying to attack powerful politicians, you are generally ignored, too. This is why collective protest is a necessary prerequisite for change. Strength comes in numbers. You have to make yourself a threat to even get noticed, and that has to happen well before you have a shot at changing things (whatever your thing is). But from the politicians point of view, since they have the power, it seems the attack principle dictates that they shouldn`t want to give too much exposure to a competitor or group they don`t support, so many politicians actually tend to attack pretty carefully. The rhetorically skilled know this very well. They think out a few moves ahead. Who should do the attacking? What`s the venue of the attack? What will the counter punch look like? Where will it come from? And when? What does it mean when no counter attack comes back at all and instead they are met with silence? And heck, what if the opponent praises in return instead of attacking as expected? The answers to these questions are imprecise at best.
I used to do competitive marketing, and I went through this exact same process. However, I always told my clients that attacks are best done by third parties and only in response to a precipitating attack. In other words, you don`t attack first. It`s not worth the headline. Instead, you be the one responding. Here`s why: those who attack first generally give away at least some of their position, and that gives you much more flexibility to respond. Unskilled politicians and marketers make this mistake all the time when they shoot their mouths off, but the concept holds up pretty well over time. I`ve said before that I think people attack for basically two reasons: (1) they are afraid that someone smaller than them may grow up and kick their butt, or (2) they are small themselves and want to pick a fight with a big guy to get attention. Either way, if you study your attacker you can learn a lot.
It's a game, granted. And everyone in it knows this. Most attacks can be quite easily turned around with some basic facts and logic. But rationality is irrelevant in the arena of delivering really good emotional propaganda for the purpose of influencing behavior. That's why attacks can work in some cases if they generate a strong reaction from the attacked. Attacks spread fear. And many times that fear shapes how people think if it`s not characterized properly. In fact, the term used to describe this process is sometimes called FUD -- fear, uncertainty, and doubt. It`s a silly sounding term, but it should be taken seriously because the best propagandists out there can be rather dangerous people if they have a power base and resources supporting them (a country, a company, an interest group, a foundation, a university, a union, whatever). In other cases, however, attacks and fear mongering backfire badly, and we saw this in the recent political campaign in the U.S. where pols on both sides took some things too far and the people (remember the people?) called them out on it.
So, what should you do if you are attacked in the marketplace? First, stop. Think. Don`t react immediately with the first counter attack you can think of in the first publication you can find. You`ve been attacked so you now have the upper hand for a period of time (not forever, though). What is the attack telling you about your attacker? Is he or she responding go your attack? If so, you deserve the counter attack so enjoy your stupid little fight. If not, though, something else is going on and you may be in a much better position than you think. It means that you got someone`s attention for some reason. You may have not even intended to get this attention, but that`s what the attack may mean and that`s valuable competiti