Variable Scopes and $PRIVATE:

Posted on by
6

In my previous post Evolution of a Script, Timing Operations, I used the technique of putting “__” in front of variable names to avoid namespace collision.  Niklas Bergius left a comment saying, “Instead of using __ prefixes, wouldn’t $private: work?”   Great question – let’s explore.

The first thing I want everyone to get in focus is that there are lots of ways to accomplish something in PowerShell.  That means that you shouldn’t freak out of you see someone using a technique you don’t use.  The culture of PowerShell is one of exploration and fun.  When you see something new, go explore and find out it’s strengths and weaknesses for yourself.  I’ve always believed that the best way to learn a system is to use it – explore and have some fun.  While PowerShell is a power tool for admins, we put in a number of features that make it easy (and often safe) to explore (e.g. -Whatif).

So let’s explore $PRIVATE:.

What is the first thing we do when exploring something? We type HELP.  In this case, we say HELP PRIVATE (I didn’t provide the ‘$’ otherwise it would try to evaluate it as a variable).  This provides a list of potential help subjects including ABOUT_SCOPES.  $PRIVATE: is a variable SCOPE – you can tell that by the “:”.

So now we want to read the help and explore what it tells us.  The way I like to do this is to use ISE and pipe the help to the clipboard (help About_scopes | clip ) and then paste it into a new editor windows (CTRL-N and then CTRL-V).  Now I can read the help and have the interactive shell to explore.  You should spend some time here – there is a LOT to learn about Scopes.  PowerShell’s implementation of variables is very rich and different from other languages.

Side note: PowerShell separates syntax from implementation.  Variables are just places you store and retrieve data.  Traditionally this is a piece of volatile RAM but in PowerShell it can be anywhere.  For example:  ${C:\Temp\Test.txt} = “This is a test”  stores the string into a file using variable syntax.  This is a trick I learned implementing GT.M MUMPS systems (which were a combined language/database system) in the 1980s.

The help explains the concept of a scope (a mechanism which determines how a element [variable, alias, drive, function, etc.] will be made available [or not] to other elements of the program) and explains the different types of scopes: Global, Local, Script, Private, and Numbered.  The explanation of Private scopes says:

Items in private scope cannot be seen outside of the current scope. You can use private scope to create a private version of an item with the same name in another scope.

So let’s start exploring:
Let’s start with the following as a baseline to remind everyone how PowerShell variables work and the power of dynamic scoping:

  1. #start with a clean slate  
  2. Remove-Variable -Name x    
  3. $x = “1″  
  4. “x = ” + $x  
  5. function test  
  6. {  
  7.     “In test, x = ” + $x  
  8. }  
  9. test  

Running it produces the following output:
x = 1
In test, x = 1

What that means is that when we tried to access the variable X in the function test, it was not available so PowerShell looked into the parent’s scope, found the variable there and used it. Now let’s put $X into a PRIVATE scope and see how things change.

  1. #start with a clean slate  
  2. Remove-Variable -Name x   
  3. $private :x  = “1″  
  4. “x = ” + $x  
  5. function test  
  6. {  
  7.     “In test, x = ” + $x  
  8. }  
  9. test  

(Apologizes if that code shows an emoticon – I’m using a Syntax Highlighter which doesn’t work properly – any recommendations?)

This produces the following result:
x = 1
In test, x =

So what that tells us is that by using $Private, the variable is hidden from other parts of the program.  Niklas’ comment showed a clear understanding of scope and variable handling because if PowerShell doesn’t find the variable in the parent, it continues up the stack looking for the variable until it finds it or fails.  The problem I had with my script was that I wanted to use a variable name in one routine and have child routines be able to use that variable but bind it to it’s grandparents value.  Let’s go through that slowly.

I was trying to write a function Measure-Time which took a variable which I wanted to call $ScriptBlock which I would then execute by calling &$ScriptBlock.  That worked fine EXCEPT when the ScriptBlock refers to a variable named $ScriptBlock (and why wouldn’t it – If I find it useful, others will as well).  The problem is that that then evaluates to itself and bad things ensue.  So what we want to do is to somehow use the variablename ScriptBlock in a way that when ONLY my routine sees it.  When my routine calls a function that walks up the stack looking for a variable called ScriptBlock – I want it to pass right over me and look for it in my parent/grandparent/etc.  That is exactly what $PRIVATE: does.

If none of that makes any sense – don’t worry – lets explore a few things and I think you’ll get it via experimentation.  So here is the experiment (I’m going to switch variable names from X to ScriptBlock so it matches the text above):

  1. #start with a clean slate  
  2. Remove-Variable -Name ScriptBlock -ErrorAction Ignore  
  3. $ScriptBlock = “top”  
  4. “ScriptBlock = ” + $ScriptBlock  
  5. function child  
  6. {  
  7.     “In child,  ScriptBlock = ” + $ScriptBlock  
  8. }  
  9. function middle  
  10. {  
  11.     $ScriptBlock = “middle”  
  12.     “In middle, ScriptBlock = ” + $ScriptBlock  
  13.     child  
  14. }  
  15. middle   

You run that and get:

ScriptBlock = top
In middle, ScriptBlock = middle
In child,    ScriptBlock = middle

When CHILD runs, it tries to access $ScriptBlock and can’t find it so it goes to the parent MIDDLE, finds it and uses it.  Now let’s modify the MIDDLE routine and declare the ScriptBlock variable to be PRIVATE.  What we are trying to do is to hide it from everyone else so that when CHILD looks for it, it skips MIDDLE and finds it at the top scope.  We only change 1 line in MIDDLE:

  1. function middle  
  2. {  
  3.     $Private:ScriptBlock = “middle”  
  4.     “In middle, ScriptBlock = ” + $ScriptBlock  
  5.     child  
  6. }  

You run that and now get:

ScriptBlock = top
In middle, ScriptBlock = middle
In child,    ScriptBlock = top

Success!

So there are 3 things I want you to get out of this blog:

  1. PowerShell has rich semantics so explore SCOPES to understand them.
  2. There are usually multiple ways to solve a problem in PowerShell each with a different set of pros and cons.
  3. Explore and Share.  I shared my work and Niklas reminded me of a better way to solve the problem.  If I didn’t share, I wouldn’t have benefitted from his insight.  Sometimes beginners are embarrassed to share their work because they are afraid of looking like a beginner - DON’T LET THIS BE YOU.   The PowerShell community has a TON of super smart people that are both very helpful and very kind.  Share your work and if there is a better way of doing things, they’ll let you know that without being a jerk about it.  (I hope that is true and someone recommends a new PowerShell syntax highlighter for HTML :-)
  4. Things don’t have to be perfect – they have to be useful.  (joke intended but it is a serious point)  I could have gone back and done a bunch of work to clean up my examples to get rid of the emoticon but you can cut-n-paste the example and it is going to work fine and if I cleaned it up – I would never get a recommendation for a better syntax highlighter.

Explore, explore, explore!
Cheers!
Jeffrey

Evolution of a Script: Timing Operations

Posted on by
2

I just love PowerShell!  One of the things I love the most about it is that you can pick the level of programming you want, write a script and then evolve it to meet whatever need you have.  Sometimes you want something quick and dirty and other times you want something you’ll share with others and then other times you want something that is going to be used in production.

When I was earning my living on Unix systems, I used to write quick and dirty CSH scripts (yeah - I was one of those guys) but then when I wanted something more formal or to share with others, often I would throw away the script and restart from scratch.  I don’t do that with PowerShell.

This morning I was reviewing someone’s scripts that they were going to use to perform a large scale upgrade to a datacenter.  The code had lots of validation and verbose output and then they had lots of code that looked like this:

  1. #Start Stopwatch  
  2. $sw = [Diagnostics.Stopwatch]::StartNew()  
  3. #…..  
  4. $sw.Stop()  
  5.           
  6. Write-Verbose “Total ElapsedTime: $($sw.Elapsed.TotalSeconds) Seconds”  

I thought that I would help out by providing a helper function so I wrote my first version of the script like this:

  1. Function Measure-Time  
  2. {  
  3. param(  
  4. #ScriptBlock that you want to measure the timing of  
  5. [Parameter(Position=0, Mandatory=1)]  
  6. [ScriptBlock]$ScriptBlock  
  7. )  
  8.     $StopWatch = [Diagnostics.Stopwatch]::StartNew()  
  9.     &$ScriptBlock  
  10.     $StopWatch.Stop()  
  11.           
  12.       
  13.     Write-Verbose “Total ElapsedTime: $($StopWatch.Elapsed.TotalSeconds) Seconds”  
  14. }  
  15.   
  16. Set-Alias Time Measure-Time  
  17.   
  18. #Example usage:  
  19. Time {gps *ss} -Verbose  

So that worked great but then I thought about some problems:

  • Using the variables $Stopwatch and $ScriptBlock, it meant that those variables couldn’t be used in the scriptblock that was passed in.  Whenever you have a situation like this, the convention is to put 2 “_” in front of a name so I renamed them $__ScriptBlock and $__StopWatch.
  • I didn’t have any help.  I can never remember the keywords to add comment-based help so what I do is to start a new script, type CTRL-J to bring up snippets, select the CMDLET snippet and then cut/paste the help from it and use it to fill it in.
  • I wanted the user to be able to specify their own message so I provided a -__FormatString variable.
  • If you are upgrading a datacenter, you want to use the belt and suspenders approach so I thought this information should be logged to the eventlog (this started the next leg of the journey but more about that later).

The resultant script was starting to look like something worth sharing:

  1. <#  
  2. .Synopsis  
  3.    Measure how long something takes  
  4. .DESCRIPTION  
  5.    This script measures how long it takes to run a sequence of code (a ScriptBlock).  
  6.    The message is sent to the Verbose channel so to see it, you either need to have  
  7.    $VerbosePreference set to “Continue” or specify -Verbose.  
  8. .EXAMPLE  
  9.    Measure-Time { get-process } -Verbose  
  10. .EXAMPLE  
  11.    time {get-process} DataCenterUpgrade “It took {0:N3} seconds to run this” -verbose  
  12. #>  
  13. Function Measure-Time  
  14. {  
  15. param(  
  16. #ScriptBlock that you want to measure the timing of  
  17. [Parameter(Position=0, Mandatory=1)]  
  18. [ScriptBlock]$__ScriptBlock,  
  19.   
  20. # Name of the LogSource in the APPLICATION EventLog   
  21. # that these events will show up in.  
  22. #   
  23. # If you don’t provide a value - no log entry will be made.  
  24. [Parameter(Position=1)]  
  25. [String]$__LogSource = $null,  
  26.   
  27. # An alternate format string for the Message  
  28. [Parameter(Position=2)]  
  29. [String]$__FormatString=“Total ElapsedTime: {0:N1} Seconds”  
  30.   
  31. )  
  32.     $_StopWatch = [Diagnostics.Stopwatch]::StartNew()  
  33.     &$__ScriptBlock  
  34.     $_StopWatch.Stop()  
  35.           
  36.     $Message = $__FormatString -f $_StopWatch.Elapsed.TotalSeconds  
  37.     Write-Verbose $Message          
  38.     if ($__LogSource)  
  39.     {  
  40.             Write-EventLog -LogName Application `  
  41.             -Source $__LogSource `  
  42.             -EntryType Information `  
  43.             -EventId 1 -Message $Message   
  44.     }  
  45. }  
  46. Set-Alias -Name Time -Value Measure-Time  
  47.   
  48. #Example usage:  
  49. Time {gps *ss} DataCenterUpgrade -Verbose  

So everything looked great until I went to run it and then it barfed complaining that there was no DataSource called “DataCenterUpgrade”.  I’ve always believed that the different between a quick and dirty script and a production script was the error handling.  When you are in a datacenter at 3am when you expected to be home at 7pm the previous evening and everything it turning to crap – you want to have GREAT error messages.

I started off by adding some code to check for the DataSource and create it if it didn’t exist:

  1. if ($__LogSource)  
  2. {  
  3.   if (![System.Diagnostics.EventLog]::SourceExists($__LogSource))  
  4.  {  
  5.     [System.Diagnostics.EventLog]::CreateEventSource($__LogSource, “Application”)  
  6.   }  
  7. }  

The problem with that is that it only works if you are running with Admin privs (which I wasn’t of course).  So I put in a TRY/Catch and spent some time crafting an excellent error message:

  1. if ($__LogSource)  
  2. {  
  3.   try {  
  4.     if (![System.Diagnostics.EventLog]::SourceExists($__LogSource))  
  5.     {  
  6.       [System.Diagnostics.EventLog]::CreateEventSource($__LogSource, “Application”)  
  7.     }  
  8.   
  9.     Write-EventLog -LogName Application -Source $__LogSource `  
  10.       -EntryType Information -EventId 1 -Message $Message  
  11.   }  
  12.   catch [Exception] # An exception means you don’t have the privs to create the source  
  13.   {  
  14.     Write-Error -Message “Cannot Write to or Create EventLogSource ($__LogSource)” `  
  15.       -Exception $_.Exception `  
  16.       -Category ResourceUnavailable `  
  17.       -TargetObject $__LogSource `  
  18.       -RecommendedAction “Create LogSource ($__LogSource) using an account with Admin privs” `  
  19.       -ErrorId “CantWriteLog,Measure-Time” `  
  20.       -CategoryReason CantFindorCreateLogSource   
  21.   }  
  22. }  

This produced a great error message so I was feeling pretty good about my script.  Next I decided to look at the EventLog and realized that it was now logging perfectly accurate and perfectly useless information.  It was saying how long it took to do something but not how long it took to do WHAT?  Now I could have handled this using the FormatString:

  1. Time {gps *ss} DataCenterUpgrade “It took {0:N} to Gps”  
  2. Time {Step2  } DataCenterUpgrade “It took {0:N} to do STEP2″  

But that looks like a horrible user experience so I decided to include the scriptblock in the eventlog.  It then hit me that I had no idea how long the scriptblock would be but that it I was sure that the eventlog had a limit on the maximum size of the entry.  I could have researched this but I decided to experiment instead.  I created a DataSource and then did a number of experiments like this:

  1. $x = “T” * 1kb  
  2. Write-EventLog -LogName Application -Source MySource $x  
  3. $x = “T” * 10kb  
  4. Write-EventLog -LogName Application -Source MySource $x  
  5. $x = “T” * 100kb  
  6. Write-EventLog -LogName Application -Source MySource $x  

From this I determined that the max size was 32KB.  I added code to only log the first 31KB of a scripblock.  Now any reasonable person would point  out that a 32KB scriptblock is enormous and we’ll never see this condition.

Here is the point – that MAY be true.  But if it isn’t true and it breaks at 3am while some poor soul is trying to stich his datacenter together, then I’ve just made his life even worse.  And so we always code for that poor soul because one day, we will be that guy.  (Remind me to tell you the story about having to sleep on the datacenter floor in a hospital in Memphis between bouts of hand stitching a database together while patients were delayed getting admitted for care.) So here is how that segment came out:

  1. if ($__LogSource)  
  2. {  
  3.   try {  
  4.     if (![System.Diagnostics.EventLog]::SourceExists($__LogSource))  
  5.     {  
  6.       [System.Diagnostics.EventLog]::CreateEventSource($__LogSource, “Application”)  
  7.     }  
  8.   
  9.     # Maximum size of a Eventlog Message is 32KB. We limit to 31KB.  
  10.     $Script = $__ScriptBlock.ToString()  
  11.     if ($Script.Length -ge 31kb)  
  12.     {  
  13.       $Script = $Script.Substring(0, 31kb)  
  14.     }  
  15.     Write-EventLog -LogName Application -Source $__LogSource `  
  16.       -EntryType Information -EventId 1 -Message $(  
  17.       $Message + “ to run `n” + $Script  
  18.     )  
  19.   }  
  20.   catch [Exception] # An exception means you don’t have the privs to create the source  
  21.   {  
  22.     Write-Error -Message “Cannot Write to or Create EventLogSource ($__LogSource)” `  
  23.       -Exception $_.Exception `  
  24.       -Category ResourceUnavailable `  
  25.       -TargetObject $__LogSource `  
  26.       -RecommendedAction “Create LogSource ($__LogSource) using an account with Admin privs” `  
  27.       -ErrorId “CantWriteLog,Measure-Time” `  
  28.       -CategoryReason CantFindorCreateLogSource   
  29.   }  
  30. }  

 

So the point of this blog is that with PowerShell, you can start off with a quick and dirty and evolve it to meet whatever the requirements you have without having to start over from scratch.  It is one of my favorite attributes of PowerShell.  It should be – we designed it to do this.  :-)   So below is the script in the state I left it today.  There is more that can be done but it is fit for the purpose it is going to be used for.  We don’t have to make every script be model of perfection.

  1. <#  
  2. .Synopsis  
  3.    Measure how long something takes  
  4. .DESCRIPTION  
  5.    This script measures how long it takes to run a sequence of code (a ScriptBlock).  
  6.    The message is sent to the Verbose channel so to see it, you either need to have  
  7.    $VerbosePreference set to “Continue” or specify -Verbose.  
  8. .EXAMPLE  
  9.    Measure-Time { get-process } -Verbose  
  10. .EXAMPLE  
  11.    time {get-process} DataCenterUpgrade “It took {0:N3} seconds to run this” -verbose  
  12. #>  
  13. Function Measure-Time  
  14. {  
  15. param(  
  16. #ScriptBlock that you want to measure the timing of  
  17. [Parameter(Position=0, Mandatory=1)]  
  18. [ScriptBlock]$__ScriptBlock,  
  19.   
  20. # Name of the LogSource in the APPLICATION EventLog   
  21. # that these events will show up in.  
  22. #   
  23. # If you don’t provide a value - no log entry will be made.  
  24. [Parameter(Position=1)]  
  25. [String]$__LogSource = $null,  
  26.   
  27. # An alternate format string for the Message  
  28. [Parameter(Position=2)]  
  29. [String]$__FormatString=“Total ElapsedTime: {0:N1} Seconds”  
  30.   
  31. )  
  32.     $_StopWatch = [Diagnostics.Stopwatch]::StartNew()  
  33.     &$__ScriptBlock  
  34.     $_StopWatch.Stop()  
  35.           
  36.     $Message = $__FormatString -f $_StopWatch.Elapsed.TotalSeconds  
  37.     Write-Verbose $Message          
  38.     if ($__LogSource)  
  39.     {  
  40.         try {  
  41.             if (![System.Diagnostics.EventLog]::SourceExists($__LogSource))  
  42.             {  
  43.                 [System.Diagnostics.EventLog]::CreateEventSource($__LogSource, “Application”)  
  44.             }  
  45.   
  46.             # Maximum size of a Eventlog Message is 32KB.  We limit to 31KB.  
  47.             $Script = $__ScriptBlock.ToString()  
  48.             if ($Script.Length -ge 31kb)  
  49.             {  
  50.                 $Script = $Script.Substring(0, 31kb)  
  51.             }  
  52.             Write-EventLog -LogName Application -Source $__LogSource -EntryType Information -EventId 1 -Message $(  
  53.                 $Message + “ to run `n” + $Script  
  54.             )  
  55.         }  
  56.         catch [Exception] # An exception means you don’t have the privs to create the source  
  57.         {  
  58.             Write-Error -Message “Cannot Write to or Create EventLogSource ($__LogSource)” `  
  59.                 -Exception $_.Exception `  
  60.                 -Category ResourceUnavailable `  
  61.                 -TargetObject $__LogSource `  
  62.                 -RecommendedAction “Create LogSource ($__LogSource) using an account with Admin privs” `  
  63.                 -ErrorId “CantWriteLog,Measure-Time” `  
  64.                 -CategoryReason CantFindorCreateLogSource   
  65.         }  
  66.     }  
  67. }  
  68. Set-Alias -Name Time -Value Measure-Time  

I hope sharing my thought process was useful for your scripting.

Cheers!
Jeffrey

On Heroes

Posted on by
7

Technology heroes are always difficult subject.  As an engineering manager, I remember the first time I participated in a “life boat” drill where you have to produce a stack rank of your engineers.  Someone explained the process saying, “The task is to figure out if you had to throw out one person from a lifeboat, who it would be?  After you figure that out, then you decide who would the next one be, etc. until you have a fully ordered list.”  I chewed on that a bit and asked, “How many people are going to be left in the lifeboat?”  When they asked me why, I replied “Because if there is only one person left in the lifeboat, it would be Mark but if there was more than one person left, Mark would be the first one I’d throw out.”  Mark was a technical hero.  Able to accomplish a great deal but at a great cost to the the organization.

I’m reminded of a story in Storming Norm Schwarzkopf’s autobiography where he was frantically trying to stand up forces in Saudi Arabia during the first Gulf War.  At some point during the buildup, someone asked him where he wanted to deploy Special Forces and Norm replied something along the lines of, “keep them the F*** out of my theater!”  (Norm is a colorful character.)   Shocked, the person responded along the lines of, “these are the best of the best, the most capable warriors in the history of the world, what do you mean you don’t want them in theater?”  Norm explained, “Look – this is all about a couple hundred thousand guys killing a couple hundred thousand other guys.  If your Seal teams can kill a couple hundred thousand guys, great, I’ll send everyone else home.  If they can’t do that, then keep them the F*** out of my theater.”

What clarity of that thought!  What it basically comes down to is that it doesn’t matter how good you are if you are wrong for the mission.  Windows (any OS really) is about a couple thousand guys writing a millions of lines of code to deliver lots of scenarios.  If a few heroes could accomplish this, we would pay them a lot of money and send everyone else home.  They can’t but let’s just say that they could.  If you build a plan around a hero and then that hero gets hit by a beer truck, you’re screwed.  Hero-centric planning is fragile and ultimately irresponsible.  But superstar technical individual contributors do exist and they do occasionally change the way the game is played.  So while it is folly to plan around heroes, it is also folly to ignore them.  Norm understood that.

After Norm got a critical mass of forces in the Gulf and had a plan he was comfortable with, he revisited the idea of using the Special Forces and used them to great effect in the war.  Navy Seal teams were some of the first to engage and served with great distinction.  But let’s be clear about it.  They were optional elements to a plan that would succeed without them.  They enhanced the plan, in some cases dramatically but Norm needed a plan to succeed even if they didn’t exist.

In the same token, there are tasks that require a hero or small group of heroes and would be screwed up with a couple hundred thousand guys tried to do them (think hostage rescue mission).  At Digital we used to say that a Consulting Engineer was someone that could solve a class of problems that other engineers could not solve no matter how many of them there were.  There were times that you just had to bring in a Consulting Engineer and cope with whatever that entailed or you would not get the problem solved.  The best Consulting Engineers were those that would roll up their sleeves and help while keeping their egos in check.

One of the problems with a hero-centric world is trying to get the hero’s to add up.  Bill Gates used to say this all the time, he talked about how we had dialed-in the ability to hire the smartest people in the world but still hadn’t figured out how to get their IQs to add up vs. negate one another.  Look, we’ve all worked with heroes before so let’s just say what we all know; they can be egotistical buttheads or prima donnas.  They don’t have to be.  Certainly Jim Gray wasn’t a butthead.  Not only was he one of the smartest guys in the world.  He was also one of the nicest, most helpful and approachable guys as well.  More often than not, the characteristics that allows someone to think dramatically differently than other people and the courage and tenacity to put that idea out there and fight to make it come to fruition are the same characteristics that make them difficult to work with.  (NOTE:  My experience is that a senior technical IC must be a butthead at times to make things happen.  The distinction is whether they are motivated by concern about the customer, the company, the technology or are they motivated by their ego and status.)

Said another way, many heroes find it hard to be followers.  I’m reminded of a conversation I had with an awesome mentor at Digital, Mark Storm.  Mark was a Consulting Engineer and had worked with 4 or 5 other rock star Consulting Engineers in the database team.  I said something along the lines that it must have been awesome to be a team with that much talent.  He shook his head saying that it would have been much better if there was only one or two of them there because they spent so much time competing with one another about what to do and how to do it.  Nietzsche one said that if you had to have virtue, have only one because as soon as you have two, they start to conflict.  Sometimes it is that way with heroes as well – but not always.  There are plenty of heroes that will lead but are also able to follow.  And not just follow but commit and execute with the same passion they would have if they were executing their own ideas.

Technical heroes can be both a blessing and a curse.  An organization, needs to be super clear about what type of problem it has and whether it is one suited for a hero or an army and the have the courage to make the appropriate decisions.

Focus on Reality

Posted on by
8

Years ago I enjoyed reading the book, Execution: The Discipline of Getting Things Done.  I got a lot of out of it but it had a small signal-to-noise ratio.  I’ll save you a few bucks and a lot of time by sharing the 2 key lessons I got from the book:

  1. Execution matters.  Vision and strategy are great but what changes the world, improves the lives of our customers and pays the bills is execution.  I remember the first time I heard Bill Gates say, “vision is cheap”.  I thought to myself, “what a jerk”.  That was before I had read this book and I didn’t “get it”.  Well Bill was right and I was wrong (it has gone the other way a few times as well but that is for a different blog post )  The book did a great job making this point.  I’m not sure whether it was this book that said it or not but the phrase that gets this in focus is, “a vision without execution is called a ‘dream’”. 
  2. It is critically important to engineer encounters with reality.  “Engineer encounters with reality.”  Say that a couple dozen times.  The point is that people have lots of visions/strategies/ideas/approaches.  They could be good or they could be bad.  At the end of the day, the best judge of that is reality.  Our job is to create a system whereby we never stray from reality for long and constantly course-correct to be better aligned with reality.

I got a great lesson in this from John Sweitzer a Distinguished Engineer at Tivoli/IBM.  I had the privilege of working closely with John for a number of years through a number of very difficult situations and boy did I learn a lot from that guy.  There was one situation where I was trying to get a group to adopt a particular design and was having difficulties in convincing them.  Having no positional authority over the group, I could not control the decision so I needed help in figuring out how to convince them to do the right thing.  John had been in this situation many times before and his advice was, “sometimes the best thing you can do is to help a team fail“.  You could have knocked me over with a feather after I heard this.  There are people that love to win and there are people that hate to fail (they are different).  I hate to fail.  I’ll do almost anything to avoid failing and viewed it as my job, hell – my reason for existing, was to help this team avoid failure and here was John telling me to help them fail. 

John went on to explain, “You are either right or you are wrong.  If you are right and they are going to fail with their current approach, then you should be able to articulate where the design is going to fail.  Use that to help them fail as quickly as possible so that they can learn the lesson and restart on the right path.  You are saying that the design won’t scale so make sure everyone agrees what the scale requirements are and then arrange to have them prove that their design can meet those goals ASAP (make it one of the first coding goals).  One of two things are going to be the case: 1) it scales in which case  you’ve learned something and it is a good thing that you didn’t get them to change or 2) it doesn’t scale in which case they are going to have to change their approach.  At that point they might take your approach or they might pick another but one way or the other, it is going to meet the scalability requirement that we’ve all agreed to.”  That was a career-changing conversation for me.  John was giving me my first lesson in “our job is to engineer encounters with reality”.

Early in my career I adopted an aphorism which has served me well and is key to embrace this idea of focusing on reality: ”My job is to ship the best ideas not come up with them“.  I come up with lots of ideas and a lot of them have turned out well but this aphorism has helped me shift focus from my internal thoughts to what is going to change the real world.  This mindset allows you to drop your idea in a heartbeat if someone comes up with a better one without your ego getting in the way and as such, it allowed me to embrace John’s advice and make it actionable.  If it think a team’s approach is broken and I have a different idea of how it should be done, I try to be clear about what problem I want addressed and then be totally open to how that problem gets addressed.  (It doesn’t always play out that way at work but when I’m doing my job well, that is what I strive to do.)

As a general rule, it is much better for a team to come up with their own approach for addressing the problem instead of you telling them how to solve it.  Sometimes the situation doesn’t give you that flexibility but when it does it is the better approach.  Given a clean slate and a problem to solve, the team might come up with a better solution than you have.  If you give them a solution then they might just adopt it and not think out all the possibilities or take responsibility for its success.

At the end of the day what matters is reality.  What we need to do is to transform conflicts in the ethereal world of ideas, opinions, egos, visions and strategies into encounters where reality will provide the data about the right thing to do.  In Tog on Software Design, Bruce Tognazzini said something along the lines of, “Engineers will argue for weeks over things that can be solved with a 30 minute user study.”  Many cloud-services have adopted a data-centric decision making process where they create various experiments and then let the user response decide which approach to take.

Sounds easy right?  It isn’t.  Knowing how to do this, when to do it and what to do when you can’t is super difficult but that is what we have to strive for.  I once had a meeting where a team had done a customer survey and wanted to use the results “as-is” as the plan.  I knew the area well and had a pretty good idea of where the survey was telling us real information and where it reflected the fact that we asked the question in the wrong way.  My comment to the team was that, “If you aren’t using your perspective and knowledge to interpret the data, then I could replace you with a computer survey and save a lot of money.”  The aphorism that helps here is, “Any idiot can listen to the customer. It takes a smart person to know WHEN to ignore them.”  Apple is famous for ignoring what customers say but clearly have done a good job focusing on reality and delivering products that people respond to. 

There are no easy answers here. The mechanisms to focus on reality will vary and require judgment.  John Sweitzer taught me a really great technique to help teams you are trying to influence by helping them fail fast.  There are many other techniques.  As the book said, execution matters so developing a strong preference for reality over opinions is the correct path forward. It not only produces better products but a better work environment.

Iranian Drone Hack and Technical Debt

Posted on by
15

This week I read the story about how Iran hacked and captured one of our most sophisticated military drones. It struck me that this was an excellent example of the potentially disastrous ramifications of ignoring technical debt.  It appears that the potential to spoof GPS was known and ignored for many years.  Acknowledging and managing technical debt is an issue that separates the whiz kids from the graybeards in the tech industry.  Let me start by saying that a military boof-a-rama of this magnitude is, by necessity, going to be followed up with multiple misinformation campaigns so I doubt we’ll really know what really happened for a few decades.  Nevertheless it provides a teachable moment for us so let’s explore this topic.

Technical debt is analogous to financial debt so let’s start there.  Imagine you are a fresh college graduate with a new job and a newly minted credit card.  Thanks to the miracle of modern financial Svengalism, you can buy things you don’t have the money to pay for.  You use the card to get set up in a new apartment and at the end of the month you have a choice, you can pay off the debt or you can pay the minimal possible.  Debt is not a problem if no one makes you pay attention to it.  And there’s the rub, at some point you reach  your limit and the party comes to an abrupt halt.  Not only do you have to start paying off your debt but you have accumulated a bunch of interest that needs to be paid off as well.  At this point, you are screwed.  You can’t buy new stuff and you have to spend all your effort trying to pay off a mountain of debt.  You pay and pay and pay but for all your effort and money going out, you have no new stuff to show for it – you are managing your financial debt.

The same thing happens with technology.  This is most clearly seen in startup companies but the same thing happens in most technology companies.  At a tech startup, the most important thing to do is to get the right set of functions in the market as quickly as possible so you can start bringing money in.  What this means is that you are often cutting corners – skipping documentation, ignoring supportability, deferring security work, doing quick and dirty designs vs. deep architectural work.  This might be a reasonable strategy for a startup because once you get the product out, you might get feedback that you need to go in an entirely different direction.  If that is the case, you can “declare bankruptcy” on your existing code, walk away from those debts and start on a new path.  On the other hand, if you stick with that code, you have a technical debt which is going to have to be paid.  In your  next development cycle, you can either begin to pay down that technical debt or you can devote 100% of your resources on the next round of quick and dirty functions.  Deciding when to start paying off your technical debt is what separates the whiz kids from the graybeards in the tech industry.

I’m sure most of us at some point have found ourselves on a project that was led by whiz kids that had completely ignored their technical debt and was now in shambles.  I was hired at DEC to lead the turnaround of just such a project.  The product was reasonably successful but the implementation “had some issues”.  When I got onboard and did an assessment of where we were I was shocked – the product was a cobbled together hack using 18 (yes 18!) different languages.  There was even some ADA code in there.  I joked with one of the developers on the project saying, “How did that get in here?  Did someone take a night course can decide to ship his homework in the product?”  Imagine my horror when then answer came back, “Yes – that is exactly what happened”.  When we explained where the code base stood and what was required to move forward, the management team cancelled the project and decided to just release it with a few new bug fixes and features.  The interesting aspect of this story is that the whiz kids responsible for this mess were viewed very positively – they got things done and delivered on time and they got promoted and moved on to go screw up bigger and better projects.  This is an example of where ignoring technical debt lead to the death of a project.  Often the results are less dramatic, you’ll get a new version of the product with fewer new features.  Sometimes the results of ignoring technical debt are much more dramatic.  Like our drone.

Drone technology is the darling of the US military/industry/political parties.  They provide a tireless presence on the field of battle, they can project force without risking the political ramifications of putting US personnel in mortal danger, and they generate new military contracts for the latest and greatest technology.  The original drones were relatively simple and effective monitoring devices.  They were great but had a lot of well understood problems.  The data they streamed back was unencrypted so anyone could grab the signals and see what we saw as well as get detailed operational information about what we looked for and how we looked for it as well as how we operated the drones.  The designers were able to anticipate some problems and attempted to address them. This provides us a great example of where you have to be just as vigilant about a mitigation as you are about the problem it tries to mitigate.  The designers understood that there were various hostile and non-hostile situations where communications between the drone and the controllers would be severed.  They addressed this by programming the drone to detect the situation and return home using GPS and a preprogrammed set of coordinates for the home base.  And here is where the technical debt problem comes in.

Apparently the military has known about the potential for GPS to be spoofed for 20-30 years.  This is a technical debt.  The Christian Science Monitor reports that what Iran did was to jam communications with the drone so it engaged it’s “fly home” program and then spoofed the GPS signal to make it land in Iran.  The Iranians now have some of our most sophisticated military technology because we failed to address our technical debt.

When we decided to go-big on drones and transform them from a marginalized science project to the sharp point of the US military spear we faced a choice: we could either address our technical debt or we could add bells and missiles.  It appears that we decided not to address our technical debt (or not to address it in time to avoid this mess).  After all, where is the excitement in making things work?  I’m sure some argued that, “They do work, they just don’t work as securely we’d like and we don’t know that others can actually spoof it”.  Given a fixed budget, any money spent on addressing technical debt is money not spend adding new bells and missiles.  To ship is to choose.  And that is exactly the point – it is what separates the whiz kids from the graybeards.

Whiz kids will ignore technical debts, putting 100% of the budget on new features.  They build houses of cards on foundations of sand and then move on before the waves come.  They come off looking like superstars and then move on to screw up something bigger.  As graybeards (and architects) we need to be the guardians of the future.  We need to understand our debts and be vigilant in addressing them even though there is no clear and immediate benefit for doing so.  This is easier said than done.  It is easy when we are in charge and just set the rules.  It gets harder when we don’t own those decisions and need to advocate for them and fight for budget.  Few people are interested in hearing it so it takes courage to stand up and be a butthead to ensure the right things happen.  Some deal with it “off the books” by padding schedules with a fixed amount of overhead to address the issues.  One way of the other, it needs to happen or you end up with a disaster.  You can only avoid problems for so long and then you have to address them.

Days Till Xmas

Posted on by
2

This morning my daughter asked my wife how many days there were until Christmas. That conversation didn’t go so well.  :-)    Seriously though, this year things started super early.  The local store started selling Christmas stuff before Halloween and a couple of weeks ago our neighbors put up their Christmas lights.  I can understand why my daughter is getting whipped up into an Christmas frenzy.  That said, as my wife contemplates a big day of effort for Thanksgiving, I can understand why questions about Christmas are not welcome.

PowerShell to the rescue (seriously).

This morning’s drama played out while I was at the computer reviewing Doug Finke’s table of contents for a PowerShell for Developers book he is starting to write.  (looks like it is going to be great!).  I called my daughter over to explain what a wonderful tool PowerShell was.  I explained that if she ever wanted to get the date, all she had to do was to type “Get-Date”.  I then explained how she could cast a string containing a date into a datetime object:  [Datetime]“12/25/2011″ .  She didn’t get that one so I quickly moved on to explain that the reason why that mattered is that PowerShell is … well … powerful and that it does something called “object math”.  Object math (the ability to perform mathematical operations on any object which supports them) made everyone’s life easier because that means you can do things like:
PS> [DateTime]“12/25/2011″ - (Get-Date)
Days              : 34
Hours             : 13
Minutes           : 21
Seconds           : 12
Milliseconds      : 588
Ticks             : 29856725886382
TotalDays         : 34.556395701831
TotalHours        : 829.353496843944
TotalMinutes      : 49761.2098106367
TotalSeconds      : 2985672.5886382
TotalMilliseconds : 2985672588.6382


Her face lit up like a Christmas tree!  I then remembered this was a repeat of something that happened years ago so I used Bing to search for “Snover daughter PowerShell Christmas” and found my Days Till Xmas post back in 2006 when we still called it Monad.  We didn’t have the Get-Date Cmdlet back then so I used [DateTime]::Now but notice that this script uses the YEAR property and string math to always calculate the days till this years Christmas.
function tillXmas ()
{
    $now = [DateTime]::Now
[Datetime](“12/25/” + $now.Year) - $Now
}

Today I’ll convert this to a proper function with a proper name and install it on my daughter’s machine so that she will always know how many seconds until Christmas and won’t have to ask my wife. 

PowerShell solves all sorts of problems!  :-)

On Rumors

Posted on by
2

I often get people asking about this or that rumor.  Invariably my response is, “I’m sure it is true but I wouldn’t worry about it”.  This blog explains that perspective and has nothing to do with any specific rumor.  I hope it will help reduce your anxiety the next time you find yourself confronted with a wave of rumors.

Rumors in the workplace are as common in corporate America as conference rooms and coffee machines.  Sometimes there are more and sometimes there are fewer but, like the poor, they are with us always.  When I started in the business, I got pretty wrapped up in the rumor trade, cultivating good sources, tracking the latest gossip and even trying to read the tea leaves (“Did you notice who sat next to whom at that meeting?”).  I got excited when rumors were positive about things that affected me and upset when their future was uncertain.  Most of us don’t like change as it brings uncertainty so rumors can be a source of distraction and frustration.  I’d worry about my career, my paycheck, my ability to pay the mortgage and stay in our house.  As I advanced in my career and got to be more involved with senior leaders in their decision making process, I gained a different perspective on rumors.  I thought I would share that perspective with you so that it might help reduce the associated anxiety and allow you can avoid wasting time, energy or emotion when the next rumor comes your way.

Fact: Most rumors are true.

Yup, that’s right, most rumors are true.  Even when the rumors contradict each other they are true.  The reason for this is that most rumors are of the form, “X is considering doing Y“.  The key is the word is considering.  Well guess what?  That is the job of leaders and executives – they consider.  We constantly consider as many possibilities and scenarios as possible and then game out our options, potential responses and probable outcomes for the business and the people involved.  Sometimes those possibilities and scenarios are positive ones but for those of us in the Andy Grove, “Only the Paranoid Survive” camp, we usually game out the worst-case scenarios.

  • “Maybe we should go big in this area and increase engineering by 30 heads, where would we cut to free 30 heads and what that do to our business?”
  • “Maybe we should move x to/from another group for greater synergies, what are the issues?”
  • “Next year we going to have a larger/smaller budget for heads/contractors/hardware/…, come up with a list of possible things we could do to respond and then let’s consider a set of factors that we’ll judge those scenarios against and prioritize accordingly.”
  • “What would we do if our suppliers pursued a forward-integration strategy?”
  • “Maybe we should get out of that business and sell off our engineers for medical experiments – check the spot price of kidneys.”

Great leaders constantly consider a wide set of options – that is core to being a leader.

The next thing to understand is that great leaders use fence-posting.  Fence-posting (or limit-testing) is when you take parameters to the extremes to force a hard conversation and uncover the underlying physics of a situation or principles that you will use when solving a problem.  For example, every organization deals with the issue, “we need to reduce X by Y% next year“.  When Y is a realistic number (which is generally relatively small), it is easy to avoid the hard conversation by peanut-buttering the cut.  This is where you spread out the cut, reducing everything by a little.  This makes it easy to deal with pushback from your people.  You can easily defend your decision by saying that everyone has to tighten their belts a little and no one was spared from the sacrifice.  Sometimes this is the right thing to do but often it is better to cut something entirely which potentially frees up additional resources which can then be used to strengthen the remaining projects.  The way you get to that conversation is by fence-posting.  You ask the question, ”What would we do if the request was to reduce X by 2Y or 3Y or 5Y next year?”   When an executive throws that question to their staff, you get a pretty engaged and creative conversation.  Fence-posting forces people to get clear and crisp about their priorities.  Once you have those in focus, you can bring them to bear responding to the actual situation.

Fence-posting is a great planning technique but just consider its effect on the rumor mill.  During fenceposting you consider a wide range of dramatic options but in the end, you typically end up doing something pretty undramatic.  That said, when portions of the conversation get retold to people who were not involved, rumors spread like wildfire.  Imagine the anxiety you’d have if you heard your management chain was considering selling you off for medical experiments!  You’d be crazy not to be concerned.  And yes – the rumors will be true – they did consider doing all those things.  They just didn’t do them.

I heard that during the Cuban missile crisis, President Kennedy had a group of people study whether to engage in an all-out first strike nuclear war against the USSR.  They considered it and came back and recommended against it and obviously that recommendation was accepted.  Some people hear this and are totally freaked out – “WHAT? We considered initiating a nuclear war?!?“.   Having been a leader for a while and attended lots of brainstorming sessions, I heard this and my reaction was, “Sure – that only makes sense.  I’m glad that they considered the options and I’m they didn’t do it.”

So in the end, it is hard not to get engaged when you hear a rumor that affects your project but if and when it happens, take it with a grain of salt.  It is probably true that people considered that as well as a range of other options but they probably won’t happen either.  Wild rumors could really just be a sign of a healthy, vibrant decision making process.  Of course they could also be a sign of a random feckless management team.  One way or the other, you are generally best off reminding yourself that leaders consider a wide range of choices and trying to ignore the rumor and focus on your job and then adjusting if/when a change is made.  It is hard to do but it is worthy trying.

Changing Datacenters, Changing IT

Posted on by
5

I just finished giving a keynote talk at the Windows Connection conference here in Las Vegas (I’m writing this while waiting to board my plane back home).  My topic is one that I’ve been doing a lot of thinking about – how datacenters going to change and what that means to IT Pros.  I’m passionate about the success of IT Pros and want to give them the tools they need to succeed.  In this case, the tool that they need to succeed is perspective.  The Readers Digest of my talk is:

  1. Datacenters are changing dramatically (largely due to virtualization and the cloud).
  2. Some will prosper.
  3. Others will not.
  4. Plan accordingly.

A good chunk of this talk deals with the topic of Clouds:

  • What are they?
  • Are they real?
  • Will they just go away if  we ignore them?
  • Do they mean the death of IT and local datacenters?

I didn’t know the answers to these questions a year ago when I started asking them but I’ve chewed on this a lot and come to some conclusions that I thought I would share.  I’ve read a lot about clouds from the pundits but honestly I haven’t gotten much out of most of that.  The one big exception to that is The Economics of the Cloud by Rolf Harms and Michael Yamartino which I consider a must-skim/must-read.  In general what you hear is a lot of absolutist ax-grinding from people with an agenda.  Let me clear, I have an agenda as well.  The difference is that my agenda is the success of IT Pros – that is what I care about.   Clearly I’m busting my butt trying to deliver the set of tools I think you need in order to succeed but I’ve always been super clear on this point: use my technology/don’t use my technology, all I care about is that you are doing what you need to do in order to succeed.

The short summary of my conclusions is that I’m very optimistic about IT pros as long as they understand the challenge and engage in the steps necessary to prosper.  But let’s be clear – change is required.  I’m less optimistic about people that are not interested or willing to invest in their careers and are unwilling to learn new skills.

The PPT version of my slides are available HERE.
A PDF version of my slides is available HERE.

I’d love to hear what you think of my analysis and conclusions.

Cheers!

On Becoming a Senior Technical Individual Contributor

Posted on by
3

I’ve done quite a bit of mentoring.  I enjoy it because actually “saying it” forces you to transform your vague ideas into crisp ones.  One of the most frequent issues people want to work through is how to become a senior technical individual contributor (IC).  We all have lots of examples of technical people that go into management and advance their careers.  That model of career advancement is as universal as it is straightforward:  Manage a bunch of people, don’t screw up, ship on time and then do it again at a bigger scope until you stumble and get fired.

The path to becoming a senior technical IC is less obvious to people so I thought I would share some thoughts on the topic.

Do your Job
Sivaprasad Padisetty (Siva) is the Director of development for the Windows 8 Management and Solutions Platform team and is one of the finest technical managers you can find.  A while ago he gave a very thoughtful detailed talk about the dynamics of being a technical IC and one of his key messages was, “do your job”.

The point Siva was making was that every job level has its lessons to learn and that you have to take the time to learn them to completion before going onto the next level.  For example, a key element of very senior technical ICs is the ability to formulate and drive strategy.  If you are a junior developer, you should be focused on great coding practices, great design and great development processes and time spent thinking about strategy may not be your best investment.  Chances are that you are not going to be invited to the meetings where strategy is discussed and determined and let’s be frank – who is going to listen to a junior engineer about strategy if they can’t write good code?  That is like a kid being convinced that he belongs in the World Series so he is not going to waste his time playing little league.  Play at the level you are, do your job and master the lessons it has for you.

The next aspect of “do your job” is really a statement about focus and priorities.  It is important to stop and think about your career – every once in a while.  Mid-year and end-of-year reviews are explicitly about this.  Other than that, you should “do your job”.  In other words, the way you advance your career is not by spending a bunch of your time thinking about how to advance your career.  It is by delivering results and having a real impact on the customers and the business.

Here is the model that you need to keep in the forefront of your mind:  you can’t prosper unless the company prospers,  the company can’t prosper unless the customers prosper.  So if you want to prosper, then focus on the customer and the company first and then the right things will happen over time.  Now sometimes things go wrong and the math doesn’t work out correctly – you can prosper while the customers/company don’t or vice versa.  My experience is that these things all work out in the end.  You are the level that you are and you will find yourself in that level in time.  If you are below where you should be, that will self-correct by the company recognizing and fixing its error or another employer will recognize and fix that error for them.

Become a Development  Lead or Manager
I tell anyone who wants to become a senior technical IC that they need to become a Developers Lead or Manager for a period of time first.  This confuses people because that is explicitly what they don’t want to do, they want to be an IC not a manager.  Here is why you need to do it.  Any senior IC has to partner with a manager or a management team in order to be successful.  If you don’t understand that job then you can’t be an effective partner.  You need to understand the job well enough that you know when you can successfully push for something and when you should just keep your mouth shut because it isn’t going to happen (e.g.  “Our org has better developers than the other orgs so we shouldn’t have to fit the curve.”)   You need to understand the full magnitude of the crap that manager has to put up with and go through in order to create an environment where his/her people can get their job done.  You need to understand it, you need to appreciate it and you need to partner with them go create a successful environment as well as a successful product.   Your team can’t succeed unless your boss succeeds so you need get that in focus and enter into a true partnership with that manager.  Knowing what that job really entails is the best way to be a great partner.

The Three Things Every Very Senior Technical IC Must Have
Most technology companies have a set of levels that your manager can promote you to.  Above that, there are a set of levels that your manager can submit you for but can only be given through some sort of community process.  At Digital, your boss could promote you through Principal Engineer but then we had a very lengthy process to become a Consulting Engineer (at least it was lengthy for me :-) ).  At Microsoft, we have something similar but we call them Partners and it is open to a wider audience (PMs and Test as well as Development).  Because your boss can’t give you this promotion and because your boss might not even be privy to the process, they might not be able to give you a clear set of steps you need to perform in order to get promoted or be able to articulate the promotion criteria.

The first thing you should realize is that this boundary is a very hard barrier to break through.  The vast majority of developers are going to top out their technical careers below this barrier.  There is nothing wrong with that.  Related to that is the fact that there is a very different set of expectations on the other side of that promotion so you if you are not ready for that game and you get promoted to that level, life is going to be super uncomfortable.   While various companies have various processes, my experience is that they all basically look for the same three things that Jeff Schriesheim explained to me years ago at Digital:

  1. Technical “twinkle”:  A history of technical accomplishments that other senior engineers look at and say, “wow”.  Not just a bunch of code or a slew of features but rather something that shows originality and accomplishes  something that could not be accomplished even if you had 30 or 50 principal engineers working on the problem.  Twinkle is the hallmark of a special engineer.
  2. Business impact:  At the end of the day, the company is about having an impact and the easiest way to measure that is money.  You can have all the twinkle you want but if you aren’t putting cash in my pocket – then your presence is indistinguishable from your absence.
  3. Community acceptance:  At big companies, major things happen by having senior people work together to solve hard problems.  If you will not be accepted by the existing community of senior technical ICs then you can’t contribute to solving this class of problems and therefore you are not a senior person.

Once you are a Senior Technical IC, there is another set of rules and dynamics that come into play but that is a topic for a future blog entry.  I’ll leave you with this. Getting promoted to Consulting Engineer at Digital was one of the proudest moments of my life – I was accepted into one of the finest technical communities the industry has seen (the list of people in that community is staggering).  One of my mentors was Mark Storm, a Consulting Engineer.  When Mark congratulated me on my promotion, I joked asking him if I know got to learn the “secret handshake”.  Mark said, “We don’t have a secret handshake but we do have a motto:  Often wrong, never uncertain.”  :-)

Jeffrey Snover| Distinguished Engineer and Lead Architect for Windows Server

Monad Manifesto

Posted on by
9

I wrote the  Monad Manifesto in 2002.  I had been working on Monad for over a year at that point and had been relying upon whiteboard conversations and demos in my office to bring people up to speed on what we were doing.  That model didn’t scale well and different people need different forms of information in order to get it.  In particular, we were originally trying to do PowerShell as a distribute development model where we had a few PMs, a Dev Lead (Bruce Payette) and myself working in Redmond and then the development and test teams working at the India Development Center in Hyderabad India.  We had all seen a number of projects fail in this configuration and were dedicated to making it work.  One of the things we heard over and over again was the importance of documenting things.  I had had a number of conversations with our team in India where everyone shook their heads and smiled and seemed to get it but then the code clearly demonstrated that they had absolutely no clue what I was saying but weren’t telling me that.   We tried a number of times to get very precise in our documentation but time and time again, it was clear that they just weren’t getting the concept so I decided to “go meta” and explain it in a Manifesto.

Writing the Manifesto was difficult.  I knew what we needed to do and could come up with examples and explain this point and that point but I had not taken the time to put in  words.  I had no grand overarching narrative.  Sometimes you just need to “say it”.  When you are forced to actually “say it”, you’ll often find that you really just sorta understand it – that you have a vague notion of what you mean but you don’t really know what you mean until you are forced to say it.  BTW – this works in the other direction as well.  Sometimes I’ll be in a meeting where someone is saying something that sounds good and everyone is grooving on it but then I realize that I don’t really know what the person is actually saying and I’m not sure that they do either.  I find that everyone benefits from stopping the conversation and asking them to precisely articulate the thought.  “Why would that be a good thing?”  “Who would use that?” “How would that actually work?”   It is hard to ask these questions because they sound stupid – they make you look like you’ve lost the thread … until there is no answer or the answer is fuzzy.  The world is completely different (and better) when you are forced to actually “say it”.   Clarity provides true context that people can actually agree or disagree and more importantly, can provide clear feedback to make the thinking better.  Wooly thinking makes people feel good but doesn’t really move the ball forward.  Clarity is progress’s best friend.

The Monad Manifesto forced me to be clear about what problem I was addressing, what my principles were, how I intended to address the problem and then who would benefit and why.  One of the most helpful parts of this process was Section 10 Value Propositions.  I was lucky enough to work for superstar Israel Gat at Digital and then was able to recruit him both to Tivoli and to Microsoft.  Boy did I learn a lot from that guy.  One of the most important things he taught me was the Geoffrey Moore Value Propositions format which I use in section 10.  What you do is to generate a set of statements of the form:  For <Customer> who <Qualifier>, <Product> <Value Statement>.  Unlike <Alternative>, <Product> <Differentiator>.   Israel explained that it was deceivingly simple and yet exceedingly difficult but that you MUST do it because when you succeed then you have absolute clarity about what you are building, why you are building it and you have a context for prioritizing your work and knowing when a feature cut is acceptable and when it is intolerable.   That was some of the best advice I’ve ever received.  After I figured out the Value Propositions, I had absolute clarity of what was important, why it was important and was able to bring that clarity to bear on the thousands and thousands of decisions that we made in the project.  It also set the overall tone for the project – we are going to succeed based upon delivering compelling value to customers.  It is all about the customer, it is not about us, not about technology, not about Microsoft, it is all about customers with real problems and our ability to deliver real value to those customers because we have a clear understanding of those problems and a unique and powerful approach to addressing them.  We often get feedback that the PowerShell team is one of the most customer-focused teams and I’d like to think that some of that comes from the clarity of our mission.

It has been over 9 years since I finished V1.2 of the Manifesto and I am amazed when I compare it against the history of Windows PowerShell.  Release after release, the vision articulated in the Manifesto has been implemented.  For instance, Section 8 describes remote execution:

A script can be executed in BestEffort or Reliable mode. BestEffort scripts are run from the existing process and if that process terminates, no effort to clean up the remote scripts is done and any outstanding results are lost. Reliable mode scripts are persisted to a local SQL store and a service handles the execution of the script. The user can log of out the machine and the service continues to process the script. The user can log back in and get the results of that job sometime in the future.

BestEffort scripting was delivered in PowerShell V2 and the description of Reliable mode scripts pretty much describes how the PowerShell V3 workflows work.    I published the document in 2002 and then didn’t read it again until many years later when we were getting V2 out the door when I read it again and was stunned by how accurate it was.

It is only fair to point out where the Manifesto has not been implemented or implemented well.  With PowerShell V3, the new admin GUIs ARE layered on top of PowerShell but most of them do not emit the command line equivalents (the AD GUI does).   I also stated that 50-70% of a generic management GUI tool could be provided for free by building the right type of Cmdlets.  We have not delivered that but if you take a look at what we are doing with the V3 Show-Command and consider what you would have if you combined that with something like PowerGUI, you can see that we are getting close.

To borrow a phrase from Buddha: drop by drop, a bucket is filled.