Keith's profileKeith Hill's BlogPhotosBlogListsMore Tools Help
    June 23

    New PSCX Eye Candy for Vista

    If you are using PSCX 1.1.1 and are running on Vista and your not too attached to your own prompt function, I encourage you to try out EyeCandy.Vista.ps1.  This works best if you are running the PSCX profile.  If so, just modify the following lines like so:

    # ---------------------------------------------------------------------------
    # You can modify every aspect of the PSCX prompt appearance by
    # creating your own eye-candy script.
    # ---------------------------------------------------------------------------
    ## $PscxEyeCandyScriptPreference = '.\EyeCandy.Jachym.ps1'
    ## $PscxEyeCandyScriptPreference = '.\EyeCandy.Keith.ps1'
    $PscxEyeCandyScriptPreference = '.\EyeCandy.Vista.ps1'

    With the EyeCandy.Vista settings there is no mistaking an elevated PowerShell session from a non-elevated session.  :-)  The elevated session turns the background color to the DarkRed.  Now it turns out that in my humble opinion DarkRead isn't dark enough to get a good contrast between the background color and error output color.  This can be easily fixed though.  Fire up an elevated PowerShell.  If you are using the PSCX profile, just execute "su".  When the new PowerShell window appears it should have a dark red background.  Select the system menu (upper left corner by the PowerShell icon) and then select Properties.  Now select the Colors tab and then the Screen Background radio button.  Select the DarkRed color and change the "Red:" value to 80 as shown below:

    CWindowsSystem32WindowsPowerShellv1.0powershell.exe Properties (2)

    This gives a pretty good contrast for error output and makes darn sure you know when you are in an elevated PowerShell session on Vista.

    Sorting IPAddresses the PowerShell Way

    A question came up on the PowerShell newsgroup about how to sort IP addresses in PowerShell.  The poster had down this in Linux like so:

    #> cat ips | awk -F. '{printf("%03d.%03d.%03d.%03d\n", $1,$2,$3,$4)};' |
    sort -n -t "." | awk -F. '{printf("%d.%d.%d.%d\n", $1,$2,$3,$4)};'

    Where the file "ips" contains the following IP addresses:

    124.5.6.8
    2.4.53.233
    12.24.3.78
    234.2.5.7
    1.5.5.5

    You could do the equivalent lexicographical sort in PowerShell like this:  Update: Jacques Barathon and MoW have shortened this approach quite a bit by doing:

    > gc ips | sort {"{0:d3}.{1:d3}.{2:d3}.{3:d3}" -f @([int[]]$_.split('.')) 

    1.5.5.5
    2.4.53.233
    12.24.3.78
    124.5.6.8
    234.2.5.7

    Which produces the correct output.  This solution does a string sort after converting each quad so that there are leading zeros which enables string sorting to give the correct answer e.g.:

    124.005.006.008
    002.004.053.233
    012.024.003.078
    234.002.005.007
    001.005.005.005

    And after the sort it round trips the string quads to ints and then reformats them as text without the leading zeros.  However this solution doesn't take advantage of PowerShell's unique ability to manipulate domain objects like say an IPAddress object which just happens to exists in the .NET Framework.  So my first take at solving this was to use the System.Net.IPAddress type:

    > gc ips | %{[Net.IPAddress]$_} | sort Address | ft IPAddressToString

    IPAddressToString
    -----------------
    1.5.5.5
    234.2.5.7
    124.5.6.8
    12.24.3.78
    2.4.53.233

    Note: It is not necessary to use the 'System." prefix since PowerShell will do that for you if it can't find the type.  Now the command above would have been a nice solution that takes advantage of PowerShell's ability to manipulate objects - if it only worked.  Doh!  It turns out that the Address field value is in little endian byte order which won't work for sorting.  We need the bytes to be interpreted in big endian order.  We can do this but it admittedly is a bit uglier:

    > gc ips | %{[Net.IPAddress]::Parse($_)} | 
      sort {$b=$_.GetAddressBytes();[array]::Reverse($b);[BitConverter]::ToUInt32($b,0)} |
      ft IPAddressToString

    IPAddressToString
    -----------------
    1.5.5.5
    2.4.53.233
    12.24.3.78
    124.5.6.8
    234.2.5.7

    What we really need is a BigEndianAddress property on the System.Net.IPAddress object.  Good luck trying to get the BCL team at Microsoft to add that anytime in the next several years.  :-)  Fortunately we don't have to wait for them to do this.  PowerShell provides us with an extensible type system where we can add our own convenience properties to a pre-existing .NET object.  It is actually pretty easy.  First you need to construct an XML that tells PowerShell how to modify a particular .NET type:

    <?xml version="1.0" encoding="utf-8" ?>
    <Types> 
      <Type> 
        <Name>System.Net.IPAddress</Name> 
        <Members> 
          <ScriptProperty> 
            <Name>BigEndianAddress</Name> 
            <GetScriptBlock> 
              $bytes=$this.GetAddressBytes() 
              [array]::Reverse($bytes) 
              [BitConverter]::ToUInt32($bytes,0)        
            </GetScriptBlock> 
          </ScriptProperty> 
        </Members> 
      </Type>
    </Types>

    Save this to a file called IPAddress.ps1xml.  Now all you have to do is tell PowerShell to update its type data with the information in this file like so:

    > Update-TypeData IPAddress.ps1xml

    You might want to put this command in your profile so that the System.Net.IPAddress.BigEndianAddress is always available whenever you start up a new PowerShell session.  Now let's use our new property to sort IP addresses:

    > gc ips | %{[Net.IPAddress]$_} | sort BigEndianAddress | ft IPAddressToString

    IPAddressToString
    -----------------
    1.5.5.5
    2.4.53.233
    12.24.3.78
    124.5.6.8
    234.2.5.7

    That is how to do sorting "the PowerShell way"!  Just remember - use the force, err I mean objects!

    Using C# 3.0 Extensions Methods to Simplify WinForms Thread Marshalling

    Most folks know by now that if you don't call a WinForms control method on the same thread that created it, you are asking for trouble.  In fact on .NET 2.0 the WinForms team did the right thing in throwing an InvalidOperationException if you call a control from a different thread than which it was created.  There a few thread safe members which include InvokeRequired and Invoke.  In the past we used a pattern like so for methods that could be called by a different thread and the method itself would handle the thread marshalling e.g.:

    private delegate void UpdateProgressHandler(int percentComplete);
    private void UpdateProgress(int percentComplete) {
        if (InvokeRequired) {
            UpdateProgressHandler del = UpdateProgress;
            this.Invoke(del, percentComplete);
            return;
        } 
        _progressBar.Value = percentComplete;
    }   

    Ido Samuelson points a way to simplify this using C# 3.0 extension methods which is really pretty clever.  I recommend that you read his blog post here.  I modified his approach to eliminate the UpdateProgress call from the lambda expression.  Here's my approach:

    public static class ControlExtentions {
        public delegate void InvokeHandler(); 

        public static void SafeInvoke(this Control control, InvokeHandler handler) {
            if (control.InvokeRequired) {
                control.Invoke(handler);
            }
            else {
                handler();
            }
        }
    }

    private void UpdateProgress(int percentComplete)
    {
        this.SafeInvoke(() =>
        {
               _progressBar.Value = percentComplete;
        });
    }   

    Now if my method were more complex than a few lines then I would probably use Ido's approach.  Having a bunch of code in a lambda makes me a bit nervous - perhaps irrationally so.

    June 22

    Elevate function in PSCX 1.1.1

    Here's a useful function especially if you are on Vista and have left UAC enabled.  If you install PSCX 1.1.1 and use the PSCX default profile you will get this function dot sourced in via the profile.  However if you don't do either of those then here's the function posted here for your benefit:
     
    # ---------------------------------------------------------------------------
    ### <Function name='Elevate'>
    ### <Author>Keith Hill</Author>
    ### <Description>
    ### Runs the specified command via an elevated PowerShell instance on Vista.
    ### To get debug info, set $DebugPreference temporarily to 'Continue'.
    ### </Description>
    ### <Usage>
    ### elevate
    ### elevate notepad c:\windows\system32\drivers\etc\hosts
    ### elevate gci c:\windows\temp
    ### elevate {gci c:\windows\temp | export-clixml tempdir.xml; exit}
    ### elevate {gci c:\windows\temp | export-clixml tempdir.xml; exit} |
    ###         %{$_.WaitForExit(5000)} | %{Import-Clixml tempdir.xml}
    ### </Usage>
    ### </Function>
    # ---------------------------------------------------------------------------
    Set-Alias su Elevate -Option AllScope
    function Elevate {
     $ndx=0
     if ($MyInvocation.PositionMessage -match 'char:(\d+)') {
      $ndx = [int]$matches[1]
     }
     
     $setDirCmd = "[Environment]::CurrentDirectory = (set-location -LiteralPath '$pwd' -PassThru).ProviderPath"
     
     $OFS = ","
     Write-Debug "`$MyInvocation.Line args index is $ndx; `$args are <$args>"
     
     if ($args[0] -is [Scriptblock]) {
            $script = $args[0]
            Write-Debug "Starting PowerShell with scriptblock: {$script}"
            start-process powershell.exe -verb runas -workingdir $pwd `
                -arguments "-noexit -command & {$setDirCmd; $script}" 
     }
     elseif ($args[0].length -gt 0) {
            $startProcessArgs = $MyInvocation.Line.Substring($ndx)
           
            $app = get-command $args[0] | select -first 1 | ? {$_.CommandType -eq 'Application'}
            if ($app) {
                $startProcessArgs = $startProcessArgs.Substring($args[0].Length).Trim()
                Write-Debug "Starting <$($app.Path)> with args: <$startProcessArgs>"
                start-process $app.Path -verb runas -workingdir $pwd -arguments $startProcessArgs 
            }
            else {
                Write-Debug "Starting PowerShell with args: <$startProcessArgs>"
                start-process powershell.exe -verb runas -workingdir $pwd `
                    -arguments "-noexit -command & {$setDirCmd; $startProcessArgs}"        
            }
     }
     else {
            Write-Debug "Starting Powershell without args"
            start-process powershell.exe -verb runas  -workingdir $pwd `
                -arguments "-noexit -command & {$setDirCmd}" 
     }
    }

    PowerShell Community Extensions 1.1.1 Released!

    It took a little while but we finally got a lot of the issues resolved that you reported in PSCX 1.1.  Tonight (well *early* this morning) I released PSCX 1.1.1.  This release is a primarily a maintenance release.  It fixes about 30 defects and adds five new features.  You can download 1.1.1 from here.  You will need to uninstall any previous version you have of PSCX before you can install 1.1.1.  We really thought we had the VS 2005 Setup project configured to do an upgrade and it "seemed" to work.  However further testing revealed that sometimes it would not re-register the updated PSCX snapin with PowerShell.  Then PowerShell gets upset because the version number of the snapin doesn't match what has been registered with PowerShell.
     
    Here is the final list of changes in 1.1.1:

    NEW CMDLETS

    • Get-ExportedType

    NEW FUNCTIONS AND FILTERS

    • Collect
    • Elevate

    NEW SCRIPTS:

    • Get-ViewDefinition.ps1
    • Out-Speech.ps1
    • Select-Random.ps1

    NEW VARIABLES

    • $RegexLib

    NEW ALIASES

    • su (Elevate)

    Other changes include the new -PageSize parameter on Get-ADObject.  Select-Xml now outputs XPathDocumentNavigator objects instead of strings.  There is also a new elevate function (alias su) that will invoke the UAC elevation UI on Vista and the RunAs dialog box on XP.  For example:

    elevate
    elevate notepad c:\windows\system32\drivers\etc\hosts
    elevate gci c:\windows\temp
    elevate {gci c:\windows\temp | export-clixml tempdir.xml; exit}
    elevate {gci c:\windows\temp | export-clixml tempdir.xml; exit} |
            %{$_.WaitForExit(5000)} | Import-Clixml tempdir.xml

    Also if you ever wonder what other views are available for a particular object, try this:

    23> Get-ViewDefinition 'System.Diagnostics.Process'

    Name       : Priority
    Path       : C:\Windows\System32\WindowsPowerShell\v1.0\DotNetTypes.format.ps1xml
    TypeName   : System.Diagnostics.Process
    SelectedBy : System.Diagnostics.Process
    GroupBy    : PriorityClass
    Style      : Table

    Name       : StartTime
    Path       : C:\Windows\System32\WindowsPowerShell\v1.0\DotNetTypes.format.ps1xml
    TypeName   : System.Diagnostics.Process
    SelectedBy : System.Diagnostics.Process
    GroupBy    :
    Style      : Table

    Name       : process
    Path       : C:\Windows\System32\WindowsPowerShell\v1.0\DotNetTypes.format.ps1xml
    TypeName   : System.Diagnostics.Process
    SelectedBy : System.Diagnostics.Process
    GroupBy    :
    Style      : Wide

    On the download page for PSCX 1.1.1 you will be able to see the details on all the work items that went into this release.

    I would like to acknowledge and thank the following developers who helped out with the 1.1.1 release: Oisin Grehan, Richard Lehrbaum and Burt Harris!

    Please be sure to keep those work item submissions coming.  I can't promise that we will get the next release out any sooner but we do take defects seriously.  And we do appreciate all the folks out there that have been using PSCX and contributing by submitting work items and participating in the discussion forums on the PSCX CodePlex site.

    --
    Keith Hill
    PSCX Project Coordinator
    Windows PowerShell MVP

    June 18

    The Top Ten Habits/Traits of a Great Software Developer

    1. Masters their primary programming language and support library.  If you're a .NET developer around here that means C# and the .NET Framework.  This one is fairly obvious, right?  These tools are your bread and butter.
    2. Masters their development tools especially the debugging tools.  You spend a lot of time here and whatever you discover that can make you faster is likely to have a big impact on your productivity. Being effective at debugging can save you days on a single nasty bug.
    3. Writes unit tests for as much of their code as possible.  This is the best way
      to avoid embarrassing yourself with buggy code and to avoid regressions.
    4. Uses Internet search.  This should be one of the first steps you take when tracking down a problem.  Odds are somebody else has hit the same problem and posted something on the Internet about it.  Also uses search to avoid NIH (not invented here).  A great software developer would not re-invent software if it already existed and were freely available. If the software is expensive then they would do a make vs buy evaluation.
    5. Automates tedious and error prone manual processes whenever possible especially in the build and test processes.
    6. Knows some flavor of regular expressions.  Regex use comes up over and over again in software development, automation and computer administration.
    7. Masters a command line interface (preferrably Windows PowerShell if you're a .NET Developer).  Automation for one-off jobs is usually best accomplished via a shell especially when it comes to file system (source tree) maintenance.
    8. Masters software diagnostics tools.  Download and use the Windows Sysinternals tool suite - these are great diagnostic tools and they are free.  Also check out Wireshark or Microsoft's NetMon packet sniffer.
    9. Manages distractions - turn off toast and exit Outlook when you need to
      get into "the zone" and stay there to finish a complex coding or debugging
      task.  If you can manage an office with a door you are that much more ahead of the game.  Where I work, the best we can do is work from home to avoid distractions - unless it is summer and you have little kids :-).
    10. Continuing Education: reads a book every month or so to stay up on the
      latest techniques and/or technologies. Reads blogs and/or listens to podcasts  - there's a wealth of shared experience out there that you can learn from.

    This is just my opinion.  I'm curious about what others think.  What would be in your top ten list?

    June 17

    Optimizing Performance of Get-Content for Large Files

    Every so often a question comes across the PowerShell newsgroup microsoft.public.windows.powershell about why Get-Content is so slow when reading large log files.  The typical answer is to use Microsoft's LogParser utility.  However that isn't always satisfying if you want to use PowerShell specific features.  If this case, you really need to be able to get the best performance out of Get-Content when reading large files. 

    As for why the performance is not so good by default, I'll chalk that up to PowerShell being a v1.0 product.  The PowerShell team has said several times that the goal for v1.0 was high quality, not necessarily high performance.  I suspect that the product team is working on performance issues for a future release but as it stands the perf isn't too bad except for some special cases - like reading large files.

    Fortunately, the Get-Content cmdlet has a parameter that allows you to improve its performance by telling to send lines down the pipe in larger chunks i.e. array of lines instead of line-by-line.  Keep in mind that when you specify this via the parameter -ReadCount, Get-Content actually sends an array of strings down the pipeline instead of one string at a time.  This can have a surprising effect.  For instance this issue came up:

    (gc test.txt -read 1 | ? {$_ -like '*a*'} | Measure-Object).count
    (gc test.txt -read 5 | ? {$_ -like '*a*'} | Measure-Object).count

    In this case the count varies based on the value of -ReadCount.  That may seem counterintuitive *if* you are expecting that Measure-Object is counting the lines in test.txt.  However in this case, Measure-Object is just counting the number of objects it sees.  In the first command, this actually corresponds to the number of lines in the file because Get-Content passes each line as a string to the Where-Object (alias ?) cmdlet.  Where-Object simply evaluates the specified expression and if it evaluates to true then the object is passed down the pipeline unmodified.  In the second case, what happens is that Where-Object receives an array of 5 strings (System.Object[]).  The -like operator works on arrays as well as scalar values.  If the expression evaluates to true then whole array is sent to Measure-Object where that is counted as "one" object. 

    You can see this easily by executing:

    gc test.txt -read 5 | %{$_.psobject.typenames[0]}
    System.Object[]
    System.Object[]

    Now when reading large files like a 75 MB log file it helps performance to use a readCount of something like 1000.  Just remember that you will be getting arrays of lines instead of single lines down the pipeline.  However this is simple to rectify if you need to process line-by-line e.g.:

    (gc test.txt -read 5 | %{$_} | ? {$_ -like '*a*'} | Measure-Object).count

    Even with shredding the arrays the performance is still better using the higher readCount.  Here's an experiment that I ran to confirm this on a 75MB text file:

    23> $ht = @{};for ($i = 1; $i -le 10MB; $i *= 10) {
    >>      write-progress "Measuring gc -readCount $i" "% Complete" `
    >>          -perc ([math]::log10($i)*100/[math]::log10(10MB))
    >>      $ts = measure-command { gc large.txt -read $i | %{$_} | measure }
    >>      $ht[$i] = $ts
    >>  }
    >>
    24> $ht.Keys | sort | 
    >>    select @{n='ReadCount';e={$_}}, `
    >>           @{n='ElapsedTime';e={$ht[$_].ToString()}} | 
    >>    ft -a
    >>

    ReadCount ElapsedTime
    --------- -----------
            1 00:03:07.4161690
           10 00:01:38.5779661
          100 00:01:17.9219998
         1000 00:01:14.9202370
        10000 00:01:22.1434037
       100000 00:01:17.8457756
      1000000 00:01:17.9850525
     10000000 00:01:19.0217524

    If you happen to have the slick PowerGadgets snapin, then you can view the data like so:

    33> $ht.Keys | sort |
    >>    select @{n='ReadCount';e={"RC: $_"}},
    >>          
    @{n='ElapsedTime';e={$ht[$_].TotalSeconds}} |
    >>    out-chart -title 'Optimal ReadCount for 75MB Text File'
    >>

    PowerGadgets

    As you can see in this particular case (75 MB file) a readCount of 1000 is optimal but if in doubt - measure, measure, measure.  :-)

    June 16

    PowerShell Community Extensions 1.1.1 Soon

    The PSCX team is getting real close to releasing PSCX 1.1.1.  This is primarily a bug fix release where we have fixed the limitation of the ADSI provider stopping at 1000 objects plus ~30 other issues.  There are a few new features:

    NEW CMDLETS
        Get-ExportedType

    NEW FUNCTIONS AND FILTERS
        Collect
        Elevate

    NEW SCRIPTS:
        Get-ViewDefinition.ps1
        Out-Speech.ps1
        Select-Random.ps1

    NEW VARIABLES
        $RegexLib   

    NEW ALIASES
        su (Elevate)

    Get-ViewDefinition is handy if you are curious if there are alternate views for an object.  Right now the only way to find this out is via the help assuming the help covers this.  With Get-ViewDefiniton you can find out easily:

    6> get-process | select -first 1 | Get-ViewDefinition

    Name       : Priority
    Path       : C:\Windows\System32\WindowsPowerShell\v1.0\DotNetTypes.format.ps1xml
    TypeName   : System.Diagnostics.Process
    SelectedBy : System.Diagnostics.Process
    GroupBy    : PriorityClass
    Style      : Table

    Name       : StartTime
    Path       : C:\Windows\System32\WindowsPowerShell\v1.0\DotNetTypes.format.ps1xml
    TypeName   : System.Diagnostics.Process
    SelectedBy : System.Diagnostics.Process
    GroupBy    :
    Style      : Table

    Name       : process
    Path       : C:\Windows\System32\WindowsPowerShell\v1.0\DotNetTypes.format.ps1xml
    TypeName   : System.Diagnostics.Process
    SelectedBy : System.Diagnostics.Process
    GroupBy    :
    Style      : Wide

    We are testing right now.  I hope to have this version released by the end of June (hopefully sooner).  Unfortunately like the other PSCX instals, you will have to uninstall your current version of PSCX first.  Too bad we don't have an MSI/installer guru on the PSCX team.  I thought I had the VS Setup project beaten into submission so it would allow us to upgrade version 1.1 but alas when we tested this, the updated PSCX snapin doesn't get registered.  PowerShell doesn't like it when the version info it has registered with it doesn't match the PSSnapin that gets loaded.  Oh well, perhaps we can crack this nut when put out the release after 1.1.1.

    Quick Test for PowerShell Dot Source Library Scripts

    We tend to write a lot of shared PowerShell functions and variable definitions which we package into a library script that we dot source into multiple scripts.  In order to really test a change to our library scripts it requires running our lengthy automated test harness that takes almost 18 hours to complete.  Cleary that isn't ideal in terms of a quick test before checking in the script.  Here's a trick I use on dot sourced scripts like this which is essentially a quick sanity test before checking them in.  Say you have a script with an issue e.g.:

    @"
    function Greeting($name) {
        "Hi there $name
    }
    "@ > test.ps1

    Now let's dot source this script:

    & { . .\test.ps1 }
    Encountered end of line while processing a string token.
    At C:\TFS\CDF\Trunk\Tests\Scripts\test.ps1:2 char:5
    +     " <<<< Hi there

    Cool.  PowerShell tells me write away about the error.  We dot source this into a nested scope so we don't modify/pollute the global scope.  This is a real simple way to check to see if your "DOT SOURCE LIBRARY SCRIPT" has any parse errors in it. 

    NOTE: Be careful about running this on scripts that aren't dot sourced because the script will actually execute.  I know - duh.  It is just in the case of a dot source library script there is (typically) no directly executable code (except for perhaps setting variables), so dot sourcing these scripts is usually harmless i.e. side-effect free. 

    Let's correct the script:

    @"
    function Greeting($name) {
        "Hi there $name"
    }
    "@ > test.ps1

    & { . .\test.ps1 }

    And now we get no errors.  Anyway that is probably obvious to a lot of folks but perhaps someone will find it useful.  :-)