Keith Hill's Blog

Off Topic: Music, DRM and Tanking CD Sales

Sun, 15 Jun 2008 23:31:54 GMT

I took my 9 year old daughter to her first concert last night. The band was Lifehouse, a relatively newish, great alternative rock band.

I had seen them last summer as an opening act for the Goo Goo Dolls. I hadn't heard of Lifehouse before but was very impressed with their "big" sound. After that concert I bought two of their CDs. Yeah I'm one of those guys who still buys CDs because I despise DRM and I don't want to buy compressed/encoded (lossy) music. I'd rather do the compression and encoding myself. Plus I'll always have access to the raw, uncompressed music for future re-encoding in better formats.

Regarding DRM, I despise it because it isn't user friendly. It tends to be tied to a particular device instead of a person. Devices come and go and the pushups you have to go through to deactivate one device and activate another get to be a right pain in the rear. This is one of the reasons I stopped using Audible.com. I got tired of the whole deactivation/activation dance every year. Yeah, I go through new devices that often.

One very unfortunate downside to the ease with which music can be copied is piracy. I'm both a musician and a software developer so the notion of protecting intellectual property and copyrighted material hits close to home. After the concert, my daughter and I got to meet the band members who were signing autographs. What a great bunch of guys! They were all down to earth and very nice. I even got a picture of my daughter with Jason Wade - the lead singer. She was thrilled. :-)

I told them that I had seen them a year ago opening for the Goo Goo Dolls whom they were touring with. I asked if that helped boost CD sales because touring with GGD had to expose them to a wider audience. The reply shouldn't have surprised me but there is nothing like hearing it come right from the horse's mouth. Their guitarist said that CD sales across the industry have tanked. The only way they're making money these days is by touring, touring and more touring. That's a shame. I have no idea how to solve this problem but it starts with folks internalizing that copying copyrighted music without paying for it is stealing. That's certainly what I'll be teaching my kids.

Give Lifehouse a listen, they have some really good tunes and if you like what you hear, buy it. We need to support the arts, right? My favorite songs? Well most folks will recognize Hanging by a Moment but my favorite is First Time followed closely by Who We Are and Bridges.

BTW for you PowerShellers, check out this album cover:

The bald headed guy, who is Lifehouse's drummer, looks an awful lot like PowerShell's language designer - Bruce Payette. Check out this picture of Bruce on Amazon. I wonder if someone is leading a double life. :-) OK now back to our regularly scheduled PowerShell programming.

Poor Man's File/Directory Name Indexer Using Windows PowerShell

Tue, 27 May 2008 02:33:11 GMT

The following is a script that I have set up on my dev PCs to run nightly via a scheduled task:

## CatalogFileSystem.ps1
param([string[]]$paths)
Set-PSDebug -Strict

function Main {
    foreach ($path in $Paths) {
        if (!(Test-Path $path -PathType Container)) {
            Write-Error "'$path' doesn't exist or isn't a directory"
            exit 1
        }
		
        $filelist = Join-Path $path filelist.txt
        $dirlist  = Join-Path $path dirlist.txt
		
        Remove-Item $filelist -ErrorAction SilentlyContinue
        Remove-Item $dirlist  -ErrorAction SilentlyContinue
		
        CatalogFolder $path	
    }
}

function CatalogFolder($path) {
    Get-ChildItem $path -ErrorAction SilentlyContinue | sort FullName | foreach {
        if ($_.PSIsContainer) {
            $_.FullName >> $dirlist
            CatalogFolder $_.FullName
        }
        else {
            $_.FullName >> $filelist
        }
    }
}

. Main
## End of CatalogFileSystem.ps1

My scheduled task is configured like so (and runs everyday at 4 am):

Program: C:\Windows\System32\WindowsPowerShell\v1.0\PowerShell.exe
Arguments: -Command C:\Bin\CatalogFileSystem C:, D:

I also set this task to run with highest privileges on Vista so it can catalog more of the nooks and crannies of my filesystem. This script creates two files setting at the root of each path supplied. In my case:

C:\dirlist.txt
C:\filelist.txt
D:\dirlist.txt
D:\filelist.txt

Searching for files on my filesystem is now very easy:

PS> Select-String afxwin\.h \filelist.txt | select Line

Line
----
C:\Program Files\Microsoft Visual Studio 9.0\VC\atlmfc\include\afxwin.h
C:\Program Files\Microsoft Visual Studio 9.0\VC\ce\atlmfc\include\afxwin.h

In fact I have created a shortcut function for the above I call Find-File (alias ff):

Set-Alias ff Find-File
function Find-File($pattern) {
    $filelist = Join-Path $pwd.drive.root filelist.txt
    if (!(Test-Path $filelist -PathType Leaf)) {
        Write-Error "$filelist doesn't exist or isn't a file"
    }
    Select-String $pattern $filelist | foreach {$_.Line}
}

And then this sort of filename search gets even easier:

PS> ff afxwin\.h
C:\Program Files\Microsoft Visual Studio 9.0\VC\atlmfc\include\afxwin.h
C:\Program Files\Microsoft Visual Studio 9.0\VC\ce\atlmfc\include\afxwin.h

Now you might be wondering why I don't just use the search built into Vista. Well first off, Vista's search doesn't index your whole hard drive or any volumes other than the contents of C: by default. While it can be set up to index more locations I worry about the performance hit because the second issue is that the Vista search indexer runs in the background all the time. The CatalogFileSystem script runs just once a day (or however often you schedule it) when you're typically not using the computer. Yeah, it could be as much as 24 hours out of date but the vast majority of the time I'm searching for files that have been on my system for a while: C++ header files, SDK header files, C runtime source files, windows DLLs, etc.

I'm sure some folks are going to think this is goofy but honestly I do search the filelist.txt file quite often (not so much the dirlist.txt file). Do you have any similar convenience scripts like this? If you do, please add a comment and let me know.

Northern Colorado .NET User Group Windows PowerShell Presentation

Tue, 13 May 2008 05:13:54 GMT

Tonight I gave a one hour introductory talk on Windows PowerShell to my local .NET user group. The primary focus of this talk was to show how PowerShell should be interesting to .NET developers. As promised, here is the slide deck and the samples. If you have any questions, please let me know.

Effective PowerShell Item 13: Comparing Arrays in Windows PowerShell

Sun, 11 May 2008 18:41:30 GMT

PowerShell has a lot of useful operators such as -contains which tests if an array contains an particular element. But as far as I can tell PowerShell doesn't "seem" to provide an easy way to test if two array's contents are equal. This if often quite handy and I was a bit surprised by this apparent omission.

I came upon this need to compare arrays while answering a question on the microsoft.public.windows.powershell newsgroup. The poster wanted to find UTF8 encoded files by inspecting their BOM or byte order mark. One relatively straight forward approach to this is:

PS> $preamble = [System.Text.Encoding]::UTF8.GetPreamble()
PS> $preamble | foreach {"0x{0:X2}" -f $_}
0xEF
0xBB
0xBF
PS> $fileHeader = Get-Content Utf8File.txt -Enc byte -Total 3
PS> $fileheader | foreach {"0x{0:X2}" -f $_}
0xEF
0xBB
0xBF

While it is easy enough to visually inspect this and see we have a match, visual inspection doesn't work in a script. :-) You could also test each individual element which isn't bad for a three element array but when you hit say 10 elements that approach might starting looking tedious.

You might think that we could just compare these two arrays directly like so:

PS> $preamble -eq $fileHeader | Get-TypeName # Get-TypeName is from the PowerShell Community Extensions
WARNING: Get-TypeName did not receive any input. The input may be an empty collection. You can either 
prepend the collection expression with the comma operator e.g. ",$collection | gtn" or you can pass the 
variable or expression to Get-TypeName as an argument e.g. "gtn $collection".
PS> $preamble -eq 0xbb
187

But comparing arrays via the -eq operator doesn't actually compare the contents of two arrays. As you can see above, this results in no output. When the left hand side of the -eq operator is an array, PowerShell return the elements of the array that match the value specified on the right hand side (shown above where I test for -eq to 0xbb).

OK so it looks like we need to roll our own mechanism to compare arrays. Here is one way:

function AreArraysEqual($a1, $a2) {
    if ($a1 -isnot [array] -or $a2 -isnot [array]) {
      throw "Both inputs must be an array"
    }
    if ($a1.Rank -ne $a2.Rank) {
      return $false
    }
    if ([System.Object]::ReferenceEquals($a1, $a2)) {
      return $true
    }
    for ($r = 0; $r -lt $a1.Rank; $r++) {
      if ($a1.GetLength($r) -ne $a2.GetLength($r)) {
            return $false
      }
    }

    $enum1 = $a1.GetEnumerator()
    $enum2 = $a2.GetEnumerator()

    while ($enum1.MoveNext() -and $enum2.MoveNext()) {
      if ($enum1.Current -ne $enum2.Current) {
            return $false
      }
    }
    return $true
}

And it works as expected:

PS> AreArraysEqual $preamble $fileHeader
True

However there turns out to be a way to do this within PowerShell but it isn't exactly obvious. At least it wasn't to me - at first.

PS> @(Compare-Object $preamble $fileHeader -sync 0).Length -eq 0
True

Good old Compare-Object will compare the arrays and if there are no differences it won't output anything. If we wrap the output of Compare-Object in an array subexpression @() then we will get an array with either 0 or more elements. A simple compare of the length to 0 will confirm that there was no output, hence the arrays are equal.

[Updated: 5/12/2008 - need to use -SyncWindow 0 to get correct result - thanks Arnoud and Roman] Let me elaborate more on this updated information. As Roman points out in the comments on this post, Compare-Object compares two objects to see if they have the same set of elements. Normally it does not care if the elements are in the same sequence in each object (each array in this case). For example:

PS> $a1 = 1,1,2
PS> $a2 = 1,2,1
PS> @(Compare-Object $a1 $a2).length -eq 0
True

Obviously that isn't what we want when comparing arrays for equality. Fortunately, as Arnoud points out, we can use the SyncWindow parameter with a value 0 to get Compare-Object to "force sequence equality" as Arnoud succinctly phrases it.

How about performance of these two approaches:

PS> $a1 = 1..10000
PS> $a2 = 1..10000
PS> (Measure-Command { AreArraysEqual $a1 $a2 }).TotalSeconds
1.236252
PS> (Measure-Command { @(Compare-Object $a1 $a2 -sync 0).Length -eq 0 }).TotalSeconds
0.3259954

Compare-Object beats out my PowerShell function by a good margin which isn't too surprising[1]. After all, one is compiled code and the other is interpreted script. So there you have it. If you need a quick way to compare to arrays, just remember that arrays are objects too and that is what Compare-Object does best - compare two objects.

[1] - Except for comparing against the same array where my function is two orders of magnitude faster. It seems that the Compare-Object cmdlet could benefit from a quick System.Object.ReferenceEquals check. :-) Admittedly this is a bit of a corner case scenario.

Effective PowerShell Item 12: Understanding ByValue Pipeline Bound Parameters

Sat, 10 May 2008 03:32:30 GMT

In item 11, I covered ByPropertyName pipeline bound parameters. In this post, I'll cover the other variety of pipeline binding - ByValue. ByValue binding takes the input object itself and attempts to bind it by type using type coercion if possible to parameters decorated as ByValue. For example, most of the *-Object utility cmdlets operate on whatever object is presented to them. The help on Where-Object shows this:

-inputObject <psobject>
    Specifies the objects to be filtered. If you save the output of a command in a variable,
    you can use InputObject to pass the variable to Where-Object. However, typically, the
    InputObject parameter is not typed in the command. Instead, when you pass an object
    through the pipeline, Windows PowerShell associates the passed object with the
    InputObject parameter.

    Required?                    false
    Position?                    named
    Default value
    Accept pipeline input?       true (ByValue)
    Accept wildcard characters? false

It turns out that ByValue isn't nearly as popular as ByPropertyValue. How can I make such a statement you ask? Well this is one of the things that I love about PowerShell. It provides so much metadata about itself. It is very "self describing". You can easily walk every parameter on every cmdlet that is currently loaded into PowerShell. First let's see what information is available for a parameter:

PS> Get-Command -CommandType cmdlet | Select -Expand ParameterSets | 
>>  Select -Expand Parameters -First 1 | Get-Member
>>

   TypeName: System.Management.Automation.CommandParameterInfo

Name                                MemberType Definition
----                                ---------- ----------
...
Aliases                             Property   System.Collections.ObjectModel.ReadOnlyCollection`1[[...
Attributes                          Property   System.Collections.ObjectModel.ReadOnlyCollection`1[[...
HelpMessage                         Property   System.String HelpMessage {get;}
IsDynamic                           Property   System.Boolean IsDynamic {get;}
IsMandatory                         Property   System.Boolean IsMandatory {get;}
Name                                Property   System.String Name {get;}
ParameterType                       Property   System.Type ParameterType {get;}
Position                            Property   System.Int32 Position {get;}
ValueFromPipeline                   Property   System.Boolean ValueFromPipeline {get;}
ValueFromPipelineByPropertyName     Property   System.Boolean ValueFromPipelineByPropertyName {get;}
ValueFromRemainingArguments         Property   System.Boolean ValueFromRemainingArguments {get;}

The interesting properties for us here are the Name and ValueFromPipeline* properties. Given this information it is easy to figure out how many of each type there are:

PS> (Get-Command -CommandType cmdlet | Select -Expand ParameterSets | Select -Expand Parameters |
>>   Where {$_.ValueFromPipeline -and !$_.ValueFromPipelineByPropertyName} | Measure-Object).Count
>>
55
PS> (Get-Command -CommandType cmdlet | Select -Expand ParameterSets | Select -Expand Parameters |
>>   Where {!$_.ValueFromPipeline -and $_.ValueFromPipelineByPropertyName} | Measure-Object).Count
>>
196
PS> (Get-Command -CommandType cmdlet | Select -Expand ParameterSets | Select -Expand Parameters |
>>   Where {$_.ValueFromPipeline -and $_.ValueFromPipelineByPropertyName} | Measure-Object).Count
>>
66

So from here we can see the following:

Type of Pipeline Binding	Count
ValueFromPipeline	55
ValueFromPipelineByPropertyName	196
Both	66

So indeed binding by property name is much more common. Binding by value from the pipeline is primarily for cmdlets that manipulate objects. In the query below we can see that the InputObject parameter is by far the most common "ByValue" pipeline bound parameter:

PS> Get-Command -CommandType cmdlet | Select -Expand ParameterSets | Select -Expand Parameters |
>>  Where {$_.ValueFromPipeline -and !$_.ValueFromPipelineByPropertyName} |
>>  Group Name -NoElement | Sort Count -Desc
>>

Count Name
----- ----
   40 InputObject
    4 Message
    3 String
    2 SecureString
    1 ExecutionPolicy
    1 Object
    1 AclObject
    1 DifferenceObject
    1 Id
    1 Command

A little further digging reveals the cmdlets that use the ByValue bound InputObject parameters as shown below. Note that a single parameter can appear in more than one parameter set on a cmdlet, which explains why there are only 36 cmdlets that account for the 40 instances of InputObject.

PS> $CmdletName = @{Name='CmdletName';Expression={$_.Name}}
PS> Get-Command -CommandType cmdlet | Select $CmdletName -Expand ParameterSets |
>>  Select CmdletName -Expand Parameters |
>>  Where {$_.ValueFromPipeline -and !$_.ValueFromPipelineByPropertyName} |
>>  Group Name | Sort Count -Desc | Select -First 1 | Foreach {$_.Group} |
>>  Sort CmdletName -Unique | Format-Wide CmdletName -AutoSize
>>


Add-History      Add-Member       ConvertTo-Html   Export-Clixml    Export-Csv      ForEach-Object
Format-Custom    Format-List      Format-Table     Format-Wide      Get-Member      Get-Process
Get-Service      Get-Unique       Group-Object     Measure-Command  Measure-Object  Out-Default
Out-File         Out-Host         Out-Null         Out-Printer      Out-String      Restart-Service
Resume-Service   Select-Object    Select-String    Sort-Object      Start-Service   Stop-Process
Stop-Service     Suspend-Service  Tee-Object       Trace-Command    Where-Object    Write-Output

As you can see most of these cmdlets are designed to deal with objects in general. Note to cmdlet developers - pipeline bound parameters is how your cmdlets receive pipeline objects. When writing a cmdlet there is no $_. If your cmdlet wants to "participate" in the pipeline it must set the ParameterAttribute property ValueFromPipeline and/or ValueFromPipelineByPropertyName to true on at least one of its parameters.

As mentioned above most ByValue parameters are of the InputObject (type psobject or psobject[]) variety so they pretty much accept anything. However not all cmdlets work that way. The -Id parameter (type [long[]]) on Get-History is pipeline bound ByValue. The follow Trace-Command output shows how PowerShell works hard when necessary to convert the input object's type to the expected type. In this case a scalar string value of '1' to an array of Int64:

PS> Trace-Command -Name ParameterBinding -PSHost -Expression {'1' | get-history}

BIND NAMED cmd line args [Get-History]
BIND POSITIONAL cmd line args [Get-History]
MANDATORY PARAMETER CHECK on cmdlet [Get-History]
CALLING BeginProcessing
BIND PIPELINE object to parameters: [Get-History]
    PIPELINE object TYPE = [System.String]
    RESTORING pipeline parameter's original values
    Parameter [Id] PIPELINE INPUT ValueFromPipeline NO COERCION
    BIND arg [1] to parameter [Id]
        Binding collection parameter Id: argument type [String], parameter type
             [System.Int64[]], collection type Array, element type [System.Int64],
             no coerceElementType
        Creating array with element type [System.Int64] and 1 elements
        Argument type String is not IList, treating this as scalar
        BIND arg [1] to param [Id] SKIPPED
    Parameter [Id] PIPELINE INPUT ValueFromPipeline WITH COERCION
    BIND arg [1] to parameter [Id]
        COERCE arg type [System.Management.Automation.PSObject] to [System.Int64[]]
            ENCODING arg into collection
            Binding collection parameter Id: argument type  [PSObject], parameter type
                 [System.Int64[]], collection type Array, element type [System.Int64],
                 coerceElementType
            Creating array with element type [System.Int64] and 1 elements
            Argument type PSObject is not IList, treating this as scalar
            COERCE arg type [System.Management.Automation.PSObject] to [System.Int64]
                CONVERT arg type to param type using LanguagePrimitives.ConvertTo
                CONVERT SUCCESSFUL using LanguagePrimitives.ConvertTo: [1]
            Adding scalar element of type Int64 to array position 0
        Executing VALIDATION metadata: [System.Management.Automation.ValidateRangeAttribute]
        BIND arg [System.Int64[]] to param [Id] SUCCESSFUL
MANDATORY PARAMETER CHECK on cmdlet [Get-History]
CALLING ProcessRecord
CALLING EndProcessing

Note that on the first attempt, PowerShell tries to convert the string to an array of Int64 and fails. Then it tries again by treating the input as psobject. It throws that psobject at an internal help class method LanguagePrimitives.ConvertTo() that successfully converts the string '1' to an Int64[] containing the value 1.

When a parameter is both ByValue and ByPropertyName bound, PowerShell attempts to bind in this order:

Bind ByValue with no type conversion
Bind ByPropertyName with no type conversion
Bind ByValue with type conversion
Bind ByPropertyName with type conversion

There is more to the parameter binding algorithm like finding the best match amongst different parameter sets. BTW one last tidbit related to parameters. The PowerShell help topics aren't completely automatically generated and as a result they aren't always correct. For instance, look up the parameters on Get-Content and see if you find a -Wait parameter - you won't. :-) However the metadata is always complete and correct e.g.:

PS> Get-Command Get-Content -Syntax
Get-Content [-Path] <String[]> [-ReadCount <Int64>] [-TotalCount <Int64>] [-Filter <String>] 
  [-Include <String[]>] [-Exclude <String[]>] [-Force] [-Credential <PSCredential>] [-Verbose] 
  [-Debug] [-ErrorAction <ActionPreference>] [-ErrorVariable <String>] [-OutVariable <String>] 
  [-OutBuffer <Int32>] [-Delimiter <String>] [-Wait] [-Encoding <FileSystemCmdletProviderEncoding>]
Get-Content [-LiteralPath] <String[]> [-ReadCount <Int64>] [-TotalCount <Int64>] [-Filter <String>]
  [-Include <String[]>] [-Exclude <String[]>] [-Force] [-Credential <PSCredential>] [-Verbose] 
  [-Debug] [-ErrorAction <ActionPreference>] [-ErrorVariable <String>] [-OutVariable <String>] 
  [-OutBuffer <Int32>] [-Delimiter <String>] [-Wait] [-Encoding <FileSystemCmdletProviderEncoding>]

Hopefully this post has given you more knowledge about ByValue parameters and how to explore and get more information on cmdlet parameters in general. In summary, there actually isn't much you need to know about ByValue pipeline bound parameters because in most cases they just work intuitively. Just be sure to keep your eye out for those parameters that bind ByPropertyName. They are the ones whose pipeline bound usage isn't as obvious.

Windows PowerShell V2 CTP2 Is Available

Sat, 03 May 2008 17:11:04 GMT

The PowerShell team just posted the announcement late last night. There is a download link in the announcement. This drop has lots of new features like Module support for organizing and loading related functionality as well as transaction support for registry operations using the Registry provider. There have also been some changes since the first CTP so take a look at the release notes.

Let me reiterate the caution that these are "preview" (the P in CTP) bits. There aren't even consider beta quality although I would qualify that caution mostly applies to the new functionality. That said, I would not put these bits on a production machine. In fact, there have been reports about incompatibility between the System Center Virtual Machine Manager latest drop and V2 CTP bits.

If you don't have a spare/test PC on which to play with these bits then go grab Virtual PC (free), Virtual Server or VMWare and create a sandbox VM with which to play with this CTP. You will want to create that image based on either Vista SP1 or Windows Server 2008 if you want to take the remote management features for a spin.

One last thing. The Graphical PoweShell is new in this release and is in need of feedback. Folks, now is the time to influence the direction of this feature. If you wait too long the team won't have time to make anything but very minor changes. So please use it for a while and send your feedback to gPSFback@microsoft.com.

PowerShell Makes It Into Top 50 Programming Languages!

Fri, 25 Apr 2008 15:41:30 GMT

At #46 in the TIOBE Programming Community Index for April 2008, PowerShell sits above such languages as:

AppleScript
Boo
Caml
Csh
Eiffel
Modula-2
Oberon
VBScript
XSLT

Now what can we do to bump it up the list a few spots past REXX (44), Tcl/Tk (39) and Bash (37)? Hmm....

Effective PowerShell Item 11: Understanding ByPropertyName Pipeline Bound Parameters

Mon, 07 Apr 2008 05:33:20 GMT

Everybody likes to be efficient, right? I mean we all generally like to solve a problem in an efficient way. In PowerShell that usually culminates in a "one-liner". Honestly for pedagogical purposes I find it better much better to expand these terse, almost 'Obfuscated C' style commands into multiple lines. However there is no denying that when you want to bang out something quick at the console - given PowerShell's current line editing features - a one-liner helps stave off repetitive stress injuries. It's not PowerShell's fault. They're just using the antiquated console subsystem in Windows that hasn't changed much since NT shipped in 1993.

One trick to less typing is to take advantage of pipeline bound parameters. Quite often I see folks write a command like:

PS> Get-ChildItem . *.cs -r | foreach { get-content $_.fullname } | ...

That works but the use of the Foreach-Object cmdlet is technically unnecessary. Many PowerShell cmdlets bind their "primary" parameter to the pipeline. This is indicated in the help file for Get-Content as shown below:

-path <string[]>
Specifies the path to an item. Get-Content retrieves the content of the item. Wildcards
are permitted. The parameter name ("-Path" or "-FilePath") is optional.

    Required?                    true
    Position?                    1
    Default value                N/A - The path must be specified
    Accept pipeline input?       true (ByPropertyName)
    Accept wildcard characters? true

<snip>

-literalPath <string[]>
    Specifies the path to an item. Unlike Path, the value of LiteralPath is used
    exactly as it is typed. No characters are interpreted as wildcards. If the path
    includes escape characters, enclose it in single quotation marks.
    Single quotation marks tell Windows PowerShell not to interpret any characters as escape sequences.

    Required?                    true
    Position?                    1
    Default value
    Accept pipeline input?       true (ByPropertyName)
    Accept wildcard characters? false

Note that there are actually four parameters on Get-Content that accept pipeline input ByPropertyName. Two of which are shown above. The other two are ReadCount and TotalCount. The qualifier ByProperyName simply means that if the incoming object has a property of that name it is available to be "bound" as input to that parameter. That is, if a type match can be found or coerced.

For instance, we could simplify the command above by eliminating the Foreach-Object cmdlet altogether:

PS> Get-ChildItem . *.cs -r | get-content | ...

While it is intuitive that Get-Content should be able to handle the System.IO.FileInfo objects that Get-ChildItem outputs, it isn't obvious based on the ByPropertyValue rule I just mentioned. Why? Well the FileInfo objects output by Get-ChildItem don't have either a Path property or a LiteralPath property even accounting for the extended properties like PSPath. So how the heck does Get-Content determine the path of a file in this pipeline scenario? There are at least two ways to find this out. The first is the easier approach. It uses a PowerShell cmdlet called Trace-Command that shows you how PowerShell binds parameters. The second approach involves spelunking in the PowerShell assemblies using Lutz Roeder's .NET Reflector. Let's tackle this problem initially using Trace-Command.

Trace-Command is a built-in tracing facility that shows a lot of the inner workings of PowerShell. I will warn you that it tends to be prolific with its output. One particularly useful area you can trace is parameter binding. Here's how we would do this for the command above:

PS> Trace-Command -Name ParameterBinding -PSHost -Expression { Get-ChildItem log.txt | get-content }

This outputs a lot of text and unfortunately it is "Debug" stream text that isn't easily searchable or redirectable to a file. Oh well. The interesting output from this command are the following lines:

     BIND PIPELINE object to parameters: [Get-Content]
         PIPELINE object TYPE = [System.IO.FileInfo]
         RESTORING pipeline parameter's original values
         Parameter [ReadCount] PIPELINE INPUT ValueFromPipelineByPropertyName NO COERCION
         Parameter [TotalCount] PIPELINE INPUT ValueFromPipelineByPropertyName NO COERCION
         Parameter [Path] PIPELINE INPUT ValueFromPipelineByPropertyName NO COERCION
         Parameter [Credential] PIPELINE INPUT ValueFromPipelineByPropertyName NO COERCION
         Parameter [ReadCount] PIPELINE INPUT ValueFromPipelineByPropertyName NO COERCION
         Parameter [TotalCount] PIPELINE INPUT ValueFromPipelineByPropertyName NO COERCION
         Parameter [LiteralPath] PIPELINE INPUT ValueFromPipelineByPropertyName NO COERCION
         BIND arg [Microsoft.PowerShell.Core\FileSystem::C:\Users\Keith\log.txt] to parameter [LiteralPath]

This output has been simplified a bit to be more readable in this post. I also changed the initial command to output just a single FileInfo object to reduce the amount of output. The information we get from Trace-Command shows us that PowerShell tries to bind the FileInfo object to the Get-Content parameters and fails (NO COERCION) on all except for the LiteralPath parameter. OK well that tells us definitively how Get-Content is getting the path but it doesn't make sense. There is no LiteralPath property on a FileInfo object and there is no extended property called LiteralPath either.

This is where the second technique of using .NET Reflector (download here) can be used to see a reverse compiled version of the PowerShell source. After starting .NET Reflector and loading the Microsoft.PowerShell.Commands.Management.dll assembly, we find the GetContentCommand and inspect the LiteralPath parameter shown below:

[Alias(new string[] { "PSPath" }),
 Parameter(Position = 0, ParameterSetName = "LiteralPath", Mandatory = true, ValueFromPipeline = false,
           ValueFromPipelineByPropertyName = true)]
public string[] LiteralPath { }

Note the Alias attribute on this parameter. It creates another valid name for the LiteralPath parameter - PSPath which corresponds to the extended PSPath property on all FileInfo objects. That is what allows the ByPropertyName pipeline input binding to succeed. The property named PSPath matches the parameter name albeit via an alias.

Where does that leave us? There are a number of cases where we can pipe an object directly to a cmdlet in the next stage of the pipeline because of pipeline input binding where PowerShell searches for the most appropriate parameter to bind that object to.

Here is another example of piping directly to another cmdlet without resorting to the use of the Foreach-Object cmdlet:

PS> Get-ChildItem *.txt | Rename-Item -NewName {$_.name + '.bak'}

You also now have a way to determine how PowerShell binds pipeline input to a parameter of a cmdlet. And thanks to Reflector we know that some parameters have aliases like PSPath to assist in this binding process.

That's it for ByPropertyName pipeline input binding. There is another type of pipeline input binding called ByValue that I'll cover in a future post.

Nothing's Perfect Including PowerShell

Sun, 02 Mar 2008 22:36:57 GMT

Today I needed to count the number of errors in a log file. Pretty straightforward stuff that I would typically accomplish like so:

5> Select-String '^\d+,Error' Messages.log | Measure-Object

And that normally works well for me - except for today. It turns out that this log file is big, really big - as in 600MB worth of log file! The command above runs for quite some time and then fails ignominiously with a System.OutOfMemoryException. Sure enough, a quick execution of "gps -id $pid" revealed that the PowerShell process was consuming 1.7 GB of private memory. No wonder we hit an OOM exception.

So back to the drawing board on how to accomplish this in PowerShell. But first I had to do something about the memory footprint of my current PowerShell session. In PowerShell Community Extensions we have a Collect function (which just calls [System.GC]::Collect()). This brought the private memory footprint back down to ~76MB which tells me that PowerShell's pipeline or one of the cmdlets above is hoarding memory. No matter. One of the best things about PowerShell is this awesome escape hatch it provides - direct access to the .NET Framework. Fortunately there is a simple class in the .NET Framework called System.IO.StreamReader that allows you to read text files a line at a time which is important when you' re dealing with huge log files. Here is the resulting solution I came up with:

7> $sr = new-object System.IO.StreamReader("$pwd\Messages.log")
8> $sum = 0; while (($line = $sr.ReadLine()) -ne $null) {if ($line -match '^\d+,Error') {$sum++}}; $sum
2702996
9> $sr.Dispose()

I monitored the private memory usage of the PowerShell process during the execution of this script. The private memory usage increased about 200K and then didn't budge until the script was finished. No doubt this contributed to the script finishing much faster as compared to the time it took my first attempt to finish, err, run out of memory - 1 min 43 secs versus 7 min 16 secs respectively.

When it comes to reading files, another useful .NET Framework method is the static method: [System.IO.File]::ReadAllText(string path) which returns a single string containing the file's entire contents. If you ever need to load the entire contents of a file into a variable for manipulation (say you need to execute a regex over an entire file's contents - not just line-by-line) this method is a good way to go. I find the ReadAllText() method a bit easier to use in this case than Get-Content piped to Out-String. The other benefit of ReadAllText() is that it doesn't add an extra line terminator to the end of the string which is something Out-String will do. It seems like Get-Content should have a parameter to indicate that it should read the entire file into a single string and output that.

PowerShell Community Extensions - It's the Little Things

Wed, 20 Feb 2008 18:55:33 GMT

Today I was posting about an issue I was having with Team Foundation Server 2008 HTML alert emails that get generated for check-ins. The second column in the details area of the email is way too narrow. I was able to copy/paste the HTML into Expression Web so I could get to the HTML. I wanted to post that HTML into the MSDN Forum post editor but, well, it doesn't like you putting in non-escaped HTML. Fortunately PowerShell and PSCX are at my beckon call. Here is all I had to do to escape my HTML so I could paste it into the HtmlView of the MSDN forum editor:

PS> Get-Clipboard | Foreach {[System.Web.HttpUtility]::HtmlEncode($_)} | Out-Clipboard

I just love tools that save me time!

Microsoft Dev Days in Denver

Tue, 08 Jan 2008 00:10:05 GMT

If you want to get some free exposure to the VS 2008 and Office technologies, sign up for Dev Days. It is coming to my local MS office (Denver) - details listed below.

When: Thursday, January 31, 2008 8:30 AM - 5:00 PM
Where: Marriott DTC, 4900 S. Syracuse St, Denver Colorado 80237
Registration is appreciated: Click here to register

XPath Expressions and PSCX's Select-Xml

Wed, 02 Jan 2008 08:20:44 GMT

MoW wrote up a nice post on invoking XPATH expressions. Check it out here. Just wanted to let the PSCX users out there know that the equivalent of MoW's

PS I:\PowerShell> Function invoke-XpathExpression ([xml]$xml,$expression) {                                             
>>   $xn = $xml.PSBase.CreateNavigator()                                                                                
>>   $xn.Evaluate($expression)                                                                                          
>> }                                                                                                                    
>>    

                                                                                                                  
PS I:\PowerShell> # Example using the function                                                                          
PS I:\PowerShell>                                                                                                       
PS I:\PowerShell> invoke-XpathExpression -xml (type gl.xml) -exp "sum(GroceryList/Item/Price)"                          
29.15

in PSCX would be:

PS> Select-Xml gl.xml -xpath '/GroceryList/Item/Price' | measure value -sum


Count    : 4
Average  :
Sum      : 29.15
Maximum  :
Minimum  :
Property : Value

Note that the PSCX cmdlet Select-Xml is oriented towards "selecting" node-sets hence the name. Unfortunately xpath expressions that don't result in node-sets will error. No worries though because PowerShell's Measure-Object cmdlet (measure alias provided by PSCX) can compute the sum easily.

OK so that was a bit shorter but what's the big deal. Here's the deal. If your XML uses XML namespaces then this all gets a good bit harder to deal with yourself. Not impossible mind you. I have written a number of posts on handling XML that uses XML namespaces but with Select-Xml it is pretty simple. For instance, let's tweak the XML ever so slightly:

<GroceryList xmlns="tempuri.org">
<Item>
    <Dept>Produce</Dept>
    <Name>Orange</Name>
    <Price>3.20</Price>
</Item>
<Item>
    <Dept>Meat</Dept>
    <Name>Steak</Name>
    <Price>13.20</Price>
</Item>
<Item>
    <Dept>Produce</Dept>
    <Name>Lettuce</Name>
    <Price>1.34</Price>
</Item>
<Item>
    <Dept>Meat</Dept>
    <Name>Ham</Name>
    <Price>11.41</Price>
</Item>
</GroceryList>

Note the default namespace declaration on the root element. Now the previous XPath expressions won't work but here is all we need to do with Select-Xml to make this work:

PS> Select-Xml gl.xml -xpath '/ns:GroceryList/ns:Item/ns:Price' -Namespace "ns=tempuri.org" |
>> measure value -sum
>>

Count    : 4
Average  :
Sum      : 29.15
Maximum  :
Minimum  :
Property : Value

All we needed to do was provide the namespace and a temp prefix (ns) to use in the xpath query. Note that the -Namespace parameter will take an array of strings that match this format: "<prefix>=<namespace>".

Renewed as Windows PowerShell MVP for 2008

Wed, 02 Jan 2008 07:13:27 GMT

Woohoo! Just got word earlier today. I look forward to another awesome year of PowerShell adoption and just maybe v 2.0 - hopefully.

Windows PowerShell Help Available Online

Tue, 18 Dec 2007 17:39:34 GMT

Just got word that the PowerShell help is now available online on the TechNet site. This is very nice because we will now be able to reference PowerShell help topics by URL like say the help topic for the Certificate provider.

Visual Studio 2008 Training Kit Requires PowerShell

Sun, 02 Dec 2007 17:32:57 GMT

I just downloaded this training kit and took a look at its requirements and guess what - PowerShell is required! Cool.

Prerequisite software
The following software packages must be installed in order to complete the labs in this training kit. Items that are required only for specific parts of the kit are indicated below.

Microsoft Windows Vista
Microsoft Visual Studio 2008
Microsoft SQL Server 2005 (Express recommended)
Microsoft IIS 7 (All components)
Microsoft Office Powerpoint 2007 or the PowerPoint Viewer 2007 - Required to view the presentations
Windows PowerShell 1.0 RTM

After looking through the installed files, it appears that PowerShell is being used for primarily testing code via verify.ps1 and testLib.ps1 scripts. Here are all the PowerShell script files used by the training kit:

D:\VS2008TrainingKit\Labs\CardSpace\Ex01-RegisterWithCardSpace\PowerShellTests\verify.ps1
D:\VS2008TrainingKit\Labs\CardSpace\Setup\scripts\elevate.ps1
D:\VS2008TrainingKit\Labs\CardSpace\Setup\scripts\powershellscripts\CertificatesLib.ps1
D:\VS2008TrainingKit\Labs\CardSpace\Setup\scripts\powershellscripts\HostLib.ps1
D:\VS2008TrainingKit\Labs\CardSpace\Setup\scripts\powershellscripts\setdefaultdocument.ps1
D:\VS2008TrainingKit\Labs\DynamicSites\Ex01-DataAccessWithLINQ\begin\PowerShellTests\testLib.ps1
D:\VS2008TrainingKit\Labs\DynamicSites\Ex01-DataAccessWithLINQ\begin\PowerShellTests\verify.ps1
D:\VS2008TrainingKit\Labs\DynamicSites\Ex01-DataAccessWithLINQ\end\PowerShellTests\testLib.ps1
D:\VS2008TrainingKit\Labs\DynamicSites\Ex01-DataAccessWithLINQ\end\PowerShellTests\verify.ps1
D:\VS2008TrainingKit\Labs\DynamicSites\Ex02-ExposeData\begin\PowerShellTests\testLib.ps1
D:\VS2008TrainingKit\Labs\DynamicSites\Ex02-ExposeData\begin\PowerShellTests\verify.ps1
D:\VS2008TrainingKit\Labs\DynamicSites\Ex02-ExposeData\begin\PowerShellTests\vs90vars.ps1
D:\VS2008TrainingKit\Labs\DynamicSites\Ex02-ExposeData\begin\PowerShellTests\wcfserviceTest.ps1
D:\VS2008TrainingKit\Labs\DynamicSites\Ex02-ExposeData\end\PowerShellTests\testLib.ps1
D:\VS2008TrainingKit\Labs\DynamicSites\Ex02-ExposeData\end\PowerShellTests\verify.ps1
D:\VS2008TrainingKit\Labs\DynamicSites\Ex02-ExposeData\end\PowerShellTests\vs90vars.ps1
D:\VS2008TrainingKit\Labs\DynamicSites\Ex02-ExposeData\end\PowerShellTests\wcfserviceTest.ps1
D:\VS2008TrainingKit\Labs\LinqToSql\Ex01-DalUsingLinq\begin\PowerShellTests\testLib.ps1
D:\VS2008TrainingKit\Labs\LinqToSql\Ex01-DalUsingLinq\begin\PowerShellTests\verify.ps1
D:\VS2008TrainingKit\Labs\LinqToSql\Ex01-DalUsingLinq\begin\PowerShellTests\vs90vars.ps1
D:\VS2008TrainingKit\Labs\LinqToSql\Ex01-DalUsingLinq\end\PowerShellTests\testLib.ps1
D:\VS2008TrainingKit\Labs\LinqToSql\Ex01-DalUsingLinq\end\PowerShellTests\verify.ps1
D:\VS2008TrainingKit\Labs\LinqToSql\Ex01-DalUsingLinq\end\PowerShellTests\vs90vars.ps1
D:\VS2008TrainingKit\Labs\WF\Ex01-CreateStateMachineWF\begin\PowerShellTests\testLib.ps1
D:\VS2008TrainingKit\Labs\WF\Ex01-CreateStateMachineWF\begin\PowerShellTests\verify.ps1

It is really great to see the devdiv folks adopting PowerShell. The requirement to have PowerShell installed will help adoption in the developer space which I think is on the verge of exploding anyway.