Keith's profileKeith Hill's BlogPhotosBlogListsMore ![]() | Help |
|
January 28 Tail-Content – Better Performance for Grabbing Last Lines From Large (ASCII) Log FilesNecessity is the mother of invention, or in the case of Windows PowerShell, a new script. I have a set of 23 large (28 MB) log files on a remote machine in which I need to verify that the last line of each of them is identical. My first “naive” approach was to do this:
Yeah unfortunately that took so long that I killed it and set out to create a script that would “efficiently” tail a file. Now my log files were ASCII encoded so it made the task much easier. Bottom line is that there is FileStream object in .NET that allows you to start at the end of a file and work backwards. The approach above using Get-Content requires that PowerShell get every single line from a log file with ~1,000,000 lines in it and over the network to boot. With FileStream you can read from the end of the file backwards. Because my files are ASCII, that is very easy to do. So I created a Tail-Content.ps1 script that you can download. Note that it doesn’t work on Unicode encoded files and it doesn’t do active tailing. However, it is very fast for large files. There are a few interesting parts of the code to examine. First, if you want to handle paths like PowerShell does, the snippets below show you how to setup your parameters to handle wildcard expansion and literal paths. This does require that you are on version 2 of PowerShell: 1: [CmdletBinding(DefaultParameterSetName="Path")] 2: param(3: [Parameter(Mandatory=$true, 4: Position=0, 5: ParameterSetName="Path", 6: ValueFromPipeline=$true, 7: ValueFromPipelineByPropertyName=$true)] 8: [string[]] 9: $Path, 10: 11: [Alias("PSPath")] 12: [Parameter(Mandatory=$true, 13: Position=0, 14: ParameterSetName="LiteralPath", 15: ValueFromPipelineByPropertyName=$true)] 16: [string[]] 17: $LiteralPath, 18: 19: <elided> 20: )Note that the default parameter set is “Path” and the Path parameter accepts pipeline input by name and by value. This means that raw strings will work as paths assuming they actually contain valid paths. Also note that both parameters are of type string array. The LiteralPath parameter is defined in a different, mutually exclusive parameter set named “LiteralPath” and it binds to pipeline input only by property name. It is important that we also decorated the LiteralPath parameter with the Alias attribute “PSPath”. This way output of Get-ChildItem (FileInfo) gets bound by property name to the LiteralPath parameter by virtue that PSPath is an alias for the same parameter. This happens because there is no Path property on FileInfo but there is a PSPath property. Remember that PowerShell extends the FileInfo type by adding the PSPath NoteProperty. That sets up the parameters, now here is what you need to do in your Process function to handle Path parameters which could specify paths with wildcards in them: 1: Process 2: {3: if ($psCmdlet.ParameterSetName -eq "Path") 4: {5: # In the non-literal case we may need to resolve a wildcarded path 6: $resolvedPaths = @()7: foreach ($apath in $Path) 8: { 9: $resolvedPaths += @(Resolve-Path $apath | Foreach { $_.Path }) 10: } 11: }12: else 13: { 14: $resolvedPaths = $LiteralPath 15: } 16: 17: foreach ($rpath in $resolvedPaths) 18: { 19: $PathIntrinsics = $ExecutionContext.SessionState.Path 20: 21: if ($PathIntrinsics.IsProviderQualified($rpath)) 22: { 23: $rpath = $PathIntrinsics.GetUnresolvedProviderPathFromPSPath($rpath) 24: } 25: 26: Write-Verbose "<cmdlet-name> processing $rpath" 27: 28: #process file here 29: } 30: }On line 3 I test which ParameterSet is being used. If it is the Path parameter set then we need to resolve the paths specified because they may contain wildcards. I do that on line 9 using Resolve-Path. Then on line 17 we iterate through each path and process it. One other detail that you may or may not need to worry about is that $rpath at this point may contain a provider qualified path e.g. Microsoft.PowerShell.Core\FileSystem::C:\foo.txt. These work fine with PowerShell however if you need to pass this path to a .NET object it won’t recognize that as a valid path. So on line 21 I check to see if we have a provider qualified path and if I do I get the raw path using $ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath as shown on line 23. The rest of this script just does low-level byte reads from the end of the file. I went back and measured my original approach of using { Get-Content \\server\share\logs\*_26_03.csv | Select -Last 1} and it took ~13 minutes. Using my Tail-Content script, it took < 1 second. That is a speed up of about 1789x! psmdtag:dotnet: FileStream Windows 7 Taskbar After Two WeeksAt this moment I have Windows 7 on the family room media center PC and dual booting on both my workstation and my laptop. On the dual boot machines I find myself almost exclusively booting into Win7. The only piece I’m missing is a driver for the fingerprint reader. Hopefully that will come shortly. Other than that, it is going great. Both machines seem snappier under Win7 than they do under Vista. With respect to the new taskbar, I have to say that *I* really like it. I still have concerns about the changing behavior of a mouse button left click on a taskbar icon depending on whether the app is already running or not. Perhaps Microsoft is banking on the non-techies not really running multiple instances of the same app and power users figuring out that they can use either shift+click or middle mouse button click to always fire up a new instance. I hope that is the case. Still, I can’t help but feel that there are going to be folks resistant to change and not giving them the option of the classic taskbar is a mistake. That said, if you don’t mind a bit of change and learning a few new tricks, I think you will find the new taskbar to be quite fun to use, maybe even somewhat addictive. There are a number of really nice little touches like when you hover the mouse over the thumbnail of a window, all the other windows go transparent so you can more easily that window. The ability of an taskbar icon to display progress is also very nice. There are also a number of new Windows key keyboard shortcuts in Windows 7 like Win+T, Win+<number> and Win+<arrow key>. Brandon Paddock has a listing of them here, be sure to check them out – if you are a keyboard shortcut kind of person. Updated 01-29-09: GPDuck on Twitter pointed me to these beta drivers for UPEK and AuthenTec fingerprint readers. I’m using it on my x64 HP Laptop and it is working fine so far. January 21 Windows 7 Media Center UpdateA few weeks ago I repaved the family Media Center to the official Windows 7 Beta. The results have been good. We no longer get any “Copying Prohibited” errors thank goodness! That was driving the family crazy. We do notice some lingering issues though with the primary one being that we can’t play movie DVDs. This one is odd because I can see all the various DVD files (IFO, etc) and the movie starts just a bit and then hangs. I can’t get to the Root or Title menus. Oh well, at least the kids can watch all their movies off of our Windows Home Server. Other issues we notice is that if we leave Media Center playing TV, turn off the TV and go away for a while, when we come back the Media Center screen is blank and won’t come back. You can Alt+F4 to close the Media Center app and re-launch it. Unfortunately this bug seems to be new to the Beta. We didn’t have this problem with the M3 build. Finally we notice random sluggishness is the Media Center UI. It sometimes comes back on its own and other times we need to kill the Media Center app and re-launch it. Other than that, we are pretty happy with this build. Just not having to worry about your shows not recording all the way because of the “Copying Prohibited” errors is a big relief. Regarding the new taskbar, I’ve been using Windows 7 on my primary PC at home (dual booting with Vista) and I have to say I like it. Except for the launching of a second instance of an application. I have come to like the middle mouse button click (or Shift+Click) to force the new taskbar to launch a new instance of the application. However I worry about this not being very obvious to the casual user. Then again, perhaps it is only us power users that launch more than one instance of an application? January 11 Could the new Taskbar Be the “UAC” of Windows 7?Let’s face it. Apple has had a significant impact on the negative perception of Windows Vista via their very successful I’m a Mac and I’m a PC switcher ads. Apple has taken a few nuggets of truth in terms Vista issues like UAC, most of which have been fixed in SP1 except UAC, and paraded them in front of the world in a attempt to get folks to switch from Windows to the Mac. Given what they spend on these commercials and increasing market share (still small though) I would say that it has been working for them. On the other hand we all know that advertising isn’t about pure truth. Many ads stretch the truth and some outright bend it to the snapping point. So the question is this. Shouldn’t Microsoft be taking extreme precautions *not* to give Apple any more ammunition for Windows 7 than they already have? I mean, you can already expect Apple to harp on Windows 7 as being nothing more than a warmed over version of Vista. Just take a look at the version number. Vista is 6.0 and Windows 7 is 6.1 – it’s a .1 release over Vista. How can that be significantly different than Vista? I’m afraid this new Taskbar a.k.a superbar is going to give Apple plenty of ammunition. The point of this post is to question the sanity of not providing a classic mode for the Taskbar in Windows 7. You might ask, why do that? Well, I agree with Paul Thurrot, the new Taskbar has some serious usability issues. I've done plenty of user centered design and usability tests in my job. I just did an albeit quick and informal Windows 7 Beta taskbar test with my wife (non-technical XP user) and the results weren't great. Note that I’ve setup Windows 7 to not hide labels. 1. Task: Start IE from the taskbar. Result: She finds the IE button, press it, done. Suffice it to say the wife acceptance factor for the taskbar is very low and she hasn't really gotten far into Windows 7. That's the problem that makes me fear for it’s acceptance. This new taskbar is the first hurdle you have to get over to appreciate the other niceties in Win7. But now, she's already got a bad taste in her mouth for Windows 7 and to be honest she is one of those folks who doesn't like Office 2007 either because of the ribbon UI. I'm more of a power user and I can respect MS trying to make Windows more approachable for the non-techies. As long as we power users can get some advanced options to tweak Windows to our advanced needs, we’re good. However I think there are some fundamental UI design tenets being broken here. Like changing the semantics of a taskbar button depending on whether the app is running or not. That's a *huge* mistake IMO. I also think it is a productivity hit because starting a new instance of an application, once you realize what's going on, is now:
It is now twice as hard to start a new instance of an application if it is already running. Tell me Microsoft, from your telemetry data, how often do folks fire up more than once instance of an app? Hmm, let's take IE and just one data point - me. Even with IE7's tab support I tend to have anywhere from 3 to 7 instances of IE running at once. BTW, I do know about the Shift+Click shortcut but I don't think that having the average computer user rely on that is viable in this case. FWIW I don't think the discoverability of the jump list is that big of a deal if and *only if* you don't need to use the jump list to start a second instance of an application. The jump list is very much like a right-click context menu so having to right-click to access it, feels natural. One minor quibble with jump lists, why does the top item in the common section say <app name> instead of “Start <app name>”? I would think that seeing an action like “Start Internet Explorer” in a list of actions makes more sense than seeing a noun “Internet Explorer” in a list of actions. I do not want to come across as bashing everything about the new taskbar. In fact, there is a lot I like about it. I really like the jump lists, the improved thumbnails, the thumbnails with mini-toolbars, button icon overlays, coloring and progress indication. I can even appreciate the desire to combine the quick launch buttons with the active task buttons to reduce redundancy and clutter. I can think of at least one reasonable solution to this *if* you’re in the mode where you are not hiding labels. What if the icon part of the button doesn’t change its semantics. It always fires up a new instance. Once an app instance is running, then you get a different actionable area (i.e. button) right next to the icon button perhaps with a very subtle vertical divider. The new button wouldn't have any icon so the overall width wouldn't increase – again assuming we aren’t hiding labels. In fact, if you have more than two instances running the width would actually be less due to the fact that there would only be one icon. This would also require less reliance on thumbnails to select from multiple running instances of an app. Frankly, if Microsoft were to ship Windows 7 today, I think this new taskbar could dog it like UAC has dogged Vista. I can just see the Apple commercials now hyping up how people hate *the* primary piece of UI in Windows 7 and how you've got *NO* choice. You either take the new taskbar or you leave it, er Windows 7 and take a Mac for a spin. I can see the Seinfeld soup Nazi inspired commercial now - "NO CLASSIC TASKBAR FOR YOU!!!". In summary, I think Microsoft would be insane to ship Windows 7 without a classic taskbar mode. Vista tried to foist UAC upon folks, which I think was the right thing to do if for no other reason than to get ISVs to fix their software to run correctly as a standard user. However it turned out badly for Vista perception and adoption. Office 2007 also took this all or nothing approach with the ribbon UI. But Office doesn’t have much in the way of a serious competitive thread yet. I’m not so sure that’s the same with Windows anymore, at least not in the consumer space. Apple has proven it can steal large chunks of market share in the consumer space as evidenced by the success of the iPod and the iPhone. And they're chipping away at the OS market share with their Vista advertising campaign and an OS that is apparently simple to use. Disclaimer: I haven’t used a Mac in 15 years but I keep hearing from friends that they are easy to use. Personally, I wouldn't want to give Apple any more ammunition than they already have for Windows 7 switcher ads. January 01 Renewed as Windows PowerShell MVP for 2009I got a very nice email today telling me that I have been awarded as a MVP for Windows Server - Admin Frameworks (aka Windows PowerShell). Woohoo! Not sure if I’ll get to make it to the summit this year because of an uncertain job situation but I’m going to try. |
|
|