Notes from #agileturas conference

It happens that I have attended Agile Tour Vilnius 2011 conference.

A lot of great speakers delivered interesting presentations on various Agile aspects from technical to management, for beginners and for those with some experience.

Keynotes were presented by J.B. Rainsberger, Mary Poppendieck and Jurgen Appelo.

Ok, here are few things that I’ve noted during the conference (typically, I try to take top 3 things for myself) so, here it comes:

• Scrum Master is not your scrum secretary (by Paul Goddard). Scrum master is not your agile PM, it is the one who should help the team to become self-organizing (also means effective).
• Jurgen has enforced the above by talking about “How to change the world” (no, not usual “change management” stuff of surviving change) by referencing complex adaptive systems theory discussed in his Management 3.0 book.
• Change cannot be implemented with a flip of the switch. It is more like dancing (guiding) to the desired result by constantly adjusting direction.

I guess that’s it for now. Time to look at our own team and have some dance

Parameterized queries, filtered indexes and LINQ

Sometimes I like to cite “Remember remember the fifth of November” in case, when I trip the mine that I teach others to avoid.

This time it happened, when reviewing some database structure and related code. So, here it goes.

The database had a table, which can be scripted as shown below. As you can probably note, we are talking about a table that holds queue of jobs that have to be performed.

CREATE TABLE [dbo].[Jobs](
    [Id] [int] IDENTITY(1,1) NOT NULL,
    [Priority] [tinyint] NOT NULL,
    [DateInserted] [datetime] NOT NULL,
    [DateAllocated] [datetime] NULL,
    [DateFetched] [datetime] NULL,
    [Status] [tinyint] NOT NULL,
    [SerializedData] [xml] NULL,
    [ParentId] [int] NULL,
    [ReferenceID] [bigint] NULL,
    CONSTRAINT [PK_Jobs] PRIMARY KEY CLUSTERED ( [Id] ASC )
    )
GO

ALTER TABLE [dbo].[Jobs] ADD  CONSTRAINT 
[DF_Jobs_DateInserted]  DEFAULT (getdate()) FOR [DateInserted]
GO

ALTER TABLE [dbo].[Jobs] ADD  CONSTRAINT 
[DF_Jobs_Status]  DEFAULT (1) FOR [Status]
GO

The use case for the table was simple:

• One part of the application inserts new jobs based on customer activities and by this sets the default date and status (1)
• Then service fetches jobs, by taking Id’s of specified amount of new (status=1) jobs and setting appropriate date on DateFetched and updating status to (2)
• Later at some point XML data will be retrieved using the Id’s retrieved previously

Initially, the code was LINQ 2 SQL in the program itself and was something like this:

var q = ctx.Jobs
            .Where(j => j.Status == 1)
            .Select(j => new { j.Id, j.Priority, j.DateInserted, j.Status })
            .Take(count);

For the testing purposes, I’ve loaded the table with 100k rows, out of which only 401 had status=1.

SELECT TOP (100) Id, [Priority], DateInserted FROM Jobs
WHERE Status=1

Running the query without any additional indexes, you will get the results, but there will be definitely a table scan with a little bit more of logical reads than you could expect.

(100 row(s) affected)Table 'Jobs'. Scan count 1, logical reads 648

 

Now, because the second step is looking for the new records (status=1), which normally should be small number of records comparing to the whole table size, so I immediately thought about creating non-clustered index for the status column, filter on the status=1 and include appropriate values (to have a small covering index). So I created one:

CREATE NONCLUSTERED INDEX IX_Jobs_NewJobs 
ON Jobs ([Status])
INCLUDE ([Priority], [DateInserted])
WHERE ([Status]=1)

This of course had a nice effect on the previous query:

(100 row(s) affected)Table 'Jobs'. Scan count 1, logical reads 2

For a moment I thought, that the rest should be fine, because query and LINQ statement were fairly straight forward with not much magic in them, but somehow, I’ve decided to see what query LINQ does produce.

The query was almost the same with one exception – parameterization, where the parameter value was of course equal to (1):

SELECT TOP (100) [t0].[Id], [t0].[Priority], [t0].[DateInserted], [t0].[Status]
FROM [dbo].[Jobs] AS [t0]
WHERE [t0].[Status] = @p0

However the shocking discovery for me at that point was the query plan:

Yes, SQL server was NOT using the non-clustered index I’ve created moments ago. WTF!
I’ve also checked the query actually shown by the SQL Server Profiler:

exec sp_executesql N'SELECT TOP (5) [t0].[Id], 
[t0].[Priority], [t0].[DateInserted], [t0].[Status]
FROM [dbo].[Jobs] AS [t0]
WHERE [t0].[Status] = @p0',N'@p0 int',@p0=1

Again, result was obviously the same.

Ok, it was obviously related to the parameterization of the query rather than some LINQ issue. It just happened that LINQ actually parameterizes (and for good) queries.

Quick Bing check resulted in http://msdn.microsoft.com/en-us/library/cc280372.aspx, which has clear statement on the subject: “In some cases, a parameterized query does not contain enough information at compile time for the query optimizer to choose a filtered index. It might be possible to rewrite the query to provide the missing information”

And this was the moment, when I cited the famous phrase. Was I sleeping during my own sessions, where I was talking about query plans, their caching, etc., like described here?

The problem becomes obvious, when you think that there are other possible values for which query plan using the non-clustered index would become invalid.

Solution

In my case, because of this and some other issues, I’ve decided to use stored procedure, which basically does all the job (retrieving the data and updating the state in one nice transaction).

UPDATE TOP (@BatchSize) Jobs 
SET Status = 2, DateFetched = GETUTCDATE()
OUTPUT inserted.Id, inserted.[Priority], inserted.DateInserted
WHERE Status = 1

Works like a charm

Just added 3TB disk to Windows Home Server v1.0

I bought recently 3TB WD drive. You may know that drives over 2.19TB are not supported in Windows XP/2003 systems due to the lack of GPT support. Windows Home Server of course doesn’t work with drives more than 2TB size, but there are some workarounds.

First, you will need a driver for HBA that comes with WD drive. I found that it is HighPoint Rocket 620 and you can find Windows XP/2003 drivers here.

So, after you will connect drive, install drivers, you should see 3TB drive in the disk manager. The rest is fairly simple, just follow the following posts:

http://spackle.wordpress.com/2011/04/22/going-for-3tb-in-whs-v1/

http://hardforum.com/showthread.php?t=1559559

For now, I’m running WHS v1.0 with 3TB drives with Drive Extender (didn’t needed to upgrade to 2011) nicely.

A little note: if Hitachi GPT Disk Manager doesn’t want to work with your disk, this isn’t a problem. It still installs GPT driver, so you can use disk manager to create GPT partition.

Windows Phone 7 from “user” perspective

One month ago I was thinking which type of phone I’d like to have. I was choosing between Android and Windows based phones. After some review reading, talking to people and hands-on experience I’ve selected Windows 7 phone: HTC Trophy.

There were couple of reasons for this:

• Didn’t wanted to have a toy to play “hacking” games
• Spending time figuring out OS version upgradeability not for me
• Wanted to know what works and what doesn’t early
• Form factor

In other words, I wanted more “user oriented” phone than a smart device.

Of course, WP7 misses a few things that I was used to on previous (Windows) phones. One of those features is: fast contact lookup using keypad – where pressing keys would search number and names at the same time. It was super convenient and fast, and for dialing I’ve used only this approach. Now – it’s gone. Bad, very bad. In the end, this particular feature reduced ergonomics of the phone a lot.

Actually, when reading reviews I was always wondering: everybody were talking about screens, sensitivity, applications, but no one (at least I haven’t found) noted that WP7 misses “phone capability”. I mean, when someone is talking about smart phone, then they mean smart device.

The second feature that I’ve missed once(!) was the ability to connect to hidden WiFi networks. Other than that – I’m pretty satisfied with the OS, applications and hardware.

And no, “copy/paste” is not on my wish list at all. Don’t have a smallest idea when I should use it. I didn’t had a single case in the month, when I’d miss copy/paste. I just don’t … and I don’t understand why people are putting it on top of the list. For me – bring back the fast contact search.

I’m using the phone for more than a month and one thing sure, my behavior and usage changed a bit. Before WP7, I barely used any smart phone more than a phone with smart address book (synced with Outlook). And initially my home screen had a few apps only:

• Phone for the easiest access to the recently dialed numbers
• People – the contact store
• Hotmail – if you don’t know it, then it is like a Gmail, but integrates with office documents better
• Messaging for SMS and parking (in Lithuania, you can pay for car park using “specially encoded” SMS messages)
• Calendar
• Twitter – for those who don’t know it, it is a stream of 140 character sized crap that you can get addicted to. And it made into my initial home screen. Stays so far …
• Settings – to deal with Wireless and some other necessities
• Weather

One month has passed and the screen has some changes:

• REMOVED: weather app – used really infrequently and live tile was updating was crap. In other words – useless.
• ADDED: Xbox Live – oh yeah baby … take a look at my Bejeweled Live achievements. Only 3 to go …
• ADDED: Outlook – corporate is corporate. Interestingly, phone became a the place, where both corporate meetings and private appointments are right in front of me and easily accessible. Much better than Outlook itself.
• ADDED: Parking-LT – finally an app that allows paying for parking done easy way. No SMS encoding anymore. Big thanks for the developers.
• ADDED: Amazon Kindle – I never thought that it will end up on my home screen, however, it did. The main reason for happening was that I’m using public transportation to get to the office quite a lot. I use this time to read books. I prefer books to twitter

In just a month, my usage patterns changed slightly and I could say that smart device features are getting into my life. However, I still miss the fast contact search with less clicks. I hope it will be fixed with “Mango”.

Next thing in my plans related to the phone – Visual Studio

After party or summary of presentation @ Powered by MVP

Today I was giving presentation at “Powered by MVP” event in Lithuania.

The topic: what to do, when programs leak, crash and misbehave in many other ways.

The summary: there are tools that can help you finding the cause problem and solving them. They only require some time to learn and some more time to master, but otherwise – it is much better than automatic IIS pool recycling because of a faulty app.

I would suggest the following way of mastering this part of skills.

Familiarize yourself with the subject like memory management

• Read this book: http://www.amazon.co.uk/gp/product/1906434409
• Read about Garbage Collection on MSDN: http://msdn.microsoft.com/en-us/library/ee787088.aspx
• Read MSDN magazine post on identifying and preventing memory leaks: http://msdn.microsoft.com/en-us/magazine/cc163491.aspx

Equip yourself with tools

• Grab sysinternals suite here. Familiarize with tools. I mean, even if just your PC startup is too long, check autoruns – I bet you’ll be amazed. Process Explorer and Process Dump – are the two developers tools that you must be familiar with.
• Windows SDK, where the WinDBG lies hidden
• Visual Studio
• CLR Profiler
• Any other third party tool like: Memory profiler from SciTech, Red Gates ANTS Memory profiler, JetBrains Profiler

Learn performance counters related to memory and performance. Don’t just say slow. It is not important. It is important to know why it is slow. Performance counters often can help answer that question.

Try some samples with those tools, read help materials and additional information on those, like:

• More on SOS extensions: http://msdn.microsoft.com/en-us/library/bb190764.aspx
• Crash dump analysis checklist: http://www.dumpanalysis.org/blog/index.php/2007/06/20/crash-dump-analysis-checklist/
• http://windbg.info/

Learn further

Read books like: “Windows Internals” and “Advanced Windows Debugging” (Recommended by @alejacma).

Start reading really interesting blogs on the subject, like:

• Blog of @TessFerrandez
• Blog of @alejacma

I guess that’s it for todays session. I hope you liked it.

If you didn’t understood everything, well – don’t say that I didn’t warned you about session being @ 400 level . Anyway, don’t worry – step by step you can learn those things fairly quickly. And sometimes, they can help a lot.

Why I will NOT upgrade to WHS 2011

I saw the this post (via @seandaniel) called “Why I plan on using the new Windows Home Server 2011” (a.k.a. “Vail”).

I have quickly ran my eyes over the blog post and well – I do NOT agree and I cannot accept the decision given the little I know.

Surely, I’m slightly different kind of persona and I am using WHS slightly differently and my priorities are different. For me it is a backup and storage machine that is used at home mostly. I’m accessing it very seldom from remote locations, just in case, when I need to grab some personal file.

So lets look at so called “just awesome stuff”.

• Improved remote access

I’m rarely using it. It is improved so that I can modify it? How nice, but it is so unimportant to me. With two small children at home I really don’t have time to “modify” it and I consider such time as wasted. I mean really, it’s not related to anything what this box is supposed to do – keep your files available and safe.

• Silverlight video streaming

I cannot imagine myself watching a Blu-ray RIP streamed via Internet and wireless connection, while I’m at hotel. Do you? One more nice to have that will be never used in my case. Most of the streaming is happening from WHS to Xbox @ home, when my kids are watching their content or myself – our personal stuff.

• Silverlight photo streaming

I would consider this to be a nice feature, if there would be a chance to share it easy with multiple relatives or casual “Oh, please …” type of friends. I’m considering writing add-in myself that would allow anonymous (with ticket) access to the defined set of photos on the WHS. Silverlight? Why not, it’s not that difficult, when you have appropriate data on photos. So is the photo streaming a “Wow”? It is most appreciated if it allows “anonymous/ticketed” access, but otherwise – thanks, but no. Thanks. I know my photos well.

• Moving files via remote access

Again, one nice feature, which will never be used by myself. If I’m about to touch any file, especially – delete. I will look (view it) carefully, if this is really the content I want to (re)move. I’ve done that locally over the shared folder and I’ll be continuing doing that.

• DLNA media streaming

It would be much more appreciated, if it could become a full Media Center with data storage in one box. DLNA itself is nice, however the only devices that support DLNA at home are my PC’s, which happen to play the content nicely without DLNA.

Regarding data protection, the talk is slightly different.

“What about the data on the server?  Server’s hold data, isn’t that data safe?  Yes of course it is.  You can backup that data to an internal, or better yet, external hard drive!”. At this point, I have only one question – so why do I need WHS then? I can have regular ***ux box with backup to an external drive. And also, considering theft/robbery scenarios, I prefer backing up critical data (photos and some videos) to Amazon (probably cheapest #cloud option) to keep the data out of home.

My main interest here is to have flexibility with hard drive management (I have replaced one dead drive, added some additional, replaced smaller with bigger one) and data availability without having to deal with limitations of RAID configurations or “Copy paste wizard”.

Sean, you say “I ultimately agree with the decision given what I know”. Well, you maybe agree, but you didn’t convinced me and, it seems, many others around the globe.

The question here is what is more important: you or customers?

IIS responds with 403 using certificate based authentication

From time to time I have to deal with certificate based authentication, when developing WCF services and from time to time I’m falling into the same pit.

Today I was configuring WCF service to use certificates for authentication (via AD certificate mapping). After configuring the IIS and WCF I’ve tried to access the SVC help page/metadata, but was getting 403.7 Forbidden: Client certificate required from IIS. The IIS logs contained something like this:

<date time> W3SVC1 <IP> GET /site/service.svc – 443 – <IP> <Browser> 403 7 64

Bing came out with the support KB article on this issue, but all possible causes were dealt with: CA was trusted, certs were not revoked or expired. And then it hit me – “not expired”, yes, of course – how the IIS checks the revocation of the certificate? Simply by looking into the certificates CRL distribution points information (if it is present there) and it must be accessible and reachable from IIS, which hosts the service. To check if everything is ok, just copy CRL’s URL from the certificate and try to open it via browser on IIS hosting service.

In my situation that was the problem, which was fixed easily by entering appropriate DNS records.

Of course, it is possible to switch off the certificate revocation checking on IIS, but that’s completely NOT recommended.