| FPS Impact: NVIDIA Forceware 175.16 vs 175.19 |
| Reviews - Featured Reviews: Video Cards | |
| Written by Olin Coles | |
| Thursday, 26 June 2008 | |
NVIDIA Forceware Driver UpdatesMost people would think that when NVIDIA, who is the world-class leader in graphics technology, releases a new driver to the public that performance would either remain the same with some stability improvements, or that both performance and stability would increase. As a hardware enthusiast for over a decade, I know that this isn't true. Very recently NVIDIA updated their Forceware driver from version 175.16 to 175.19. Anyone familiar with their updates would agree that the version number difference is so small that it would suggest nothing has really changed. But did NVIDIA take two steps backward instead? Benchmark Reviews offers this small article to demonstrate what we discovered while testing the GeForce 9800 GTX and 9800 GX2 video cards. With the recent launch of NVIDIA's GeForce GTX 280 behind me, I began setting my sights on testing AMD/ATI's counter-offer to the market. Having just spent the better part of a week with a fresh new test system and about a dozen new video cards, our benchmark results were consistent and complete. This meant that I had to do little more than uninstall the Forceware driver and use Driver Cleaner Pro to remove the leftover traces, and then install ATI's Catalyst 8.6 to test the Radeon HD 4850.
Normally, this would be the end of the story. There would be another round of tests for the new card, and I would write my story and move on to the next project. Not this time. As it turns out, NVIDIA updated their Forceware driver from version 175.16 to 175.19. I didn't think anything of it at first, but since I must keep everything documented I decided to give the new updated Forceware driver a test. The GeForce 9800 GX2 was awaiting my direction so a new driver could be found and installed, and like always I use the "Have Disk" option to manually select the nv4_disp driver. Keeping in mind that my test procedure has been repeated about a dozen times each day for almost a month, it goes without saying that this has become second nature to me. My natural reflex had developed an uninstall Forceware > Driver Cleaner > restart Windows XP routine in my muscle memory. So would you think that I noticed any real differences? I figured that a wide variety of concentrated stresses would give me the tale of the tape in short order; but I had no idea that my results were about to become so different that I would postpone my review to write this article. Benchmark Reviews tests Crysis, Unreal Tournament 3, Lightsmark 2007, and 3dMark06 in this article to show how three tiny version steps forward would make a video card produce two completely different results. Read on for the big surprise, and be prepared to go backwards. Newer is better, right?Let me preface this entire article with a disclaimer: I hate seeing other articles that compare driver X against driver Y. If it were nothing serious, I would say to hell with it and let you all come to your own conclusions. But this is night and day, and it's going to ruin someone's fun if they aren't prepared for it. After all, I am like many of you, and feel that a newer should translate into better. Additionally, I often times update to the latest driver just to simply have the latest an greatest; usually based on the previously mentioned principal. I should also make another disclaimer: up to this point in our short 16-month existence, Benchmark Reviews has been testing video cards predominantly from NVIDIA. It's not that we dislike AMD/ATI, it's just that until recently they just didn't care to work with us. Now that we're launching product from both sides, I personally find myself bipartisan to the entire business. Despite what some of you might think, I review for results and nothing more. Nobody pats me on my back when I'm done, and neither NVIDIA or ATI offer me any sort of incentive for my opinion.
So what prompted all of this was something so simple, it was almost overlooked. As I had previously mentioned, my testing of the entire GeForce product family had just been concluded on a freshly re-imaged test system. Some of you who actually read my reviews (and not just skip over charts and conclusion) will know that I like the benchmark tool Lightsmark very much. This tool is so consistent that I can test a video card at any set resolution, restart the computer a day later, and when I retest my score will be within 0.01 FPS of the previous score. If only the games I used were so consistent. But I digress... What prompted this was a simple practice I follow whenever testing. Before I finish my testing on any particular video card product, I return to Lightsmark and run a consistency test to make sure the results haven't somehow changed (we've had cards go bad mid-test before). So when I did this re-test, I was ensuring that my scores were 100% reliable. About this time I discover that a new Forceware driver is available, and so I decide to give it a quick (or what should have been quick) comparison against the previous version. Of course, the rest is history... at least it will be after this article is done. Here's what NVIDIA mentions is their latest release notes:
Is 175.16 better than 175.19?My first and last test is always Lightsmark. So having just tested (and re-tested) Lightsmark at 1680x1050 using the version 175.16 Forceware driver, I was expecting a minor 0.01 FPS difference if there was to be any change at all going into the version 175.19 tests. What caught my attention is the difference in performance you see charted below. Lightsmark 2007
Now normally, this wouldn't be enough to raise suspicion, but when you're used to seeing result become so consistent that a frame rate score can remain within the same whole number for an entire series of tests you slowly learn to trust experience. So the very first thing I did is question my results, which meant that back the previous driver version for more retesting just to be sure. After a few more hours of testing back and forth... I was sure. NVIDIA: we've got a problem. 3dMark06 DifferencesSo what I did next was see just how deep the sink hole went, and began testing and retest, uninstall and driver clean, and then do some more testing. Apparently, the sink hole is more like a fault line, because 3dMark06 seemed to divide its previous score in half when the new 175.19 driver was used.
Wow! That's the only word I could summon from my vocabulary when I first compared the results. With all of the recent discussion surrounding NVIDIA and 3dMark, perhaps this is an argument in the opposite direction. I can't seem to find anything in the release notes that would support such a major drop in performance, but needless to say, it's there for sure. In our next section, we hit the target for PC video game stress with our Crysis benchmark tests. For the record, I don't like to play Crysis, but it serves as a very good performance benchmark for other high-demand video games. So here we are, at the heart of my article. Before I dive right into the Crysis benchmark results, I should admit that my history with Forceware performance fluctuations is minimal at best. For all I know, these dramatic changes have happened before on more than one occasion, but for the past sixteen months that Benchmark Reviews has been in existence I can't recall such an event. Crysis Benchmark TestsCrysis uses a new graphics engine: the CryENGINE2, which is the successor to Far Cry's CryENGINE. CryENGINE2 is among the first engines to use the Direct3D 10 (DirectX10) framework of Windows Vista, but can also run using DirectX9, both on Vista and Windows XP. Roy Taylor, Vice President of Content Relations at NVIDIA, has spoken on the subject of the engine's complexity, stating that Crysis has over a million lines of code, 1GB of texture data, and 85,000 shaders. To get the most out of modern multicore processor architectures, CPU intensive subsystems of CryENGINE 2 such as physics, networking and sound, have been re-written to support multi-threading. Crysis offers an in-game benchmark tool, which is similar to World in Conflict. This short test does place some high amounts of stress on a graphics card, since there are so many landscape features rendered. For benchmarking purposes, Crysis can mean trouble as it places a high demand on both GPU and CPU resources. Benchmark Reviews uses the Crysis Benchmark Tool by Mad Boris to test frame rates fo the Island timedemo to run in batches, which allows the results of many tests to be averaged.
Our first tests were at the ultra-low resolution of 1024x768. At this resolution, the GX2 has a tough time keeping up with the rapid-pace demands of such short frame buffer calls. Nevertheless, the GeForce 9800 GTX performance drops from 69.285 FPS with Forceware v175.16 down to 64.53 with version 175.19. Alternatively, the GeForce 9800 GX2 produces 65.64 FPS with Forceware 175.16 and drops way down to 52.63 with Forceware 175.19.
Without a doubt, the 9800 GX2 does not seem to perform well with Forceware 175.19 at low resolutions. Although it holds its ground as the graphical demands are raised, as you would expect a dual-GPU laden video card to do, the GeForce 9800 GTX takes a much smaller performance hit with the 175.19 driver when compared to the major performance reduction that the 9800 GX2 experiences. In the final comparison test, I focus on the GeForce 9800 GX2 and add post process anti-aliasing to the settings as the resolution is raised to 1680x1080.
Now, I'm not an expert on Forceware drivers, but I think that the idea is to retain performance with each new revision; or improve upon it whenever possible. Someone might want to remind NVIDIA's Forceware team that now is not the best time to have your top-tier products limp along as product analysts like myself are testing them against the competition. What makes things worse is that the average enthusiast will give very little regard to the driver version when the observe benchmark results, since they often simply scan the conclusion and glance at the charts. For this reason, I always include a detailed test methodology, so that you can duplicate my results if you desire. Video Card Testing MethodologyBefore the tests begin, the previous display adapter driver is uninstalled and trace components are removed using Driver Cleaner Pro. We then restart the computer system to establish our display settings and define the monitor and install new drivers fresh. Once the hardware is prepared, we begin our testing. The synthetic benchmark tests in 3DMark06 will utilize shader models 2.0 and 3.0. Every test is conducted at the following resolutions: 1600x1200 (20.1/21" Standard LCD's), 1280x1024 (19" Standard LCD), and 1024x768 (17" Standard LCD). In some tests we utilized widescreen monitor resolutions, since more users are beginning to feature these products for their own computing. Each benchmark test program begins after a system restart, and the very first result for every test will be ignored since it often only caches the test. This process proved extremely important in the World in Conflict and Supreme Commander benchmarks, as the first run served to cache maps allowing subsequent tests to perform much better than the first. Each test is completed five times, with the average results displayed in our article. Our site polls and statistics indicate that the over 90% of our visitors use their PC for playing video games, and nearly 70% of you are using one of the screen resolutions mentioned above. Since all of the benchmarks we use for testing represent different game engine technology and graphic rendering processes, I feel that this battery of tests will provide a diverse range of results for you to gauge performance on your own computer system. Since most gamers and enthusiasts are still using Windows XP, it was decided that DirectX 9 would be used for all tests until demand and software support improve for Windows Vista. Test System
Benchmark Applications
Test Products
Final Thoughts on ForcewareAs I have previously mentioned in this article, it's my belief that drivers should improve stability and performance with each new revision. Now, one could argue (if they had the data) that NVIDIA enhanced stabililty at the expense of performance. That someone would probably be playing Devils advocate, because I can't believe any reasonable person would decide that very good stability raised to extremely good should result in very good performance dropping to fair. This kind of argument also doens't explain why 3dMark06 would see a 55% reduction in performance between driver versions. Of course, I will have to reiterate for the ump-teenth time that all Forceware drivers were uninstalled using the Add/Remove programs interface, and then Driver Cleaner Pro was utilized to remove any remaining traces. I have to repeat all of this, because of the hordes of "readers" who don't bother to actually read. So now that this information is on nearly half the pages of the article, I should only receive half the hate mail from visitors claiming I should use Driver Cleaner Pro. Anyway... It's not up to me to make decisions for you. I know that as a hardware reviewer, benchmark scores are worth their weight in gold because of the time spent on gathering them. If one tiny driver revision changes the entire table of scores, then there's very good reason for concern. For me, as a reviewer, I have to be careful now that I not test with the Forceware v175.19 driver as I benchmark ATI's Radeon HD 4850 and 4870. All of my GeForce GTX and 9800 series benchmarks were conducted with the 175.16 driver, and that's what I will use for future comparison in the coming articles. But I'm not the only one who should be alarmed: you should be, too. My money doesn't get spent on video cards, since most manufactures will loan the websites a sample to test that gets returned or exchanged for another. So when it comes to reading the reviews out on the web, you'll want to make sure they disclose the full details of the test methodology. Most website mention the brand and model of the product, but often times it's nearly impossible to figure out which driver they're using. This doesn't mean that the benchmark results on other sites are wrong, but it does mean you should be wary of those who are unaware of the impact that Forceware 175.19 has on video cards like the 9800 GTX and GX2. Ultimately though, you don't even have to take my word for it. Go benchmark the two driver versions yourself. 3dMark06 offers a trail version (to the best of my knowledge), and Driver Cleaner Pro is available for free from our affiliate driverheaven.net. After all I've done my homework and it's your time and money at stake, not mine. Questions? Comments? Benchmark Reviews really wants your feedback. We invite you to leave your remarks in our Discussion Forum. EDITORS NOTE: I apologize that this article did not get more involved with control panel changes and other benchmark results, but my time was cut very short due to unexpected travel. UPDATE 06/27/08: Thank you to forum member slickric21 who points out that others are producing similar results with Forceware 175.19. He mentions several other discussions in his post, which identify missing profiles in this version of the driver. Read more about this issue in our FPS Impact: NVIDIA Forceware 175.16 vs 175.19 discussion thread. UPDATE 06/27/08: Only one day after this article was published, NVIDIA has pulled the Forceware 175.19 driver from its website and re-published the version 175.16 driver. Benchmark Reviews agrees with their decision, as the Forceware 175.16 worked considerably well in all of our tests. UPDATE 07/07/08: Since this article has published, NVIDIA has pulled the v175.19 driver and then later re-posted it. Presently the 175.16 driver is available to the open public. It would be my advice to wait for an entirely new driver that doesn't wear the 175.19 name.
|
|
