FOF Extractor (Updated to Version 4.3.0, 6/18/2006: Draft History Screen)

[Subby]

Greg - no worries, really - it is pretty easy to figure out the csv header on the analayzer output using the ananlyzer itself + the help file for extractor.

Much appreciated!

[Dutch]

Something has gone terribly wrong with my version of the gstelmack extractor. I installed the newest version and get a Win XP error when I try to open the extracted FOFRoster.txt file. So I remove it through the control panel, go back to the old version and the extractor's FOFRoster.txt outputs only the very last guy I try to extract.

No!!!!

[Dutch]

For a better picture, here's some screenshots of my GEFL league. The first one pops up because it thinks it's coming from TCY, it's a historic modified file. The second one is the result of the effort. Any help is appreciated!

[gottimd]

Try the load button on the FOFRoster instead of the Open Draft file button.

[Dutch]

I tried that and it gave me another XP crash error.

However, immediately after that, I re-ran Open Draft File and it worked as it's supposed to.

That was odd, but perhaps the extraction didn't work right the first time (I re-extracted the data).

Thanks.

[gstelmack]

The TCY stats dialog should ALWAYS come up when opening an Extractor file. If it doesn't, you clicked the wrong load button. Normally you just hit Cancel on that dialog, but if you've got a TCY career you can use it (combined with TCY HTML Export utility) to grab TCY stats for Analyzer.

More than likely you had a problem with the export. Remember to leave Extractor alone while it's running. Having any window open on top of the FOF window will screw up Extractor. It's best to start it and then just walk away until it finishes.

Of course, now that Interrogator is running and dumping lots of this data, I may be able to rip out the contract stuff from Extractor and speed it up again...

[Dutch]

Originally Posted by gstelmack

The TCY stats dialog should ALWAYS come up when opening an Extractor file. If it doesn't, you clicked the wrong load button. Normally you just hit Cancel on that dialog, but if you've got a TCY career you can use it (combined with TCY HTML Export utility) to grab TCY stats for Analyzer.

More than likely you had a problem with the export. Remember to leave Extractor alone while it's running. Having any window open on top of the FOF window will screw up Extractor. It's best to start it and then just walk away until it finishes.

Of course, now that Interrogator is running and dumping lots of this data, I may be able to rip out the contract stuff from Extractor and speed it up again...

Hmmm, that might be it. I opened up IE (it opens maximed) right over the top of the extractor, then I minimized it, and drug it to my secondary monitor. In that space of time, it may have ruined the extraction process.

I'll be more careful in the future. Thanks again for a great utility. It rocks!

[stevew]

One minor thing that would be nice, if you could possibly add on a future version(if you think its a good idea) would be the option to Extract a numbered amount of players. For instance, Most of the time i only want to look at the first 100 or so players to make my next selection. Its a pain in the ass to have to wait till all the 12/21 punter types extract. Not to mention the horror of earlier today when i had extracted everything, and then must have accidentally hit Extract again, and found one player in my file

[sovereignstar]

Originally Posted by stevew

One minor thing that would be nice, if you could possibly add on a future version(if you think its a good idea) would be the option to Extract a numbered amount of players. For instance, Most of the time i only want to look at the first 100 or so players to make my next selection. Its a pain in the ass to have to wait till all the 12/21 punter types extract. Not to mention the horror of earlier today when i had extracted everything, and then must have accidentally hit Extract again, and found one player in my file

Can't you just manually stop the extractor or close a player window in FOF? I think the extracted data should be fine.

[stevew]

Originally Posted by sovereignstar

Can't you just manually stop the extractor or close a player window in FOF? I think the extracted data should be fine.

I always get an error if i do anything to it.

[gstelmack]

Yeah, I need to get around to adding a cancel button one of these days...

[gstelmack]

Version 3.2.0 posted. Changes include:

Now built with Visual Studio .NET 2005 final, so no more beta runtime!
Fixed calculation of player indices for the most recent year's PlayerGame file in Interrogator.

[gottimd]

Can't get this to work and I have updated all necessary files.

[gstelmack]

Originally Posted by gottimd

Can't get this to work and I have updated all necessary files.

I forgot to properly update version numbers / GUIDs in the installer with the new version. If anyone else is having problems, please try re-downloading and re-installing.

Uninstalling the old version may help as well.

[gottimd]

That worked! Thanks for the hard work again.

[3ric]

Greg, something I found with the Interrogator: after trying it out for the first time (yesterday) I started pushing all the data into a database and queried it for a roster and then pulling game-by-game stats for a single player. But the player id matched with the name doesn't match his stats in the stat files. In the 2008 file he's got a different id, and another one in the 2007 file, another one for 2006 and so on...
Basically it looks about like this:
Player id/Year/Team/Game
308 2008 0 0
308 2008 0 1
308 2008 0 2
...

Player id/Year/Team/Game
325 2007 0 0
325 2007 0 1
325 2007 0 2
...

Player id/Year/Team/Game
316 2006 0 0
316 2006 0 1
316 2006 0 2
...

when his id should be 256 when you look it up using his name. The only way I can tell it's the right player is to compare the stat lines with the game.

[gstelmack]

I'll have to check. Off the top of my head, I though the player ID was right there in the files, so I'm not sure what's up, but I've already fixed one bug in this area, so there could be something else.

I'll have to find time in and among my work to obsolete your game viewer

[gstelmack]

Originally Posted by 3ric

Greg, something I found with the Interrogator: after trying it out for the first time (yesterday) I started pushing all the data into a database and queried it for a roster and then pulling game-by-game stats for a single player. But the player id matched with the name doesn't match his stats in the stat files. In the 2008 file he's got a different id, and another one in the 2007 file, another one for 2006 and so on...

Here's how the routine I've got works:

For each year of the league
For each player in the PlayerRec2 file that has an entry for that year
Read his record from that year's record book

This works for at least the first few players each year in my test league. More than likely there is either a player without an entry for that year in the player rec2 file that has an entry in the year, or the reverse is true.

If you can figure out which player is skipped or added (who is the first player in that year's data that is wrong?) and can then send me those league files (not any of the box scores, just the GAMENAME.* files), I can probably figure out what flag or indicator I'm missing.

[3ric]

I investigated this with the CFL files, you can download them here: http://www.thecfl.net/files.html

I checked the first players in the rec2 file with their 2008 stats, and got this:
id/name/position/status
1 Douglas hartman qb
3 Francis Burch qb
5 (Jose Rohrer retired rb)
6 Brooks Creel rb
7 Andrew Kendall fb
8 (jack Wright retired rb)
10 (Mervin Braddy retired te)
11 (Jarvis Taylor retired te)
12 (Vito Pulley retired te)
13 *Nestor Lambrecht free agent wr has his stats matched to id 14 - the first error
14 Chris Chavez wr, has his stats matched to id 17 - the second error

Let me know if I can look up something else to help you out.

[gstelmack]

Originally Posted by 3ric

I investigated this with the CFL files, you can download them here: http://www.thecfl.net/files.html

I checked the first players in the rec2 file with their 2008 stats, and got this:
id/name/position/status
1 Douglas hartman qb
3 Francis Burch qb
5 (Jose Rohrer retired rb)
6 Brooks Creel rb
7 Andrew Kendall fb
8 (jack Wright retired rb)
10 (Mervin Braddy retired te)
11 (Jarvis Taylor retired te)
12 (Vito Pulley retired te)
13 *Nestor Lambrecht free agent wr has his stats matched to id 14 - the first error
14 Chris Chavez wr, has his stats matched to id 17 - the second error

Let me know if I can look up something else to help you out.

Thanks, I'll check it out tonight.

[gstelmack]

Originally Posted by 3ric

I investigated this with the CFL files, you can download them here: http://www.thecfl.net/files.html

I checked the first players in the rec2 file with their 2008 stats, and got this:
id/name/position/status
1 Douglas hartman qb
3 Francis Burch qb
5 (Jose Rohrer retired rb)
6 Brooks Creel rb
7 Andrew Kendall fb
8 (jack Wright retired rb)
10 (Mervin Braddy retired te)
11 (Jarvis Taylor retired te)
12 (Vito Pulley retired te)
13 *Nestor Lambrecht free agent wr has his stats matched to id 14 - the first error
14 Chris Chavez wr, has his stats matched to id 17 - the second error

Let me know if I can look up something else to help you out.

In this particular example, I'm not sure there is an actual bug. #13 Lambrecht is not in the output data because he did not play in any games. I only write the output for games in which a player actually played. So a guy can have an entry for a particular year in PlayerRec2, but not have any data written out for that year.

I can change this if desired, but you'll end up with much longer yearly data files filled with lots of rows of 0s.

I can take a look at the original player example if you are convinced that the guy has the wrong IDs in the CSV files. But I'd double-check to make sure the CSVs are incorrect, and not that you are assuming that every player in the PlayerRec2 file has 16 games (or even one game) in the yearly stat file when you do your import.

[3ric]

No, Lambrecht did play in seven games in 2008, with 31 yds in week 1 and 15 yds in week 3 when I look in the game. But going by the csv files (and id number 13) he didn't play at all, because his stats has another id number (in this case 14)

[gstelmack]

Originally Posted by 3ric

No, Lambrecht did play in seven games in 2008, with 31 yds in week 1 and 15 yds in week 3 when I look in the game. But going by the csv files (and id number 13) he didn't play at all, because his stats has another id number (in this case 14)

Hmm, okay, I'll try and figure that out. Sounds like there is a disconnect where guys that don't play any games in a season don't have that year marked in the PlayerRec2 but do have an entry in the season data file. I'll have to see if I can figure out how to identify those guys.

[gottimd]

I noticed this too when I ran my interrogator program macro against the CSV outputs to compile a player card with yearly statistics on it. I always forgot to mention it, thanks 3ric for bringing that up.

[gstelmack]

It looks like retired players are messing this up. Vito Pulley, TE (index 12), retired in 2008, and it looks like I'm attributing his records to Lambrecht, and then throwing them out because he didn't play in any games. Looks like I need to figure out a "year retired" column and add it in.

[gstelmack]

Originally Posted by gstelmack

It looks like retired players are messing this up. Vito Pulley, TE (index 12), retired in 2008, and it looks like I'm attributing his records to Lambrecht, and then throwing them out because he didn't play in any games. Looks like I need to figure out a "year retired" column and add it in.

I think I've figured this out. Next release will include "year retired" and will account for this when reading season stats. It will also include some hall of fame fields. You'll need to update your PlayerRec2 readers to account for these new fields. Let me know if any of this still looks off (especially Hall of Fame, as I have limited test cases and only THINK I have these fields figured out).

I'm hoping to release this either tomorrow or Monday. I'm also including a preview of my new game viewer, but I have one or two more bits to clean up in it.

[gstelmack]

Version 4.0.0 was released today. Changes include:

• New Utility: FOFGameViewer. This is an early release of a new graphical game viewer. It is usable, but is missing stat tracking and planned on-field animations (formations, ball movement, etc.) Comments and suggestions are welcome.
• Interrogator now deals better with retired players. Figures out what year a player retired (written to the PlayerRec2 file) and accounts for those players when reading stats for a year.
• Interrogator now dumps out Hall of Fame fields into the PlayerRec2 file.

Next on my list will be to deal with any Interrogator issues related to these tweaks (let me know how the changes work), and to start adding features to the GameViewer. Within the next couple of days I'd like to release a new game viewer with the ability to display endzone and midfield logos for the home team so people can start working on logos for their leagues...

[flere-imsaho]

I think I just wet myself....

[gottimd]

Originally Posted by gstelmack

Version 4.0.0 was released today. Changes include:

• New Utility: FOFGameViewer. This is an early release of a new graphical game viewer. It is usable, but is missing stat tracking and planned on-field animations (formations, ball movement, etc.) Comments and suggestions are welcome.

If you can get it to look like Sportslines in game viewer (stats for players included with on field movement graphics) I may ask you to marry me.

[gstelmack]

Originally Posted by gottimd

If you can get it to look like Sportslines in game viewer (stats for players included with on field movement graphics) I may ask you to marry me.

That's the plan. You can already see the tabs set aside for player and team stats, and I intend to use all that field real estate to show formations and ball movement (where the run goes, where the catch is made, where the tackle ends up being, etc etc etc, whatever the game log lets me show).

[gstelmack]

• Game Viewer now shows the status of the current drive (plays, yards, time).
• Game Viewer now supports endzone and field logos for the home team. See the documentation for details.

Details on the logos with this version (copied from the docs included with the installer):

Originally Posted by Documentation

The Game Viewer supports custom field graphics. It will look for two files based on the home team abbreviation in the "Images" folder: ABBREV_endzone.png, and ABBREV_field_logo.png, where ABBREV is the 3-letter city abbreviation for the home team. For example, if New England is hosting a game, the utility will use NED_endzone.png and NED_field_logo.png for on-field graphics. Two default files are provided that will be used if the home team graphics aren't found.

PNG files are used because they support transparencies and because tool support for them is fairly widespread. The endzone graphic should have the same size as the provided default graphic to properly fill the endzone, while the field_logo one may be of any reasonable size. The viewer simply centers the field_log and uses its actual size, so be careful of making one that is TOO big. The viewer uses 1 pixel = 6 inches, so 6 pixels = 1 yard. The default graphic is 120x120 pixels, or 20 yards by 20 yards.

Note that updates will probably slow down a bit now (except for bug fixes). Next up is to start working on stat tracking, and that's a lot of work even if I go stat type by stat type. Have to figure out penalties as I need to know which penalties are added on to the play, and which ones override the play, which isn't always obvious in the logs.

[Dutch]

gtelmack is en fuego this weekend.

[3ric]

To continue with my previous bug report, the 2008 data for the CFL looks correct now, but I've found another one with the 2007 data - Edward Levy with id 236 has got the rushing stats of Joshua Urrutia, with id 229. Urrutia did play in 2007, but has no stats for that year. But WR Scot Volpe, with id 186, has the correct rushing stats displayed in the csv for 2007.

[gstelmack]

Originally Posted by 3ric

To continue with my previous bug report, the 2008 data for the CFL looks correct now, but I've found another one with the 2007 data - Edward Levy with id 236 has got the rushing stats of Joshua Urrutia, with id 229. Urrutia did play in 2007, but has no stats for that year. But WR Scot Volpe, with id 186, has the correct rushing stats displayed in the csv for 2007.

I'll take a look.

[gottimd]

Everything loaded fine from the installer, but when I try and run the interrogator now with this new version, I am getting the following error:

"Unhandled exception has occurred in your application"

"Index was outside the bounds of the array"

See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.
************** Exception Text **************
System.IndexOutOfRangeException: Index was outside the bounds of the array.
at FOFReader.FGADataFile.WriteStaffCSV(String outPath)
at FOFReader.FGADataFile.WriteCSV(String outPath)
at FOFInterrogator.MainForm.buttonProcess_Click(Object sender, EventArgs e)
at System.Windows.Forms.Control.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ButtonBase.WndProc(Message& m)
at System.Windows.Forms.Button.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

************** Loaded Assemblies **************
mscorlib
Assembly Version: 2.0.0.0
Win32 Version: 2.0.50727.42 (RTM.050727-4200)
CodeBase: file:///C:/WINDOWS/Microsoft.NET/Fra...7/mscorlib.dll
----------------------------------------
FOFInterrogator
Assembly Version: 4.0.1.0
Win32 Version: 4.0.1.0
CodeBase: file:///C:/Program%20Files/Greg%20St...terrogator.exe
----------------------------------------
System.Windows.Forms
Assembly Version: 2.0.0.0
Win32 Version: 2.0.50727.42 (RTM.050727-4200)
CodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL...dows.Forms.dll
----------------------------------------
System
Assembly Version: 2.0.0.0
Win32 Version: 2.0.50727.42 (RTM.050727-4200)
CodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL...089/System.dll
----------------------------------------
System.Drawing
Assembly Version: 2.0.0.0
Win32 Version: 2.0.50727.42 (RTM.050727-4200)
CodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL...em.Drawing.dll
----------------------------------------
FOFReader
Assembly Version: 4.0.1.0
Win32 Version: 4.0.1.0
CodeBase: file:///C:/Program%20Files/Greg%20St.../FOFReader.DLL
----------------------------------------
************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.
For example:



When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.

[gstelmack]

Originally Posted by gottimd

Everything loaded fine from the installer, but when I try and run the interrogator now with this new version, I am getting the following error:

"Unhandled exception has occurred in your application"

"Index was outside the bounds of the array"

See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.
************** Exception Text **************
System.IndexOutOfRangeException: Index was outside the bounds of the array.
at FOFReader.FGADataFile.WriteStaffCSV(String outPath)

The only way this happens is if a coach has an ability value < 0 or > 100. I'll clamp the range of these variables to that of my text mapping and get a new version out shortly (I also want to check out 3ric's report).

[gstelmack]

Originally Posted by 3ric

To continue with my previous bug report, the 2008 data for the CFL looks correct now, but I've found another one with the 2007 data - Edward Levy with id 236 has got the rushing stats of Joshua Urrutia, with id 229. Urrutia did play in 2007, but has no stats for that year. But WR Scot Volpe, with id 186, has the correct rushing stats displayed in the csv for 2007.

I think Hugh Jenkins is messing this up. Or in other words, I'm not calculating retirement years correctly for undrafted free agents (since they don't have a "year drafted" field to key off of). I'll have to adjust that calculation again.

[gstelmack]

Originally Posted by gstelmack

I think Hugh Jenkins is messing this up. Or in other words, I'm not calculating retirement years correctly for undrafted free agents (since they don't have a "year drafted" field to key off of). I'll have to adjust that calculation again.

Bah, this gets worse. Trying a different tack now, but may take a bit. Figuring out which guys appear where in which record files is turning out more difficult than it first appears.

[gstelmack]

Interrogator Fixes:

• Fixed calculation of player indices in the transaction log.
• Guarded against out-of-range errors on staff attributes.
• Match correct players up to historical records.
• Remove "Year Retired" field in the PlayerRec2 fields as unreliable to calculate.

I verified the historical records thing by checking the last guy in the 2007 CSV against his real stats for the CFL files provided by 3ric, and he was correct, so I'm somewhat confident I've got this nailed. But that wasn't a thorough check, so let me know if anything else slipped through. There will be a bunch more records in those files now. I'm too busy dumping this data out to have time to write utilities to make it useful, so most of my checking is by hand. Please feel free to keep pointing out the errors as it's easier for you guys to catch these than it is for me.

Also note the transaction log issue. Based on some retirement checks it looks like I was using the index of the guy's entry in the PlayerRec2 file rather than his actual player index, but let me know if player transactions look off now.

[gottimd]

Originally Posted by gstelmack

Interrogator Fixes:

• Fixed calculation of player indices in the transaction log.
• Guarded against out-of-range errors on staff attributes.
• Match correct players up to historical records.
• Remove "Year Retired" field in the PlayerRec2 fields as unreliable to calculate.

I verified the historical records thing by checking the last guy in the 2007 CSV against his real stats for the CFL files provided by 3ric, and he was correct, so I'm somewhat confident I've got this nailed. But that wasn't a thorough check, so let me know if anything else slipped through. There will be a bunch more records in those files now. I'm too busy dumping this data out to have time to write utilities to make it useful, so most of my checking is by hand. Please feel free to keep pointing out the errors as it's easier for you guys to catch these than it is for me.

Also note the transaction log issue. Based on some retirement checks it looks like I was using the index of the guy's entry in the PlayerRec2 file rather than his actual player index, but let me know if player transactions look off now.

Solved the issue I was having, Thanks for all your hard work!

[3ric]

Originally Posted by gottimd

Solved the issue I was having, Thanks for all your hard work!

What he said! Thank you for a terrific utility, Greg!

[gstelmack]

Bump for the workweek crowd.

[sovereignstar]

Awesome work, Greg.

[brewcrewmaroon]

Just checked out the utilities. Very "Suite". However, was having trouble using the game viewer. The log processor has no trouble finding the game logs, but for some reason the game viewer does not list. As far as I know the logs are in the default directory. Am I missing something here?

[gottimd]

Originally Posted by brewcrewmaroon

Just checked out the utilities. Very "Suite". However, was having trouble using the game viewer. The log processor has no trouble finding the game logs, but for some reason the game viewer does not list. As far as I know the logs are in the default directory. Am I missing something here?

I think that this (Game viewer) is basically in a "beta" stage of development, but I can't speak for gstelmack.

[gstelmack]

Originally Posted by brewcrewmaroon

Just checked out the utilities. Very "Suite". However, was having trouble using the game viewer. The log processor has no trouble finding the game logs, but for some reason the game viewer does not list. As far as I know the logs are in the default directory. Am I missing something here?

Log processor does a dumb "list all the game logs in a given directory", which can be slow when you have lots of games.

Game Viewer opens up every FOF saved game in the FOF directory, grabs the list of game logs from it, and lists those. If it's not finding any games, then either:

1) It's having trouble finding your FOF installation (it looks for the registry key). Is FOF correctly installed on your computer?

2) It's not finding any saved games that have logs in that FOF installation.

If you're just dumping logs off somewhere and trying to get it to find them, it won't work. It needs a valid FOF installation to work from. At the moment this is just for speed, but in the future if I get ambitious it would allow me to do things like "RB x just went over 1000 yards on the season" or "QB y just broke the record for most completions in a game" or the like.

[gottimd]

Originally Posted by gstelmack

but in the future if I get ambitious it would allow me to do things like "RB x just went over 1000 yards on the season" or "QB y just broke the record for most completions in a game" or the like.

Like I said before, if this ever gets up and running, I will vote for you for president.

[brewcrewmaroon]

Got it. Once I saved the game in progress and went back all the logs were there. Thanks!

[Coder]

Tested the gameviewer yesterday.. wohoo . Of course, seeing the team-logos as you do in the other one is a nice feature (hint hint, nudge nudge).

One thing though.. in the Imperial League, my team is the Vermont Mountaineers, having been renamed so from the Burlington Bolts. The abbreviation is VMT, which is used in the game. However, in the gameviewer, despite the box-score mentioning Vermont and VMT, the teams say BUR rather than VMT.

[gstelmack]

Originally Posted by Coder

Tested the gameviewer yesterday.. wohoo . Of course, seeing the team-logos as you do in the other one is a nice feature (hint hint, nudge nudge).

I suck at art. I mean, did you look at the default logos I included? However, there is full documentation (which isn't much) on what format the endzone and midfield logos need to be for the game to use them. So any intrepid artist could create logo packs for any league they wanted to (hint hint nudge nudge).

Originally Posted by Coder

One thing though.. in the Imperial League, my team is the Vermont Mountaineers, having been renamed so from the Burlington Bolts. The abbreviation is VMT, which is used in the game. However, in the gameviewer, despite the box-score mentioning Vermont and VMT, the teams say BUR rather than VMT.

I'll investigate where I'm pulling the city abbreviations from. Probably not grabbing the correct universe versions or something.